Add Helper Function to split a Nested Column by a categorical column value

[dougbrn]

Feature request
Splitting a nested structure into separate parts is probably a pretty common operation, especially splitting apart a Nested lightcurve into individual single-band lightcurves. It can be done relatively simply:

for band in nf["nested.band"].unique():
    nf[f"nested_{band}"] = nf.query(f"nested.band=='{band}'")["nested"]

but a direct helper function could be a nice value add. I'm not sure whether it would be best to add it as a NestedSeries function or a NestedFrame function, it would affect an individual NestedSeries so that is a strong argument, but NestedFrame handles a lot of the work of managing nested columns already and may lead to a cleaner workflow:

NestedSeries mockup

split_lcs = nf["nested"].split(by="band") # yields a NestedFrame
nf.join(split_lcs) # or perhaps direct assignment

NestedFrame mockup

nf = nf.split("nested", by="band") # directly handles new column creation on nf

Before submitting
Please check the following:

• I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
• I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
• If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

[delucchi-cmu]

I would request that we be able to specify the distinct values that we would like in the split (e.g. values=['g', 'r', 'i'] for a subset of columns, or values='ugrizy' if you want all bands, even if they're not all present in the nested column) (and see how I snuck in there iteration over the characters in a string as a list of values).

[delucchi-cmu]

Might also be nice to have the option to drop the column you're splitting on. e.g. after splitting on band, you'll know that all values of split_lcs_r.band == 'r'.

[dougbrn]

Somewhat similarly, it may be nice to have the option to drop the original nested column, as duplication of nested data can be an anti-pattern for performance/memory-consumption

[Ebraam-Ashraf]

Hi @dougbrn @delucchi-cmu can I work on this?

I also thought about proposing an output= parameter (output="dict" or output="flat") but looking at the API reference I noticed there are already converters like (to_flat, to_lists, ....)
the same idea for a sort parameter

what do you think about this ? make sense or would it go against the current design?

[dougbrn]

Hi @Ebraam-Ashraf , feel free to make a PR for this!

For output=, I think as you noticed there are existing format converters and so in my opinion this function should stay focused on producing a set of nested structures as output exclusively.

to_flat is a bit of an interesting case, as splitting apart a nested structure and then trying to flatten each structure in succession will lead to excessive data duplication, but I'm not sure if we really want to try to add functionality for that when a user could just as easily flatten up front and then split in a pure pandas workflow.

[hombit]

I agree about at least having an option to specify bands explicitly, it would help with deriving the output schema needed by LSDB/Dask.