Metadata mismatch found when x-matching ps1 and ztf lc

[hombit]

Bug report

import lsdb
from dask.distributed import Client
from upath import UPath

search_filter = lsdb.ConeSearch(280.0, 60.0, radius_arcsec=360.0)
ps1 = lsdb.read_hats(
    UPath('s3://stpubdata/panstarrs/ps1/public/hats/otmo', anon=True),
    columns=['objName', 'raMean', 'decMean'],
    search_filter=search_filter,
)

ztf_lc_columns = ['hmjd', 'mag', 'magerr', 'catflags', 'clrcoeff']
ztf = lsdb.read_hats(
    'https://data.lsdb.io/hats/ztf_dr22/ztf_lc',
    margin_cache='https://data.lsdb.io/hats/ztf_dr22/ztf_lc_10arcs',
    search_filter=search_filter,
    columns=['objectid', 'objra', 'objdec'] + ztf_lc_columns,
)
ztf = ztf.nest_lists(
    base_columns=[col for col in ztf.columns if col not in ztf_lc_columns],
    list_columns=ztf_lc_columns,
    name="lc",
)

catalog = ps1.crossmatch(
    ztf,
    radius_arcsec=0.5,
    n_neighbors=20,
    suffixes=["_ps1", ""]
)

with Client() as client:
    ndf = catalog.compute()

2025-02-21 14:03:09,503 - distributed.worker - ERROR - Compute Failed
Key:       ('from-delayed-f33e3e7ceda6b21e9a5aeb397598d6da', 1)
State:     executing
Task:  <Task ('from-delayed-f33e3e7ceda6b21e9a5aeb397598d6da', 1) check_meta(...)>
Exception: 'ValueError("Metadata mismatch found in `from_delayed`.\\n\\nPartition type: `nested_pandas.nestedframe.core.NestedFrame`\\n+--------+---------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\\n| Column | Found                                                                                       | Expected                                                                                                                                                               |\\n+--------+---------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\\n| \'lc\'   | nested<hmjd: [double], mag: [float], magerr: [float], catflags: [int32], clrcoeff: [float]> | nested<hmjd: [list<element: double>], mag: [list<element: float>], magerr: [list<element: float>], catflags: [list<element: int32>], clrcoeff: [list<element: float>]> |\\n+--------+---------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+")'

Before submitting
Please check the following:

• I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
• I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
• If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

[hombit]

Fixed in nested-dask v0.3.3 by lincc-frameworks/nested-dask#83