get_or_create in load_from_grds() creates duplicate missions when thumbnail_filename or notes_filename change between runs

[ksalamy]

In scripts/load.py line 1448, Mission.objects.get_or_create() includes thumbnail_filename, notes_filename, grid_bounds, and directory as lookup fields alongside name:

mission, created = Mission.objects.get_or_create(
    name=os.path.dirname(fp).replace(MBARI_DIR, ""),
    grid_bounds=grid_bounds,
    notes_filename=notes_filename,
    thumbnail_filename=thumbnail_filename,
    directory=os.path.dirname(fp),
)

Because all fields are part of the lookup key, if any field changes between bootstrap runs (e.g. thumbnail_filename is empty on the first run because locate hasn't indexed the file yet, then non-empty on a subsequent run), Django creates a second Mission with the same name rather than updating the existing one.

This causes cascading failures: the --spreadsheets phase calls Mission.objects.get(name=...) which crashes with MultipleObjectsReturned when duplicates exist.

Fix: Move all fields except name into a defaults={} dict, and use update_or_create so re-runs safely update rather than duplicate:

mission, created = Mission.objects.update_or_create(
    name=os.path.dirname(fp).replace(MBARI_DIR, ""),
    defaults={
        "grid_bounds": grid_bounds,
        "notes_filename": notes_filename,
        "thumbnail_filename": thumbnail_filename,
        "directory": os.path.dirname(fp),
    }
)

[ksalamy]

Hi @MBARIMike

I am just following up on this issue with findings from the full sync that ran on smdb-dev overnight (2026-08-05 16:39 → 2026-08-06 04:14 PDT).

The sync confirmed duplicates: After the sync completed, I ran a cleanup script (dedup_missions.py) and found 6 duplicate Mission records created during this run:

MappingAUVOps2014/20140424m1
MappingAUVOps2015/20150330m1
OceanImaging2011/20111003VentanaPortugueseLedge/multibeam
Packard/2025/DP2025_01/QAT
Packard/2025/DP2025_01/RogueCanyonLeg1
Packard/2025/DP2025_01/RogueCanyonLeg2

The dedup_missions.py script removed all 6 on smdb-dev by keeping the original (first-created) record for each mission name and migrating any M2M relationships to it before deleting the duplicate.

Potential Interim workaround: smdb/dedup_missions.py has been committed locally ONLY on smdb-dev (commit 773bbd8) and is called at the end of the weekly full sync. It will catch and clean up any duplicates post-run on smdb-dev.

Proposed fix (as described above): Changing get_or_create to update_or_create with defaults={} would prevent duplicates from being created in the first place. Checking in to ask: Is there a reason the current get_or_create signature was written this way? Is the update_or_create approach a safe fix for production?

[MBARIMike]

Yes, the "update_or_create with defaults={}" would be safe for production. I did not know about this Django pattern when the code was initially written. The mission.name serves as a primary key and the other fields should not have been used the way they were in the original get_or_create().