Conversation
…...)" purge_and_archive_results() deleted each of a workunit's results with a separate "delete from result where id=X" query. Collect the IDs while archiving and delete them in one batched "where id in (...)" query instead, split into chunks of 1000 to keep the query well within MAX_QUERY_LEN. Archiving, per-result logging, --dont_delete handling, number_results, and the "don't purge a workunit whose results failed to delete" invariant (BOINC#5694) are unchanged. One behaviour change: because the result deletes are now issued after the whole workunit has been archived rather than one-by-one, a hard kill between archiving and deleting re-archives that workunit's entire result set on restart (previously at most one result); archive entries are keyed by ID so consumers can de-duplicate. Partly addresses BOINC#6735 (workunit and assignment deletes are still one-per-row). Assisted-by: Claude:sonnet-5
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 1 file
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Move the batched result delete into a helper that logs each purged result right after its chunk's DELETE commits, instead of after the whole set. For a workunit with more than 1000 results, if a later chunk's DELETE fails, the "Purged result" lines for the chunks that already succeeded are no longer skipped by the early return. Assisted-by: Claude:sonnet-5
|
Addressed the cubic P3 (per-result "Purged result" log skipped when a later The batched delete now lives in a helper that logs each result immediately after its own chunk's Note: the failing |
|
The number of results per workunit will rarely exceed 2, and never exceed 10. The performance of db_purge has never been an issue, so is this needed at all? |
|
As you wish. Actually, if a project doesn’t even exceed 2 workunits, there won’t usually be any significant improvements. Do we want to close the PR along with the issue, or did you have something else in mind? |
|
I'll close it. We have a lot of stuff going on right now, so I'd prefer to stay away from changes that don't fix something concrete. |
What
db_purgedeleted each of a workunit's results with its owndelete from result where id = Xquery inpurge_and_archive_results(). For a project withtarget_nresults = Nthat'sNround-trips per workunit on top of the one for the workunit itself.This collects the result IDs while archiving and issues a single
delete from result where id in (id1, id2, ...)per workunit instead. TheIN ()list is chunked at 1000 IDs so the generated query stays well withinMAX_QUERY_LEN(262144) — 1000 twenty-digit IDs plus separators is ~21 KB.Partly addresses #6735 — workunit and assignment deletes are still one-per-row; batching those defers deletes across a whole
do_pass()and widens the archived-but-not-deleted window much further, so I left it for a separate change.Unchanged
archive_result()/archive_result_gz()paths.--dont_delete— still logsDidn't purge result [...]per result and deletes nothing (the ID list stays empty).Purged result [id] batch Nis still emitted for every purged result (now grouped after the batched delete rather than interleaved with individual deletes).do_pass()doescontinue, and the workunit + results are retried next pass.--max,--max_wu_per_file,enable_assignment, the workunit delete itself,number_results— untouched.Behaviour change
Result deletes are issued once per workunit, after the workunit has been fully archived, instead of one immediately after each result is archived. If
db_purgeis killed (OOM / power loss) between archiving a workunit's results and the batched delete committing, the restart re-enumerates the workunit and re-archives its entire result set — previously this window was at most one result. No DB inconsistency (the workunit is still never deleted while any of its results survive) and archive entries are keyed by ID for de-duplication, but it's a real change to crash-recovery behaviour in the area #5694 touched.Also: the two
DB_RESULTdeletes previously happened while amysql_store_result-buffered enumeration overresultwas open; they now happen after that enumeration finishes, which is strictly safer.Testing
g++ -fsyntax-onlyagainst the real headers is clean, no new-Wall -Wextrawarnings.delete_ids_from_db()was unit-tested in isolation: 0 IDs -> 0 queries, 3 -> 1, exactly 1000 -> 1, 1001 -> 2, 2500 -> 3; no leading/trailing commas; full-width IDs format correctly.Assisted by Claude (Sonnet 5) per the BOINC AI Assistants Usage Policy; the commit carries the
Assisted-by:trailer. I've read the full diff and verified the behaviour described above.Summary by cubic
Batches result deletions in
db_purgefrom one query per result to a singleWHERE id IN (...)query per workunit, reducing database round-trips. The query is chunked at 1000 IDs to stay withinMAX_QUERY_LEN.Behavior change
--dont_deletehandling, and the per-result log format are unchanged.Written for commit 41a4f11. Summary will update on new commits.