ChunkManager._readFromDatabase crashes on large files: V8 max string length exceeded by single _all_docs request

[pike00]

Summary

ChunkManager._readFromDatabase() issues a single _all_docs({ keys: [...all chunk IDs for one file...], include_docs: true }) request. For large files (e.g. a few hundred MB), the JSON response body exceeds V8's maximum string length (0x1fffffe8 ~= 512MB). PouchDB-HTTP reads the entire body into one string before parsing, so .json() throws Cannot create a string longer than 0x1fffffe8 and the read surfaces as Corrupted document.

Reproduction

1. Sync a file large enough that its chunk count produces a response >512MB (roughly: file size / average chunk size * per-chunk JSON overhead). With 60KB chunks, a ~300MB file can trigger this.
2. Observe the bridge log: the read fails, the file is never written to disk, and the error is opaque ("Corrupted document").

Fix

Batch the keyed _all_docs request. Order is irrelevant here — results land in a Map keyed by _id, and ChunkManager.read() restores the caller's order afterwards. A batch size of 100 keys per request keeps every response well under the ceiling.

 async _readFromDatabase(readIds: Set<DocumentID>, resultMap: Map<DocumentID, false | EntryLeaf>) {
-    const results = await this.database.allDocs({ keys: [...readIds], include_docs: true });
-    for (const row of results.rows) {
-        if ("doc" in row && row.doc) {
-            const chunk = row.doc as EntryLeaf;
-            resultMap.set(chunk._id, chunk);
-            readIds.delete(chunk._id);
-            this.cacheChunk(chunk);
-        } else if (!isMissingError(row)) {
-            throw new LiveSyncError(\`Failed to read chunk ${row.key}\`, { status: 404, cause: getError(row) });
-        }
+    const READ_BATCH_SIZE = 100;
+    const allReadIds = [...readIds];
+    for (let i = 0; i < allReadIds.length; i += READ_BATCH_SIZE) {
+        const batchKeys = allReadIds.slice(i, i + READ_BATCH_SIZE);
+        const results = await this.database.allDocs({ keys: batchKeys, include_docs: true });
+        for (const row of results.rows) {
+            if ("doc" in row && row.doc) {
+                const chunk = row.doc as EntryLeaf;
+                resultMap.set(chunk._id, chunk);
+                readIds.delete(chunk._id);
+                this.cacheChunk(chunk);
+            } else if (!isMissingError(row)) {
+                throw new LiveSyncError(\`Failed to read chunk ${row.key}\`, { status: 404, cause: getError(row) });
+            }
+        }
     }
 }

Impact

Any user syncing files above ~200-300MB (depending on chunk size) hits this crash. The error message is misleading, making it hard to diagnose.

[pike00]

I plan on implementing this. I'll submit a PR once it's ready.