[bigquery-firestore-export]: Scalability Issues #863
Description
This extension suffers from scalability issues:
It currently tries to pull all rows from the BQ query into memory, and write them to firestore. This causes OOM and timeout for large queries.
This could be improved by moving to a v2 function (allows for greater memory and longer timeout). We would need to double check all triggers are supported by the function.
A more long-term solution would be to fan-out and push chunks of rows to a Cloud Task queue, though this would take more design work to ensure it's robust and deterministic.
This was the real reason behind firebase/extensions#2544
Expected behaviour:
The extension is able to process queries with millions of rows
Actual behaviour:
The function times out or runs out of memory.
TODO:
- upgrade to v2
- redesign extension to fan out
(we will track these as sub-issues as they are prioritised)