Docker container terminates due to 'Invalid string length' error

[SonnyRR]

Hi, I'm trying to insert around 7 million rows of fake data inside of a table storage resource. The rows consists only of several columns and they don't include any complex structures. I'm inserting the data through transactions, each containing 100 Add actions. The data is generated through Bogus & I've set sensible values for the PartitionKey & RowKey columns. Azurite is spun up as a Docker container w/ persistent storage.

Just below the 1 million row count the container terminates due to a runtime error from /opt/azurite/node_modules/lokijs/src/lokijs.js:1331:

/opt/azurite/node_modules/lokijs/src/lokijs.js:1331
        case "normal": return JSON.stringify(this, this.serializeReplacer);
                                   ^
 
RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at Loki.serialize (/opt/azurite/node_modules/lokijs/src/lokijs.js:1331:36)
    at Loki.saveDatabaseInternal (/opt/azurite/node_modules/lokijs/src/lokijs.js:2780:66)
    at Loki.saveDatabase (/opt/azurite/node_modules/lokijs/src/lokijs.js:2822:12)
    at Timeout.autosaveHandleInterval [as _onTimeout] (/opt/azurite/node_modules/lokijs/src/lokijs.js:2923:16)
    at listOnTimeout (node:internal/timers:594:17)
    at process.processTimers (node:internal/timers:529:7)

Going through the container logs I'm not seeing any other errors or warnings, all previous batches resolve with HTTP 202 status codes.

It doesn't seem to happen at random, it's always around the 930k record mark, up until that it's reasonably fast. I've also looked at the recent issues in https://github.com/techfort/LokiJS, but I've yet to find any similar experiences.

I've also verified that I'm not using any in-memory storage, so I'm not sure what causes this dependency to error out. Also, I've reviewed the data in the transaction batch which causes this, there's nothing out of the ordinary. Artificial delays also don't resolve this issue.

[SonnyRR]

I've also discovered that once this error is thrown I cannot insert any other data in different/new tables. The json table blob is around 500mb. I haven't seen any mention of a storage limit for the table storage option, which is in preview, did I miss something around the limitations ?

[blueww]

Hi @SonnyRR,

Thanks for raising this issue!

However, Azurite is just a emulator but not a scalable storage service. We can't commit for large data size.

How about using the product storage table service for your scenario?

[SonnyRR]

Hi @blueww, indeed that's what we end up doing. We were trying to benchmark read operations with couple of different PK and RowKey values. Thats why the need to insert large quantity of fake data. Even though this is not the fully fledged product we were hoping to give us a rough idea of the performance hit if we changed the PK and RK values.