Skip to content

Conversation

@refi93
Copy link

@refi93 refi93 commented Jul 19, 2023

Motivation: When I was trying to upload bigger data to polybase it was failing with a RangeError: Maximum call stack size exceeded which turned out to be caused by the encodeBase64() function which apparently wasn't able to handle data bigger than ~100kB.

This PR reimplements the encodeBase64() in a safer way which works with bigger arrays without issues, even beyond 1MB - not sure if there's an upper bound (in practice I expect that code to be too slow/blocking for files larger than several MB), I guess a proper solution for bigger data would be to stream/chunk it somehow which would require designing a whole new API call, nevertheless, this solution is an improvement respective to the current implementation and it was already enough for my usecase

more info on the issue e.g. here: mathiasbynens/base64#13

How to test: Try creating a colletion with a buffer field containing a data item of more than 100kB:

const pb = new Polybase({
  defaultNamespace: 'mynamespace',
})
await pb
  .collection('MyCollection')
  .create([Array(1_000_000).fill(0)]) // on main this would fail with "RangeError: Maximum call stack size exceeded"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant