How to support files larger than 2GB ?

[Charltsing]

File.radAllbytes() does not support 2GB+files

[EliotJones]

It should be possible to pass a Stream to Open, e.g. by using https://learn.microsoft.com/en-us/dotnet/api/system.io.file.openread?view=net-10.0

[Charltsing]

1. Using file streams and long data types
2. Read page objects (text, images, font) on demand
3. Lazy page loading and parsing
4. Use global page cache to constrain memory usage

[BobLd]

@Charltsing unsure what you mean here, did you try using Stream instead of byte array?

[Charltsing]

@Charltsing unsure what you mean here, did you try using Stream instead of byte array?不太明白你的意思，你试过用 Stream 代替字节数组吗？

Using streams instead of byte arrays can solve the problem of opening large files, but a series of subsequent tasks still need to be addressed. Most importantly, on-demand loading and a global cache are required to control memory usage.

This is likely to be a complex undertaking.

[Charltsing]

Today I came across a PDF document with 114,521,072 lines of operators in a page content stream (230M). Processing such documents requires the use of a chunked list.

ChunkedList.zip

[BobLd]

@Charltsing what happens if you don't use the ChunkedList? What error do you get?

Do you have a link where we could get this file?

[Charltsing]

@Charltsing what happens if you don't use the ChunkedList? What error do you get?

Do you have a link where we could get this file?

I tried using multiple libraries to render this file, but they all failed. I could only use iTextSharp to read its content stream.
Adobe Reader can open it, but rendering the pages takes a very long long long time

By the way, this PDF might cause Windows Defender to time out, so you may need to disable it.

This is a split-archive PDF. In order to upload it, I appended .zip to the filenames. You will need to remove that extension before extracting.

billion.zip

billion.z01.zip

billion.z02.zip

billion.z03.zip

billion.z04.zip

billion.z05.zip

billion.z06.zip

billion.z07.zip

I tried loading it with a List, but it consumed too much memory (another key issue is that the time for parsing operators is so long that I don't want to wait.). A ChunkedList would manage/allocation memory better, although you may still require more RAM (128M or more ?).

This document is a special case, perhaps we don't need to support it.Rendering billions of instructions at once may not be realistic for average users