Consider Releasing the Actual Data as a Torrent

[colthouse]

I hope this message finds you well, crt.sh is an amazing project which has become hugely important for my work.

Loosing this treasure to history would be terrible. Could you consider releasing a complete dump to the public for everyone to download? It might also help with prefromance, since there are scripts out to scrape crt.sh, but if those people could just download the db there would be no need to scrape the website, thus reducing load.

I assume the dump would be pretty big so maybe torrenting would be the way to go

Kind regards

[BenBE]

There is direct PostgreSQL access to the database AFAIR, which can help with direct access to largeer(ish) quantities of data too.

The problem with providing the data as a torrent that the dataset itself is multiple hundrets of GiB, which hardly fits on one single disk easily (that's after compression).

I think a better approach would be providing an dump similar to how the Sulight CT logs work, which is mostly static data. From there you can provide rsync or other file access capabilities to static data. For the additional information, an additional set of "leaf" nodes may be provided (for the processed metadata).

[colthouse]

I don't think a couple hundreds GB are a problem to store, after all, nowadays there are USB sticks holding multiple TB. However I believe the torrent would likely get stale quite quickly because people won't seed something this big.

Anything that would allow someone to get access to the whole dumb would be amazing. I don't think it'd be necessary to retain the database scheme.

What do you think? @robstradling