Skip to content

TESTING.md

Latest commit

 

History

History
102 lines (84 loc) · 4.1 KB

TESTING.md

File metadata and controls

102 lines (84 loc) · 4.1 KB
Raw

This repository has extensive tests to ensure correctness. On every checkin, CI compares the generators' output against the original reference implementations (dbgen for TPC-H, the Trino Java port and dsdgen for TPC-DS) using MD5 checksums committed to the repo.

The conformance suites live next to the unified CLI crate:

  • TPC-H: tpcgen-cli/scripts/tpch/ — compare-all-tables.sh checks output against tpcgen-cli/tests/fixtures/tpch/scale-N/MD5SUMS (generated from C dbgen by generate-fixtures.sh, which requires docker or podman).
  • TPC-DS: tpcgen-cli/scripts/tpcds/ — see its README for the Trino (--compat trino) and C dsdgen (--compat c) suites.
./tpcgen-cli/scripts/tpch/compare-all-tables.sh --scale 1
./tpcgen-cli/scripts/tpcds/compare-all-tables.sh --scale 1

Conformance runs in two tiers:

  • Every PR (tpch-conformance.yml, tpcds-conformance.yml): fast MD5-only checks against the committed MD5SUMS — this proves output is byte-identical to what the recorded reference hashes describe, with no container runtime or reference build needed.
  • Every merge to main (full-conformance.yml): the slow pass. It regenerates reference data from the reference implementations themselves (C dbgen in docker, the C dsdgen published dataset, and a fresh Maven build of the Trino Java port), byte-for-byte diffs our output against it, and fails if the committed MD5SUMS have drifted from the living reference.

tpcgen-cli tpch generates exactly the same bytes as the original dbgen program. You can verify this for yourself by using shasum, for example:

$ shasum /tmp/sf10/lineitem.tbl tpch-dbgen/lineitem.tbl
c3f5d0218b6623125887d7021922d1e91da11613  /tmp/sf10/lineitem.tbl
c3f5d0218b6623125887d7021922d1e91da11613  tpch-dbgen/lineitem.tbl

Below we provide the SHA256 checksums for each table generated by dbgen program for several scale factors as it takes a long time to generate the data. These checksums are generated using the code in the [tpch-dbgen] repo and the following commands on MacOS

git clone https://github.com/electrum/tpch-dbgen
cd tpch-dbgen
make
./dbgen -s <scale_factor>
cd ..
shasum tpch-dbgen/*.tbl

You can compare these yourself with the output of tpchgen-cli to verify correctness:

Scale Factor 1

$ shasum tpch-dbgen/*.tbl
bee45e9c240e87d63786324696b1babad18a5e0b  tpch-dbgen/customer.tbl
4802b21c9975d965aa11893214a879df0d8d9e01  tpch-dbgen/lineitem.tbl
f361dffd3d927f5aa64e71cff91458fb5ea1315f  tpch-dbgen/nation.tbl
00d790a08a116feec992cea14272a4f1e5c55925  tpch-dbgen/orders.tbl
06615f7433806c06162af49c7bc27166c64a31d6  tpch-dbgen/part.tbl
db0fcb935904765a9085505b5feb5260752f8bf3  tpch-dbgen/partsupp.tbl
ac61de9604337e791f1bdbcef8f0cdcc21b01514  tpch-dbgen/region.tbl
baad047476a2720d99b707b6f7a7c9e50c170d5a  tpch-dbgen/supplier.tbl

Scale Factor 10

$ shasum tpch-dbgen/*.tbl
b717482bde38c8312cf232e7ca73aab62f5e1eca  tpch-dbgen/customer.tbl
c3f5d0218b6623125887d7021922d1e91da11613  tpch-dbgen/lineitem.tbl
f361dffd3d927f5aa64e71cff91458fb5ea1315f  tpch-dbgen/nation.tbl
dddffc12e235da9cd8d17584dc1eab237654cb0f  tpch-dbgen/orders.tbl
efb2a169b6ce80d8ed3989147e8d70e7f2a38d6c  tpch-dbgen/part.tbl
eae140257dc91ba3b4a929c32ebe3d08f3605618  tpch-dbgen/partsupp.tbl
ac61de9604337e791f1bdbcef8f0cdcc21b01514  tpch-dbgen/region.tbl
42a76ba965916326e52adca1725ed9ee18b8e61b  tpch-dbgen/supplier.tbl

Scale Factor 100

$ shasum tpch-dbgen/*.tbl
18f5a1784d3adbd4662c35ed1d98897a9773a0dc  tpch-dbgen/customer.tbl
d5a3d8a3ccf7bb20d4ff5f01589b5004504907ec  tpch-dbgen/lineitem.tbl
f361dffd3d927f5aa64e71cff91458fb5ea1315f  tpch-dbgen/nation.tbl
dbfe1ff7481a8e1c2deeba00a0b36c3efb093d0b  tpch-dbgen/orders.tbl
f6eb11ed8a2b4d7d70e30b334fc4fc5a28e03ea4  tpch-dbgen/part.tbl
0d9070467528371790f43e1a6463358ddfcd5f62  tpch-dbgen/partsupp.tbl
ac61de9604337e791f1bdbcef8f0cdcc21b01514  tpch-dbgen/region.tbl
48bc62481b58ff96e5e50a70b3892f4d95f7372f  tpch-dbgen/supplier.tbl