|
| 1 | +# LakeBench Codebase Reference |
| 2 | + |
| 3 | +> Quick-reference for Copilot and contributors. Keep this in sync when adding major features. |
| 4 | +
|
| 5 | +--- |
| 6 | + |
| 7 | +## What is LakeBench? |
| 8 | + |
| 9 | +LakeBench is a **Python-native, multi-modal benchmarking framework** for evaluating performance across multiple lakehouse compute engines and ELT scenarios. It supports industry-standard benchmarks (TPC-DS, TPC-H, ClickBench) and a novel ELT-focused benchmark (ELTBench), all installable via `pip`. |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## Project Layout |
| 14 | + |
| 15 | +``` |
| 16 | +src/lakebench/ |
| 17 | +├── __init__.py |
| 18 | +│ |
| 19 | +├── benchmarks/ |
| 20 | +│ ├── base.py # BaseBenchmark ABC — result schema, timing, post_results() |
| 21 | +│ ├── elt_bench/ # ELTBench: load, transform, merge, maintain, query |
| 22 | +│ ├── tpcds/ # TPC-DS: 99 queries, 24 tables |
| 23 | +│ ├── tpch/ # TPC-H: 22 queries, 8 tables |
| 24 | +│ └── clickbench/ # ClickBench: 43 queries on clickstream data |
| 25 | +│ |
| 26 | +├── datagen/ |
| 27 | +│ ├── tpch.py # TPCHDataGenerator (uses tpchgen-rs, ~10x faster than alternatives) |
| 28 | +│ ├── tpcds.py # TPCDSDataGenerator (wraps DuckDB TPC-DS extension) |
| 29 | +│ └── clickbench.py # Downloads dataset from ClickHouse host |
| 30 | +│ |
| 31 | +├── engines/ |
| 32 | +│ ├── base.py # BaseEngine ABC — fsspec, runtime detection, result writing |
| 33 | +│ ├── spark.py # Generic Spark engine |
| 34 | +│ ├── fabric_spark.py # Microsoft Fabric Spark (auto-authenticates via notebookutils) |
| 35 | +│ ├── synapse_spark.py # Azure Synapse Spark |
| 36 | +│ ├── hdi_spark.py # HDInsight Spark |
| 37 | +│ ├── duckdb.py # DuckDB |
| 38 | +│ ├── polars.py # Polars |
| 39 | +│ ├── daft.py # Daft |
| 40 | +│ ├── sail.py # Sail (PySpark-compatible engine) |
| 41 | +│ └── delta_rs.py # Shared DeltaRs write helper (used by non-Spark engines) |
| 42 | +│ |
| 43 | +└── utils/ |
| 44 | + ├── query_utils.py # transpile_and_qualify_query(), get_table_name_from_ddl() |
| 45 | + ├── path_utils.py # abfss_to_https(), to_unix_path() |
| 46 | + └── timer.py # Context-manager timer; stores results for post_results() |
| 47 | +``` |
| 48 | + |
| 49 | +--- |
| 50 | + |
| 51 | +## Core Abstractions |
| 52 | + |
| 53 | +### `BaseEngine` (`engines/base.py`) |
| 54 | +Abstract base for all compute engines. |
| 55 | + |
| 56 | +| Attribute | Description | |
| 57 | +|---|---| |
| 58 | +| `SQLGLOT_DIALECT` | SQLGlot dialect string for auto-transpilation (e.g. `"duckdb"`) | |
| 59 | +| `SUPPORTS_SCHEMA_PREP` | Whether the engine can create an empty schema-defined table before data load | |
| 60 | +| `SUPPORTS_MOUNT_PATH` | Whether the engine can use mount-style URIs (`/mnt/...`) | |
| 61 | +| `TABLE_FORMAT` | Always `'delta'` | |
| 62 | +| `schema_or_working_directory_uri` | Base path where Delta tables are stored | |
| 63 | +| `storage_options` | Dict passed through to DeltaRs / fsspec for cloud auth | |
| 64 | +| `extended_engine_metadata` | Dict of key/value pairs appended to benchmark results | |
| 65 | + |
| 66 | +Key methods: `get_total_cores()`, `get_compute_size()`, `get_job_cost(duration_ms)`, `create_schema_if_not_exists()`, `_append_results_to_delta()`. |
| 67 | + |
| 68 | +Runtime is auto-detected at init via `_detect_runtime()` — returns `"fabric"`, `"synapse"`, `"databricks"`, `"colab"`, or `"local_unknown"`. |
| 69 | + |
| 70 | +### `BaseBenchmark` (`benchmarks/base.py`) |
| 71 | +Abstract base for all benchmarks. |
| 72 | + |
| 73 | +| Attribute | Description | |
| 74 | +|---|---| |
| 75 | +| `BENCHMARK_IMPL_REGISTRY` | `Dict[EngineClass → ImplClass]` — maps engines to optional engine-specific implementations | |
| 76 | +| `RESULT_SCHEMA` | Canonical 21-column result schema (see below) | |
| 77 | +| `VERSION` | Benchmark version string | |
| 78 | + |
| 79 | +The result schema includes: `run_id`, `run_datetime`, `lakebench_version`, `engine`, `engine_version`, `benchmark`, `benchmark_version`, `mode`, `scale_factor`, `scenario`, `total_cores`, `compute_size`, `phase`, `test_item`, `start_datetime`, `duration_ms`, `estimated_retail_job_cost`, `iteration`, `success`, `error_message`, `engine_properties` (MAP), `execution_telemetry` (MAP). |
| 80 | + |
| 81 | +`post_results()` collects timer results → builds result rows → optionally appends to a Delta table via `engine._append_results_to_delta()`. |
| 82 | + |
| 83 | +--- |
| 84 | + |
| 85 | +## Engine & Benchmark Registration |
| 86 | + |
| 87 | +Benchmarks declare engine support via `BENCHMARK_IMPL_REGISTRY`. If an engine uses only shared `BaseEngine` methods, the value is `None`; otherwise it maps to a specialized implementation class. |
| 88 | + |
| 89 | +```python |
| 90 | +# Register a custom engine with an existing benchmark |
| 91 | +from lakebench.benchmarks import TPCDS |
| 92 | +TPCDS.register_engine(MyNewEngine, None) # use shared methods |
| 93 | +TPCDS.register_engine(MyNewEngine, MyTPCDSImpl) # use custom impl class |
| 94 | +``` |
| 95 | + |
| 96 | +To add a new engine, subclass an existing one: |
| 97 | +```python |
| 98 | +from lakebench.engines import BaseEngine |
| 99 | + |
| 100 | +class MyEngine(BaseEngine): |
| 101 | + SQLGLOT_DIALECT = "duckdb" # or whichever dialect applies |
| 102 | + ... |
| 103 | + |
| 104 | +from lakebench.benchmarks.elt_bench import ELTBench |
| 105 | +ELTBench.register_engine(MyEngine, None) |
| 106 | +benchmark = ELTBench(engine=MyEngine(...), ...) |
| 107 | +benchmark.run() |
| 108 | +``` |
| 109 | + |
| 110 | +--- |
| 111 | + |
| 112 | +## Query Resolution Strategy (3-Tier Fallback) |
| 113 | + |
| 114 | +For each query, LakeBench resolves in this order: |
| 115 | + |
| 116 | +1. **Engine-specific override** — `resources/queries/<engine_name>/q14.sql` (rare; e.g. Daft decimal casting) |
| 117 | +2. **Parent engine class override** — `resources/queries/<parent_class>/q14.sql` (rare; e.g. Spark family) |
| 118 | +3. **Canonical + auto-transpilation** — `resources/queries/canonical/q14.sql` transpiled via SQLGlot using the engine's `SQLGLOT_DIALECT` |
| 119 | + |
| 120 | +Tables are automatically qualified with catalog and schema when applicable. To inspect the resolved query: |
| 121 | + |
| 122 | +```python |
| 123 | +benchmark = TPCH(engine=MyEngine(...)) |
| 124 | +print(benchmark._return_query_definition('q14')) |
| 125 | +``` |
| 126 | + |
| 127 | +--- |
| 128 | + |
| 129 | +## Optional Dependency Groups |
| 130 | + |
| 131 | +Install only what you need: |
| 132 | + |
| 133 | +| Extra | Installs | |
| 134 | +|---|---| |
| 135 | +| `duckdb` | `duckdb`, `deltalake`, `pyarrow` | |
| 136 | +| `polars` | `polars`, `deltalake`, `pyarrow` | |
| 137 | +| `daft` | `daft`, `deltalake`, `pyarrow` | |
| 138 | +| `tpcds_datagen` | `duckdb`, `pyarrow` | |
| 139 | +| `tpch_datagen` | `tpchgen-cli` | |
| 140 | +| `sparkmeasure` | `sparkmeasure` | |
| 141 | +| `sail` | `pysail`, `pyspark[connect]`, `deltalake`, `pyarrow` | |
| 142 | + |
| 143 | +```bash |
| 144 | +pip install lakebench[duckdb,polars,tpch_datagen] |
| 145 | +``` |
| 146 | + |
| 147 | +--- |
| 148 | + |
| 149 | +## Supported Runtimes & Storage |
| 150 | + |
| 151 | +**Runtimes**: Local (Windows), Microsoft Fabric, Azure Synapse, HDInsight, Google Colab (experimental) |
| 152 | + |
| 153 | +**Storage**: Local filesystem, OneLake, ADLS Gen2 (Fabric/Synapse/HDInsight), S3 (experimental), GCS (experimental) |
| 154 | + |
| 155 | +**Table format**: Delta Lake only (via `delta-rs` for non-Spark engines) |
| 156 | + |
| 157 | +--- |
| 158 | + |
| 159 | +## Timer (`utils/timer.py`) |
| 160 | + |
| 161 | +`timer` is a context-manager function with a `.results` list attached. Use it inside benchmark `run()` implementations to time each phase/test item: |
| 162 | + |
| 163 | +```python |
| 164 | +with self.timer(phase="load", test_item="q1", engine=self.engine) as t: |
| 165 | + t.execution_telemetry = {"rows": 1000} # optional metadata |
| 166 | + do_work() |
| 167 | + |
| 168 | +self.post_results() # flush timer.results → self.results → optionally Delta |
| 169 | +``` |
| 170 | + |
| 171 | +--- |
| 172 | + |
| 173 | +## Key Conventions |
| 174 | + |
| 175 | +- **All Delta writes for non-Spark engines** go through `engines/delta_rs.py` (`DeltaRs().write_deltalake(...)`). |
| 176 | +- **SQLGlot transpilation** is the default path; engine-specific SQL files are exceptions, not the rule. |
| 177 | +- **`storage_options`** on `BaseEngine` is the single place for cloud auth credentials (bearer token, SAS, etc.). |
| 178 | +- **`extended_engine_metadata`** on `BaseEngine` is the right place to attach runtime-specific metadata that ends up in the `engine_properties` MAP column of results. |
| 179 | +- **TPC-DS / TPC-H spec compliance**: LakeBench intentionally diverges from `spark-sql-perf` to follow the official specs (see `customer.c_last_review_date_sk` and `store.s_tax_percentage` fixes in README). |
| 180 | +- **New benchmarks** should subclass `BaseBenchmark`, define `RESULT_SCHEMA`, `BENCHMARK_IMPL_REGISTRY`, `VERSION`, and implement `run()`. |
0 commit comments