You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We evaluate NL2Semantic2SQL under a dual-track benchmark protocol. The GIS track uses a 20-question benchmark covering four difficulty levels (Easy, Medium, Hard, Robustness) and six spatial query categories (Attribute Filtering, Spatial Measurement, Spatial Join, Spatial Filtering, Centroid Calculation, Proximity Buffer, K-Nearest Neighbors, Spatial Topology, Complex Multi-Step Spatial, Security Rejection, Anti-Illusion, OOM Prevention, Data Tampering Prevention, Schema Enforcement). The warehouse track uses 50 questions sampled from BIRD mini_dev V2, spanning three difficulty levels (simple, moderate, challenging) across 11 database schemas imported into PostgreSQL. Both tracks use execution accuracy (EX) as the primary metric: a prediction is correct if and only if its execution result set matches the gold SQL result set under set-equality comparison with numeric tolerance.
61
+
62
+
For each track, we compare two modes:
63
+
-**Baseline**: Direct LLM generation (Gemini 2.5 Flash) with schema dump only, no semantic grounding.
64
+
-**Full pipeline**: NL2Semantic2SQL with semantic layer resolution, schema-aware context construction, few-shot retrieval (GIS track only), SQL postprocessing, and execution-time self-correction.
65
+
66
+
### 3.2 GIS Track Results
67
+
68
+
Table 1 presents the GIS benchmark results.
69
+
70
+
| Difficulty | N | Baseline EX | Full EX | Delta |
The full pipeline achieves an overall EX of 0.800, a 15-percentage-point improvement over the baseline (0.650). The most striking gain is in the Robustness category, where the baseline scores 0.000 and the full pipeline scores 1.000. Robustness questions test security rejection (refusing DELETE/UPDATE requests), anti-illusion (refusing queries about nonexistent columns), OOM prevention (adding LIMIT for large-table full scans), and data tampering prevention. The baseline LLM generates syntactically valid but semantically dangerous SQL for all five robustness questions, whereas the full pipeline correctly refuses or constrains each one through its postprocessing and safety enforcement layers.
79
+
80
+
For Medium-difficulty spatial queries (spatial measurement, spatial join, spatial filtering, centroid calculation), the full pipeline achieves perfect accuracy (1.000 vs. 0.800 baseline). The improvement is attributable to geometry-aware grounding: the semantic layer injects explicit rules about geography casting, SRID handling, and ROUND(::numeric, N) syntax that the baseline LLM frequently violates.
81
+
82
+
The single remaining failure in the full pipeline is a Hard-difficulty K-Nearest Neighbors question (CQ_GEO_HARD_02), where the model uses `ORDER BY ST_Distance(...)` instead of the PostGIS KNN index operator `<->`, producing a different row ordering. This failure illustrates a known limitation: even with explicit prompt guidance, LLMs sometimes default to more familiar SQL patterns over domain-specific operators.
83
+
84
+
### 3.3 Warehouse Track Results (BIRD)
85
+
86
+
Table 2 presents the BIRD mini_dev benchmark results.
87
+
88
+
| Difficulty | N | Baseline EX | Full EX | Delta |
On the warehouse track, the full pipeline achieves an overall EX of 0.520, marginally below the baseline (0.540). However, the aggregate number masks a divergent pattern across difficulty levels. For simple questions, the full pipeline outperforms the baseline by 8 percentage points (0.720 vs. 0.640), indicating that semantic grounding helps with straightforward filtered and aggregation queries by providing better schema context and column disambiguation. For moderate questions, the full pipeline underperforms by 15.8 percentage points (0.316 vs. 0.474), and for challenging questions, performance is identical.
96
+
97
+
Execution validity (the fraction of generated SQL that executes without error) is 0.960 for baseline and 0.900 for full, indicating that the full pipeline occasionally generates SQL that fails at execution time, likely due to schema-qualified table references or operator mismatches introduced by the grounding layer.
98
+
99
+
### 3.4 Error Analysis
100
+
101
+
We categorize the 24 full-pipeline failures on the BIRD track into three types:
| No SQL generated (agent did not produce SQL) | 3 | 12.5% |
107
+
| Invalid SQL (execution error) | 2 | 8.3% |
108
+
109
+
The dominant failure mode is semantically incorrect SQL that executes successfully but returns wrong results. Manual inspection reveals three recurring patterns:
110
+
111
+
1.**Join path confusion** (8/19): The model selects incorrect join paths between fact and dimension tables. For example, joining `transactions_1k` with `yearmonth` on `CustomerID` when the gold SQL joins through `gasstations` on `GasStationID`. This reflects insufficient understanding of the warehouse schema's entity-relationship structure.
112
+
113
+
2.**Aggregation semantics** (6/19): The model applies COUNT(DISTINCT ...) where the gold SQL uses COUNT(*), or vice versa. It also confuses per-row aggregation with per-group aggregation, particularly in percentage calculations.
114
+
115
+
3.**Date/temporal parsing** (5/19): The BIRD dataset uses non-standard date formats (e.g., `Date` column containing `'201309'` as YYYYMM). The model sometimes applies SUBSTRING-based parsing differently from the gold SQL, or fails to recognize the implicit temporal granularity.
116
+
117
+
These patterns are consistent with the hypothesis that the current semantic layer, designed primarily for GIS schema disambiguation, does not provide sufficient structural metadata for warehouse-style fact/dimension reasoning.
118
+
119
+
### 3.5 Cross-Domain Comparison
120
+
121
+
Figure 1 (to be rendered) summarizes the cross-domain comparison:
122
+
123
+
```
124
+
GIS Track: Baseline 0.650 → Full 0.800 (+0.150, +23.1%)
125
+
BIRD Track: Baseline 0.540 → Full 0.520 (-0.020, -3.7%)
126
+
```
127
+
128
+
The asymmetry is striking. On the GIS track, semantic grounding provides a substantial and consistent advantage. On the warehouse track, the same grounding architecture provides marginal improvement on simple queries but introduces slight degradation on moderate queries. This suggests that the framework's grounding mechanisms are well-calibrated for GIS-specific challenges (geometry types, spatial operators, coordinate systems) but insufficiently adapted for warehouse-specific challenges (entity-relationship navigation, temporal reasoning, multi-table aggregation patterns).
129
+
130
+
## 4. Discussion
131
+
132
+
### 4.1 Why Semantic Grounding Helps in GIS
133
+
134
+
The GIS track results demonstrate that semantic grounding provides substantial value when the query domain involves specialized operators and schema conventions that general-purpose LLMs handle unreliably. Three mechanisms drive the improvement:
135
+
136
+
First, **geometry-aware type injection** ensures the model knows which columns are geometry-bearing, what their SRID is, and when geography casting is required. Without this, the baseline frequently omits `::geography` casts, producing area/distance values in degrees rather than meters.
137
+
138
+
Second, **safety and robustness enforcement** through SQL postprocessing catches dangerous operations (DELETE, UPDATE) and injects LIMIT constraints for large-table full scans. The baseline LLM has no such guardrails and scores 0.000 on all robustness questions.
139
+
140
+
Third, **domain vocabulary grounding** through the semantic layer's hierarchy matching and alias resolution helps the model correctly interpret Chinese domain terminology (e.g., 地类名称, 图斑面积) and map it to the correct quoted column references. The baseline must infer these mappings from raw schema text alone.
141
+
142
+
### 4.2 Why Semantic Grounding Underperforms on Warehouses
143
+
144
+
The warehouse track reveals that the same grounding architecture can introduce slight degradation when applied to domains it was not designed for. We identify three contributing factors:
145
+
146
+
**Factor 1: GIS-centric retrieval bias.** The semantic layer's source ranking algorithm deprioritizes geometry-bearing tables for non-spatial queries, but it does not provide positive signals for warehouse-specific patterns such as star-schema fact/dimension relationships. As a result, candidate table selection for moderate BIRD questions sometimes surfaces irrelevant tables or misses critical join partners.
147
+
148
+
**Factor 2: Absence of entity-relationship metadata.** The current semantic layer stores column-level annotations (domain, aliases, units) but does not encode table-level roles (fact vs. dimension) or explicit join paths. For warehouse queries that require multi-hop joins through intermediate tables, the model must infer the join graph from raw foreign-key structure, which it does less reliably than the baseline that receives the full schema dump without intermediate semantic interpretation.
149
+
150
+
**Factor 3: Few-shot suppression for non-spatial queries.** The framework intentionally suppresses GIS-oriented few-shot examples for non-spatial queries (to avoid polluting warehouse prompts with irrelevant ST_* patterns). However, this means warehouse queries receive no few-shot guidance at all, whereas the baseline benefits from its own implicit pattern matching over the schema dump.
151
+
152
+
### 4.3 The Case for MetricFlow-Style Modeling
153
+
154
+
The error analysis suggests that the primary bottleneck for warehouse performance is not SQL syntax generation but semantic schema navigation. The model can generate syntactically correct PostgreSQL but frequently selects wrong join paths or applies incorrect aggregation granularity. This is precisely the problem that MetricFlow-style semantic modeling addresses: by explicitly declaring entities (join keys), measures (aggregatable facts), and dimensions (descriptive attributes), the system can provide the LLM with pre-computed join paths and aggregation rules rather than requiring it to infer them from raw DDL.
155
+
156
+
We hypothesize that introducing entity/measure/dimension metadata for BIRD schemas would primarily improve moderate-difficulty questions, where the failure mode is join-path confusion rather than SQL syntax errors. This represents a natural next step for the framework's cross-domain generalization.
157
+
158
+
### 4.4 Cross-Lingual Observations
159
+
160
+
Both tracks involve cross-lingual challenges. The GIS track uses Chinese questions over Chinese-named columns (DLMC, BSM, TBMJ), while the BIRD track uses English questions over English-named columns. The framework handles both through its alias-matching mechanism, which supports multilingual synonyms registered in the semantic layer. Preliminary experiments with Chinese translations of BIRD questions (using pre-registered Chinese aliases for 75 tables and 209 columns) show successful table/column resolution, suggesting that the framework's cross-lingual capability is functional but requires systematic evaluation in future work.
161
+
162
+
### 4.5 Limitations
163
+
164
+
Several limitations should be noted. First, the GIS benchmark contains only 20 questions, which limits statistical power for per-category analysis. Second, the BIRD evaluation uses 50 questions from a single database cluster (debit_card_specializing dominates), which may not represent the full diversity of warehouse query patterns. Third, execution-based evaluation treats any result-set mismatch as failure, even when the predicted SQL is semantically equivalent but produces results in a different order or with different numeric precision. Fourth, the framework currently uses a single LLM (Gemini 2.5 Flash) for both baseline and full pipeline; comparative evaluation across model families would strengthen the generalizability claims.
165
+
166
+
### 4.6 Future Work
167
+
168
+
Three directions emerge from this study:
169
+
170
+
1.**MetricFlow integration**: Introduce explicit fact/dimension/entity metadata for non-GIS schemas to provide join-path guidance and aggregation constraints, targeting the moderate-difficulty warehouse failures.
171
+
172
+
2.**Expanded benchmark scale**: Increase BIRD evaluation to 500 questions across all 11 databases, and expand the GIS benchmark to 50+ questions with finer spatial operator coverage.
173
+
174
+
3.**Cross-lingual evaluation**: Systematically evaluate Chinese-to-English and English-to-Chinese query translation using the registered multilingual aliases, producing a cross-lingual text-to-SQL benchmark that spans both GIS and warehouse domains.
0 commit comments