Brief
SQLMorph mutates text-to-SQL queries to expose join and wording failures, and scores partial answers
A new arXiv paper describes SQLMorph, a framework that generates harder text-to-SQL evaluation sets by mutating queries and adds execution-level precision and recall metrics. The evidence is the paper's own description and experiments; no code release, licence, pricing or availability is stated.
SQLMorph targets a stated problem: public text-to-SQL benchmarks are said to miss enterprise schema complexity, while private evaluation sets are costly and nondeterministic to build, making results hard to reproduce. The framework uses two generators. Join Query Expansion adds valid joins to raise structural complexity; Textual Query Augmentation applies controlled natural-language perturbations, to test robustness to wording.
The paper reports that applying JQE to state-of-the-art systems increases query coverage and exposes accuracy degradation as join counts grow, and that heavy abbreviation can cut accuracy by up to 17%. It also proposes Execution Precision and Execution Recall, combined as F1, in place of binary Execution Accuracy. The authors state these metrics expose over- and under-prediction that binary measures obscure.
Our reading
The useful part for this desk is the harness idea, not a single benchmark number: generating targeted choke points plus precision and recall turns a pass/fail score into a debugging signal about which component failed — join handling or wording brittleness. Teams running text-to-SQL over their own schemas, and anyone maintaining an evaluation harness, should care.
What to do or watch
Watch for a released implementation or dataset; the evidence does not establish one. Meanwhile the bounded step is to check whether your own harness scores partial answers with precision and recall, not only exact-match execution.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- SQLMorph is a framework for Text-to-SQL evaluation via query mutation that introduces Join Query Expansion and Textual Query Augmentation.
- Join Query Expansion systematically increases structural complexity through valid join additions.
- Textual Query Augmentation generates controlled natural language perturbations to assess robustness to linguistic variation.
- TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%.
- SQLMorph defines Execution Precision (EXP) and Execution Recall (EXR) and combines them via F1 for unified scoring.
Sources
- arXivText stored 16 September 2026
How this story was checked. Written from the 1 page listed above, stored 16 September 2026; claims checked against that stored text on 16 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.