Enterprise Text-to-SQL — fine-tuned Qwen3-8B

Ask a business question in plain English. A QLoRA-fine-tuned Qwen3-8B turns it into PostgreSQL against a 12-table enterprise schema, and the SQL is statically validated before you see it.

Fine-tuning took strict execution accuracy from 43.71 % → 68.43 %, and 70.86 % with a self-correction loop, measured on 453 held-out questions by executing every query against a real database — never by string similarity.

📦 Model · adapter: 43.6 M trainable params (0.917 % of the model) · trained on one free T4 in 7 h 05 m

Question

Try one

Generated SQL

Ablation — 453 held-out questions

Every column scored on the same questions, same harness, same database. Only the named component changes.

metric	1 base	3 fine-tuned	5 FT + repair
strict execution accuracy	43.71 %	68.43 %	70.86 %
projection-tolerant	47.68 %	68.43 %	70.86 %
executable SQL	95.58 %	94.48 %	98.23 %
schema hallucination	0.22 %	3.75 %	0.66 %

Configurations 2 and 4 are schema retrieval, which the first iteration measured as harmful on this 12-table schema. They are listed as not measured on v2 rather than quietly dropped.

By difficulty — base → fine-tuned

tier	n	base	fine-tuned	Δ
easy	126	92.9 %	100.0 %	+7.1
medium	63	69.8 %	96.8 %	+27.0
hard	144	7.6 %	54.9 %	+47.2
very_hard	36	0.0 %	13.9 %	+13.9
enterprise	84	30.9 %	46.4 %	+15.5

The number went up 30 points. The SQL barely improved.

This is the project's second iteration. The first scored 52.10 %, and its failure analysis showed most of that gain was the model learning which columns the benchmark wanted, not better SQL. So the benchmark was rebuilt to state its conventions, a business glossary was added to the prompt, and the model retrained.

Scoring both pipelines on this identical test set:

	iteration 1	iteration 2
strict accuracy	40.18 %	70.86 %
right rows, wrong columns	174	33
genuinely wrong rows	85	84
row-level accuracy	78.6 %	78.1 %

Genuinely wrong answers fell by one query. The model finds the right rows about 78 % of the time and that did not change. The entire +30.68 pp is the benchmark and the prompt finally agreeing on what a correct answer looks like.

That is the most useful thing this project has to say, and it is only visible because the harness scores projection separately instead of reporting one number.

Schema retrieval made things worse — twice

Showing only the retrieved tables cost the base model 1.55 points and the fine-tuned model 9.27. The adapter only ever saw full-schema prompts, so subsets are out of distribution for it; and the damage lands on hard (−20.8) and enterprise (−13.1), the tiers that need joins. Drop a table a join needs and the model invents one.

With 12 tables the whole schema is 5,455 characters. Retrieval solves a problem this database does not have.

Self-correction: two rates, not one


repair success — now executes	11 / 19	57.9 %
repair correctness — returns the gold rows	5 / 19	26.3 %

Six of eleven "successes" turned a crash into a confident wrong answer. One rate would have claimed 57.9 % and hidden that. Repair works when the error names the fix (column "method" does not exist → rename to payment_method) and fails when it names only the symptom (ambiguous column reference).

Training

QLoRA on one free Kaggle T4, one epoch, 135 steps, 7 h 05 m.


base	`Qwen/Qwen3-8B` @ `b968826d9c46`, 4-bit NF4 + double quant
adapter	LoRA r=16, α=32, dropout 0.05, all 7 attention + MLP projections
trainable	43,646,976 of 4,761,498,624 — 0.917 %
loss	completion-only, prompt masked to `-100`
supervised token share	2.41 %
train / eval loss	0.083 / 0.184 → 0.143 → 0.145

Iteration 1 ended at train 0.006 / eval 0.335 — it had memorised its templates. This one fits the training set less hard and generalises more than twice as well.

Why completion-only loss is the detail that matters. Each example is ~1,814 prompt tokens and ~45 completion tokens, and the prompt is ~97 % schema, byte-identical across all 2,175 examples. Training on the full sequence sends ~97 % of the gradient into memorising a schema the model is handed at inference — and the loss curve looks excellent throughout, because predicting a constant is easy.

How it is evaluated

By execution, never string similarity. Every prediction runs against PostgreSQL and its result set is fingerprinted. These are different strings and both count as correct:

SELECT COUNT(*)           FROM customers WHERE country = 'India';
SELECT COUNT(customer_id) FROM customers WHERE country = 'India';

Guardrails: splits are leakage-free by template group — 74 train templates, 48 test templates, zero overlap — the prompt and schema fingerprints are asserted at every stage, the GPU host that generates predictions never receives gold SQL, and an oracle self-test (gold SQL through the scorer) must return 100 % before any real number is trusted.

Reproducibility


prompt fingerprint	`4e72cc5f722ce436`
schema fingerprint	`d03619e711661bc5`
base model revision	`b968826d9c46`

Limitations

Schema-specific. It learned this database's conventions — that is most of the measured gain. Accuracy elsewhere will be far lower.
Roughly half its answers are still wrong.
Test questions are template-generated, from the same generator as training. Real user phrasing is untested — the biggest caveat on the number.
One epoch, one seed, one run. No variance estimate.
Live Database Execution: Queries are executed live against a read-only Neon DB instance when credentials are configured.