GASP

Accuracy benchmark

We asked two Claude models 24 questions about a SaaS warehouse, one per Core metric, and ran the SQL they wrote. From the table definitions alone they got 60% and 70% of the runs right. With the GASP definition of the metric as well, both 100%.

Schema only
The table definitions and the question
Claude Sonnet 5.5
60%
72 of 120 runs matched the standard
Claude Opus 5.5
70%
84 of 120 runs matched the standard
Schema and the GASP definition
The same, plus the metric’s canonical formula and the fields it reads
Claude Sonnet 5.5
100%
120 of 120 runs matched the standard
Claude Opus 5.5
100%
120 of 120 runs matched the standard

5 runs per question per model, 24 questions. The GASP tools, which compile the definition to SQL, returned the standard's value for 24 of 24.

A run is correct when its SQL returns one row and one column within one part in a million of the value the canonical SQL returns on the same data.

What a miss looks like

The schema-only misses were almost never broken SQL. The queries ran, and gave a number for a metric of the same name computed a different way, or nothing at all. Ask for Customer Acquisition Cost for March 2026 and in 8 of 10 schema-only runs the models divided the costs tagged acquisition by the customers whose acquisition date fell in the month, and answered 3,350. GASP's CAC divides go-to-market cost by new-business events and gives 3,000. Both are a defensible CAC, and nothing in the schema says which one the company reports, so the model chose.

Gross Margin went the same way: the model looked for a cost labelled as cost of goods sold, found nothing and reported a margin of 1.0. Net Revenue Retention was summed from per-subscription start and end columns, which also gives 1.0. Ordway found 48 of 97 public SaaS companies using different NRR formulas. Here two models produced the same disagreements over one warehouse, from the same table definitions.

CAC (Customer Acquisition Cost)

8 of 10 schema-only runs answered 3,350. The standard's value is 3,000.

Claude Opus 5.5, schema only

WITH acq_costs AS (
  SELECT COALESCE(SUM(cost), 0) AS total_cost
  FROM gasp_cost_events
  WHERE incurred_on >= DATE '2026-03-01'
    AND incurred_on <  DATE '2026-04-01'
    AND lower(lifecycle_attribution) LIKE 'acqui%'
),
latest_customers AS (
  SELECT DISTINCT ON (customer_id) customer_id, acquisition_date
  FROM gasp_customers
  ORDER BY customer_id, as_of_date DESC
),
new_customers AS (
  SELECT COUNT(*) AS n
  FROM latest_customers
  WHERE acquisition_date >= DATE '2026-03-01'
    AND acquisition_date <  DATE '2026-04-01'
)
SELECT acq_costs.total_cost / NULLIF(new_customers.n, 0) AS cac
FROM acq_costs, new_customers

The SQL the GASP tools compile

SELECT (CAST((SELECT COALESCE(SUM(cost), 0) FROM gasp_cost_events WHERE incurred_on >= DATE '2026-03-01' AND incurred_on <= DATE '2026-03-31' AND cost_function = 'gtm') AS DOUBLE PRECISION) / NULLIF(((SELECT COUNT(DISTINCT account_ref) FROM gasp_revenue_events WHERE booked_on >= DATE '2026-03-01' AND booked_on <= DATE '2026-03-31' AND event_type = 'new')), 0)) AS cac

Gross Margin

9 of 10 schema-only runs answered 1. The standard's value is 0.9404.

Claude Opus 5.5, schema only

WITH rev AS (
  SELECT SUM(revenue) AS revenue
  FROM gasp_financials
  WHERE period_start >= DATE '2026-03-01'
    AND period_start <  DATE '2026-04-01'
),
cogs AS (
  SELECT SUM(cost) AS cogs
  FROM gasp_cost_events
  WHERE incurred_on >= DATE '2026-03-01'
    AND incurred_on <  DATE '2026-04-01'
    AND lower(cost_type) IN ('cogs', 'cost_of_goods_sold', 'cost of goods sold', 'cost_of_revenue', 'cost of revenue')
)
SELECT (rev.revenue - COALESCE(cogs.cogs, 0)) / NULLIF(rev.revenue, 0) AS gross_margin
FROM rev CROSS JOIN cogs

The SQL the GASP tools compile

SELECT (CAST(((SELECT COALESCE(SUM(revenue), 0) FROM gasp_financials WHERE period_start >= DATE '2026-03-01' AND period_start <= DATE '2026-03-31') - (SELECT COALESCE(SUM(cost), 0) FROM gasp_cost_events WHERE incurred_on >= DATE '2026-03-01' AND incurred_on <= DATE '2026-03-31' AND cost_function = 'infrastructure')) AS DOUBLE PRECISION) / NULLIF((SELECT COALESCE(SUM(revenue), 0) FROM gasp_financials WHERE period_start >= DATE '2026-03-01' AND period_start <= DATE '2026-03-31'), 0)) AS gross_margin

Net Revenue Retention (NRR)

4 of 10 schema-only runs answered 1. The standard's value is 0.9819.

Claude Sonnet 5.5, schema only

WITH latest AS (
  SELECT DISTINCT ON (subscription_id)
         subscription_id, mrr_start, mrr_end
  FROM gasp_subscriptions
  WHERE snapshot_day <= DATE '2026-03-31'
  ORDER BY subscription_id, snapshot_day DESC
)
SELECT SUM(COALESCE(mrr_end, 0)) / NULLIF(SUM(mrr_start), 0) AS nrr
FROM latest
WHERE mrr_start > 0

The SQL the GASP tools compile

SELECT (CAST((((((SELECT COALESCE(SUM(mrr), 0) FROM gasp_subscriptions WHERE snapshot_day = DATE '2026-03-01' AND status = 'active' AND customer_id IN (SELECT customer_id FROM gasp_customers WHERE as_of_date = DATE '2026-03-01' AND lifecycle_status = 'active'))) + ((SELECT COALESCE(SUM(mrr_change), 0) FROM gasp_revenue_events WHERE booked_on >= DATE '2026-03-01' AND booked_on <= DATE '2026-03-31' AND event_type = 'expansion' AND account_ref IN (SELECT customer_id FROM gasp_customers WHERE as_of_date = DATE '2026-03-01' AND lifecycle_status = 'active')))) - ((-(SELECT COALESCE(SUM(mrr_change), 0) FROM gasp_revenue_events WHERE booked_on >= DATE '2026-03-01' AND booked_on <= DATE '2026-03-31' AND event_type = 'contraction' AND account_ref IN (SELECT customer_id FROM gasp_customers WHERE as_of_date = DATE '2026-03-01' AND lifecycle_status = 'active'))))) - ((-(SELECT COALESCE(SUM(mrr_change), 0) FROM gasp_revenue_events WHERE booked_on >= DATE '2026-03-01' AND booked_on <= DATE '2026-03-31' AND event_type = 'churn' AND account_ref IN (SELECT customer_id FROM gasp_customers WHERE as_of_date = DATE '2026-03-01' AND lifecycle_status = 'active'))))) AS DOUBLE PRECISION) / NULLIF(((SELECT COALESCE(SUM(mrr), 0) FROM gasp_subscriptions WHERE snapshot_day = DATE '2026-03-01' AND status = 'active' AND customer_id IN (SELECT customer_id FROM gasp_customers WHERE as_of_date = DATE '2026-03-01' AND lifecycle_status = 'active'))), 0)) AS nrr

Every question

Metric Standard's value Claude Sonnet 5.5, schemaClaude Sonnet 5.5, with GASPClaude Opus 5.5, schemaClaude Opus 5.5, with GASP
MRR (Monthly Recurring Revenue) 18,400 5/55/55/55/5
ARR (Annual Recurring Revenue) 220,800 5/55/55/55/5
ARPA (Average Revenue Per Account) 1226.6667 5/55/55/55/5
MRR Growth Rate 0.1084 5/55/55/55/5
New MRR 2,100 5/55/55/55/5
Expansion MRR 100 0/55/55/55/5
Churned MRR 0 5/55/55/55/5
Contraction MRR 400 3/55/55/55/5
Reactivation MRR 0 5/55/55/55/5
Carry MRR 16,200 0/55/50/55/5
Net New MRR 1,800 5/55/55/55/5
Total Customers 15 4/55/55/55/5
New Customers 2 5/55/55/55/5
Churned Customers 0 5/55/55/55/5
Net Revenue Retention (NRR) 0.9819 1/55/54/55/5
Gross Revenue Retention (GRR) 0.9759 4/55/55/55/5
Logo Churn Rate (Customer Churn) 0 5/55/55/55/5
Revenue Churn Rate 0.0241 5/55/55/55/5
NPS (Net Promoter Score) -25 0/55/50/55/5
CAC (Customer Acquisition Cost) 3,000 0/55/50/55/5
LTV (Customer Lifetime Value) 47870.9480 0/55/50/55/5
LTV:CAC Ratio 15.9570 0/55/50/55/5
CAC Payback Period 2.6007 0/55/50/55/5
Gross Margin 0.9404 0/55/50/55/5

Method

The warehouse is the standard's synthetic SaaS dataset, 53 tables in Postgres, the same fixture the test suite runs every compiled formula against. Its columns carry warehouse names rather than the standard's (the revenue event date is booked_on, the customer key on events is account_ref), so neither condition was handed the answer by a column name. The ground truth for each question is the canonical compiled SQL run on that data.

Three conditions, the same question and the same model in each.

  • Schema only. The CREATE TABLE statements for every table and the question.
  • Schema and the GASP definition. The same, plus the metric's canonical formula, the formula of every metric it references, a note on reading the grammar, and the table and column each field maps to. This is what a semantic layer holds, given to the model as text.
  • The GASP tools. The SQL that generate_query compiles for the metric and the period, run as is. Correct by construction, shown because it is what a client calling the tools receives.

One system prompt for every run, asking for a single SELECT that returns one row and one column. The prompt is in the repository with the runner.

Caveats

  • Synthetic data, one month, 24 questions. Rule of 40 is left out because its one-year lag has no data in March 2026. Real warehouses, more periods and harder questions ("what moved NRR?") are the next rounds.
  • Claude graded Claude. A run on another vendor's models is the obvious follow-up.
  • The GASP condition knows which column each field lives in, because a semantic layer holds that. This is not a test of schema linking on its own.

Reproduce

cd gasp-wiki-mcp && npm run build
npx tsx scripts/benchmark.ts --dry-run
npx tsx scripts/benchmark.ts --runs 5 --models claude-sonnet-5-5,claude-opus-5-5

The runner, the method and every answer with its SQL are in the public repository under docs/benchmark. This run (2026-11-07) is downloadable as JSON and Markdown.

Try searching for:

↑↓navigate↵select