Appendix 2: The Bugs of CQL

The human-found era in the public history of CG/SQL

Study date: October 13, 2026; scope revised November 2, 2026

Discovery-era window: October 7, 2020 through May 14, 2026

Analyzed endpoint: 8793b00f

Repository: ricomariani/CG-SQL-author

How to read this appendix

This appendix covers the human-found era: failures reported by users, observed in ordinary use, or found during human-directed feature development. The AI-assisted discovery period beginning May 15, 2026 is covered separately in The AI-Found Bugs of CQL .

During the era described here, CQL was getting regular daily use from hundreds of developers at Meta. It still gets that use, but not with this open-source version. The core compiler and schema-upgrade paths got a great deal of use. QIDs and the two public blob serialization facilities were lightly used.

There were no mainline commits between January 28 and May 15, so the endpoint is the last public mainline state before the discovery boundary. Source and test measurements below have been recomputed there. I removed the earlier fix counts and correlations that mixed the two eras.

The public history begins with an already mature compiler. It does not contain CQL’s earlier internal development, when the architecture and defect profile would have been substantially different. This appendix is about the maintenance history of the mature, open-source codebase.

How I made this appendix

I used an AI coding agent to mine the complete public Git history, screen candidate fix commits, measure the source and test corpus, retrieve historical patches, compare them with current code, and produce the first draft. It did a lot of the raw work, so I did not personally research and write every sentence from scratch.

I deliberately left the agent room to read the history and reach its own conclusions because the independent reading was useful. Then I heavily curated it. I added my own domain knowledge, supplied operational and architectural context that Git cannot provide, checked the examples against the source and historical patches, and corrected bad assumptions.

That changed several conclusions:

• I corrected the interpretation of CQL’s fail-fast AST and tree-shape development model;
• I separated observed operational bugs from development-time failures, CI-prevented pushes, and changes that did not represent field incidents;
• I added what I remember from sustained production use: roughly 12 real bugs per year;
• I corrected an overbroad interpretation of compiler global state and reentrancy;
• I distinguished incomplete first fixes from additional field failures;
• I audited every code sample and expanded examples that did not show enough setup, operation, or test evidence.

The agent did a lot of the research and drafting. I checked and rewrote the result, and I own it.

Use this as an engineering study. There is no authoritative incident ledger behind these numbers, so read them with the limits stated below.

Summary

The bugs cluster where a fact has to survive several compiler phases, representations, backends, or ownership conventions. Individual operations usually work; keeping them in agreement is harder.

I came away with seven ideas that organize the history:

1. Cross-phase semantic invariants - nullability, sensitivity, error state, identifier spelling, and ownership must survive semantic analysis, rewriting, SQL generation, and multiple backends. Novel AST kinds are a special case: the normal path is deliberately fail-fast, so missing handlers are forced out during development.
2. Schema evolution needs an observable-state oracle - create order, drop order, dependency order, declaration order, and quoted-name handling are related but not interchangeable. More importantly, version CRCs and housekeeping cannot be allowed to overrule the physical schema they claim to describe.
3. Global compiler state and C ownership - the intentionally stateful implementation is concise and effective, but nested compilation work, repeated invocation, aliasing, and +1 reference ownership require explicit protocols.
4. Representation boundaries and changing contracts - growable buffers, serialized blobs, offset tables, platform integer widths, and C aliasing/alignment rules produce a disproportionate share of low-level fixes. The two binary serializers had very different starting requirements, which explains why their histories also look different.
5. Rationale comments as design memory - non-obvious transformations, invariants, constraints, ordering rules, ownership exceptions, and warnings against tempting changes preserve the WHY that the C type system cannot express.
6. Coverage-driven fail-fast development plus independent tests - 100% coverage and fail-fast contracts force new tree shapes through every stage that can see them. Golden output is still only a consistency oracle, so the hardest remaining bugs survived when the compiler and its expected outputs shared the same mistaken assumption.
7. Outputs and examples can become infrastructure - JSON became a stable intermediate contract, and some Python examples eventually replaced large C backends. Their required standard changed when their job changed.

CQL is also a useful counterexample to a fair amount of conventional C advice. It has a 27,000-line semantic-analysis file, extensive file-static state, large macros, and several very long functions. At the same time, many handlers are small, naming is exceptionally regular, the comment corpus is unusually large and purposeful, contracts are dense, arenas simplify allocation lifetimes, and the test suite compiles and executes generated code through several independent paths.

The naming, contracts, comments, and tests make this arrangement practical.

Scope and method

I examined only the public Git history. The root commit is:

At the human-era endpoint, I found:

These are history counts, not bug counts. Some commits called “fix” are cleanup. Some real fixes are titled “Handle”, “Coverage”, or simply state a new rule. Rapid revert/reapply chains can record one real bug plus a bad attempted fix that CI prevented from progressing. I use commit messages to locate evidence, not as an exact defect ledger.

I then inspected representative commits by patch, affected production files, tests, and generated-reference changes. The safety boundary is deliberate: I did not search for new defects, try to trigger unknown failures, or assess exploitability.

The public defect profile

Activity by year

Commit messages can be misleading. An important 2025 reentrancy fix, for example, was hidden in a commit whose subject began with coverage work. Activity is useful context, but neither a commit count nor a keyword screen establishes how many defects users experienced.

CI-caught changes are bugs prevented

Git history includes bad pushes that CI caught before they went any further.

The cql_multinull object-handling chain is the cleanest example:

1. f97fb8be74dc attempted to fix a real missing runtime case.
2. 5b70a230f899 reverted that attempt 14 minutes later.
3. 64d4305c205e applied the properly constrained fix the next day.

What happened is simple: I pushed the first fix too aggressively without running the full suite. CI caught it immediately and forced the correction. The original missing object case was a real bug. The bad attempted fix was not a second operational bug; it was a bug prevented by the test and delivery system.

This distinction matters to the statistics. A raw fix-commit count can assign two fixes, a revert, and a reapplication to one escaped defect. Rapid CI-driven follow-ups should be collapsed into the originating issue and credited to the prevention machinery, not counted as additional field failures.

Where fixes occurred

The cases below span semantic analysis, code generation, runtime behavior, SQL rewriting, schema migration, parsing, and downstream tools. They often cross phase boundaries, so even a manually classified fix is not necessarily one subsystem’s bug.

There is one CQL-specific twist. Traditional optimizing compilers often find many wrong-code bugs in optimization passes. CQL has extensive rewrites and code generation, without a GCC/LLVM-style tower of aggressive machine-level optimization passes. CQL’s specific concentration is in semantics-preserving rewrites, schema migration, backend/runtime agreement, generated ownership, and SQLite-facing output–not machine-level optimization.

Is CQL reliable?

Yes.

My recollection from the years of constant daily use by hundreds of developers at Meta is about 12 real bugs per year. That is operational context from the maintainer, not a number derived by counting public fix commits.

The historical cases fit the character of that experience, but Git cannot prove the exact incident rate. That would require a complete issue and incident ledger. In particular, findings from a different discovery process must not be added to this era and divided by its years of operational use.

Even if I set my recollection aside, I judge that rate to be exceptionally low. The endpoint has roughly 83,000 lines of core C and headers, with a grammar, multiple generators, a runtime, schema migration, several representation formats, and years of sustained production use. About one observed bug a month across that surface is a very strong result.

I do not have a statistically calibrated comparison population, so I cannot attach a percentile to “exceptionally low.” It is my engineering judgment, not a measured statistic.

So yes, the compiler works. The low operational bug rate, fail-fast covered tree walking, rationale-heavy comments and preserved design history, layered execution tests, and long production use all point in the same direction.

Ranked findings

1. Fail-fast tree walking works; cross-phase semantics are harder

The largest defect family cuts across labels such as “parser bug”, “semantic bug”, and “code-generator bug.” The common ingredient is a fact represented in more than one place.

I would separate AST kind dispatch from the rest because CQL is deliberately designed to blow up on a novel tree shape. The semantic expression dispatcher does this:

cql_noexport void sem_expr(ast_node *ast) {
  // These are all the expressions there are, we have to find it in this table
  // or else someone added a new expression type and it isn't supported yet.
  symtab_entry *entry = symtab_find(exprs, ast->type);
  Invariant(entry);
  sem_expr_dispatch *disp = (sem_expr_dispatch*)entry->val;
  disp->func(ast, disp->str);
}

The C expression generator does the same thing after semantic analysis:

// These are all the expressions there are, we have to find it in this table
// or else someone added a new expression type and it isn't supported yet.
symtab_entry *entry = symtab_find(cg_exprs, expr->type);
Invariant(entry);
cg_expr_dispatch *disp = (cg_expr_dispatch*)entry->val;
disp->func(expr, disp->str, is_null, value, pri, disp->pri_new);

The extraction macros then check the detailed child shape at each step. A grammar change that adds or changes a reachable tree shape should therefore produce an immediate assertion at the first phase that has not been updated.

The 100% coverage norm is what makes this useful rather than aspirational. A new parser production is exercised, semantic analysis fails until the new shape is handled, and any shape that survives semantic analysis fails again in codegen until that stage handles it. Development becomes a sequence of concrete failures. You do not have to forecast every registration in advance because the compiler tells you where the next one is.

That works extremely well. In the normal covered path, a missing handler cannot quietly persist.

The persistent examples are the facts that can be handled locally and still be wrong globally:

• a semantic error reported but not recorded in the enclosing node;
• nullability or sensitivity copied from only one operand;
• a quoted identifier handled by static SQL but not dynamic upgrade SQL;
• a rewritten expression preserving its value but not its ownership or type wrapper;
• a logical table name renamed in ordinary SQL expressions but not dot expressions.

The missing registrations were mostly deliberate no-op cases

In May 2022, declare_interface_stmt was a valid AST kind, but the C backend had no no-op registration for it:

   NO_OP_STMT_INIT(declare_const_stmt);
   NO_OP_STMT_INIT(declare_named_type);
   NO_OP_STMT_INIT(declare_proc_no_check_stmt);
   NO_OP_STMT_INIT(declare_select_func_no_check_stmt);
+  NO_OP_STMT_INIT(declare_interface_stmt);

Commit: 3a9f41f21a266ab5377d23d9df3594c9a2a2f9c7

At that point, statement code generation immediately required a registered handler:

symtab_entry *entry = symtab_find(cg_stmts, stmt->type);
Contract(entry);

That commit changed only the registration line; it added no focused regression test. What supports the interpretation is the fail-fast lookup, the crash-specific commit subject, and the missing no-op registration itself.

The same narrow family reappeared in December 2024, when two macro-definition AST kinds needed no-op registration in both C and Lua backends:

diff --git a/sources/cg_c.c b/sources/cg_c.c
@@
   NO_OP_STMT_INIT(expr_macro_def);
   NO_OP_STMT_INIT(stmt_list_macro_def);
   NO_OP_STMT_INIT(query_parts_macro_def);
   NO_OP_STMT_INIT(cte_tables_macro_def);
+  NO_OP_STMT_INIT(select_expr_macro_def);
+  NO_OP_STMT_INIT(select_core_macro_def);

diff --git a/sources/cg_lua.c b/sources/cg_lua.c
@@
   LUA_NO_OP_STMT_INIT(expr_macro_def);
   LUA_NO_OP_STMT_INIT(stmt_list_macro_def);
   LUA_NO_OP_STMT_INIT(query_parts_macro_def);
   LUA_NO_OP_STMT_INIT(cte_tables_macro_def);
+  LUA_NO_OP_STMT_INIT(select_expr_macro_def);
+  LUA_NO_OP_STMT_INIT(select_core_macro_def);

Commit: 868ec5b7

This commit likewise added no dedicated regression test. These definitions had already served their semantic purpose, so the correct backend evidence was the explicit no-op classification rather than generated output.

These nodes were deliberately ignored by codegen. They are an awkward case for the fail-fast scheme: the correct handler is NO_OP, so there may be no ordinary generated output to make the omission obvious except the assertion itself.

I would not mechanically require every parser AST kind in every backend. Semantic analysis rewrites the tree, some parser shapes disappear, and some semantic-only declarations should never reach code generation as active work. Each phase intentionally sees a different set of shapes.

So I read the history more narrowly than “distributed dispatch tables drift.” Normal reachable shapes fail fast and coverage forces them out. The residual registration misses are mainly intentional no-op classifications or other cases where a phase is supposed to ignore a shape.

Error reporting and error state were separate obligations

In July 2021, assignment compatibility correctly detected a lossy conversion, but the error was attached to the wrong expression and the enclosing update node did not receive error state:

-  if (!sem_verify_assignment(ast, sem_type_left, sem_type_right, left->sem->name)) {
+  if (!sem_verify_assignment(right, sem_type_left, sem_type_right, left->sem->name)) {
+    record_error(ast);
     return;
   }

The focused test checks both the diagnostic and propagation through the update subtree:

-- TEST: update with loss of precision
-- + {update_stmt}: err
-- + {update_list}: err
-- + {update_entry}: err
-- + Error % lossy conversion from type 'LONG_INT' in 1L
-- +1 Error
update foo set id = 1L where id = 2;

Commit: 5084c83c1f751dcfd436253c88bad3fbf1bb85f7

The same distinction surfaced again in December 2023:

   // check if length of columns to update and provided values match
   for ( ; name_list && insert_list; name_list = name_list->right, insert_list = insert_list->right) {}
   if (name_list || insert_list) {
     report_error(ast, "CQL0157: count of columns differs from count of values", NULL);
+    record_error(ast);
+    return;
   }

Commit: 7933769f72b4a7d18e73895eac150fa200378c76

The second bug hid inside batched negative tests. Before the repair, four invalid constructs lived in one procedure and the aggregate +4 error: count could not prove that each construct produced its own error:

-proc test_update_from_shape_errors(like update_test_1)
-begin
-  -- Swapped ordering of columns lead to incompatible types.
-  update update_test_1
-  set (name, id) = (from arguments);
-
-  -- Use of non existent shape in values
-  update update_test_1 set (id, name) = (from cursor_not_exist);
-
-  -- Use of non existent shape in column spec
-  update update_test_1 set (like cursor_not_exist) = (from arguments);
-
-  -- Count of columns differ from count of values
-  update update_test_1 set (id) = (from arguments);
-end;

The repaired tests isolate all four operations and check each exact count:

-- TEST: update from_shape sugar error handling, type mismatch
-- + {update_stmt}: err
-- + {update_list}: err
-- + {update_entry}: err
-- + {dot}: err
-- + {name ARGUMENTS}
-- + {name id}
-- * error: % incompatible types in expression 'name'
-- +1 error:
proc test_update_from_shape_errors0(like update_test_1)
begin
  update update_test_1
  set (name, id) = (from arguments);
end;

-- TEST: update from_shape sugar error handling, FROM invalid
-- + {update_stmt}: err
-- + {insert_list}: err
-- + {name cursor_not_exist}: err
-- * error: % name not found 'cursor_not_exist'
-- +1 error:
proc test_update_from_shape_errors1(like update_test_1)
begin
  update update_test_1 set (id, name) = (from cursor_not_exist);
end;

-- TEST: update from_shape sugar error handling invalid like shape
-- + {update_stmt}: err
-- + {columns_values}: err
-- + {shape_def}: err
-- + {name cursor_not_exist}: err
-- * error: % must be a cursor, proc, table, or view 'cursor_not_exist'
-- +1 error:
proc test_update_from_shape_errors2(like update_test_1)
begin
  update update_test_1 set (like cursor_not_exist) = (from arguments);
end;

-- TEST: update from_shape sugar error handling, missing count
-- + SET (id, name) = (ARGUMENTS.id);
-- + {update_stmt}: err
-- * error: % count of columns differs from count of values
-- +1 error:
proc test_update_from_shape_errors3(id int!)
begin
  update update_test_1 set (id, name) = (from arguments);
end;

We had two operations, report_error and record_error, that callers had to remember to pair. We kept making the same mistake.

Combining semantic flags

The 2021 substr fix combined nullability and sensitivity from all arguments instead of copying flags from only the text operand:

-  // This will give us the same type nullability and sensitivity as the original string
-  sem_t flags = sem_type & (SEM_TYPE_NOTNULL | SEM_TYPE_SENSITIVE);
+  // The result is nonnull if all arguments are nonnull, and sensitive if any
+  // arguments are sensitive.
+  sem_t flags = combine_flags(arg1->sem->sem_type, arg2->sem->sem_type);
+  if (arg3) {
+    flags = combine_flags(flags, arg3->sem->sem_type);
+  }

Commit: 2020413560de1f84369d15dd5f793bde8258e02b

The six focused cases prove that nullable and sensitive flags can originate in the string, start, or count argument:

-- + {name substr}: text
create proc substr_test_nullable_string(t text)
begin
  select substr(t, 1, 2) as t;
end;

-- + {name substr}: text
create proc substr_test_nullable_start(t text not null)
begin
  select substr(t, nullable(1), 2) as t;
end;

-- + {name substr}: text
create proc substr_test_nullable_count(t text not null)
begin
  select substr(t, 1, nullable(2)) as t;
end;

-- + {name substr}: text sensitive
create proc substr_test_sensitive_string(t text @sensitive)
begin
  select substr(t, 1, 2) as t;
end;

-- + {name substr}: text sensitive
create proc substr_test_sensitive_start(t text)
begin
  select substr(t, sensitive(1), 2) as t;
end;

-- + {name substr}: text sensitive
create proc substr_test_sensitive_count(t text)
begin
  select substr(t, 1, sensitive(2)) as t;
end;

Nullability, sensitivity, object kind, shape, and error state all decorate expressions. Every builtin and every rewrite has to preserve or recompute the right subset. That is why semantic bugs can recur in a well-tested compiler: each special case is effectively a tiny compiler pass.

Flow analysis failed on the branch that contained nothing

In December 2021, empty IF branches were not counted as branches. That made an improvement performed by every non-empty branch appear to be performed by every branch:

// Records the presence of an empty branch within the branch group. This is
// equivalent to calling `_flow_push_context_branch` immediately followed by
// `_flow_pop_context_branch`.
cql_noexport void flow_context_branch_group_add_empty_branch() {
  Contract(current_context);
  Contract(current_context->kind == FLOW_CONTEXT_KIND_BRANCH_GROUP);

  current_context->branch_group.branch_count++;
}

Commit: d17a350684151871d4ed78b1126f8d6e1a86cbf9

The semantic callers now record an absent body as a real branch:

   if (stmt_list) {
     FLOW_PUSH_CONTEXT_BRANCH();
     sem_set_notnull_improvements_for_true_condition(expr);
     sem_stmt_list_in_current_flow_context(stmt_list);
     FLOW_POP_CONTEXT_BRANCH();
     if (is_error(stmt_list)) {
       record_error(ast);
       return;
     }
+  } else {
+    flow_context_branch_group_add_empty_branch();
   }

@@
     if (stmt_list) {
       FLOW_PUSH_CONTEXT_BRANCH();
       sem_stmt_list_in_current_flow_context(stmt_list);
       FLOW_POP_CONTEXT_BRANCH();
       if (is_error(stmt_list)) {
         record_error(elsenode);
         record_error(if_alt);
         goto error;
       }
+    } else {
+      flow_context_branch_group_add_empty_branch();
     }

The regression puts the empty branch in each significant position and verifies that all six post-branch variables remain nullable:

-- TEST: An empty branch in an IF or SWITCH prevents the IF or SWITCH from
-- persisting any improvements.
-- + {name x0}: x0: integer variable
-- + {name x1}: x1: integer variable
-- + {name x2}: x2: integer variable
-- + {name x3}: x3: integer variable
-- + {name x4}: x4: integer variable
-- + {name x5}: x5: integer variable
-- - error:
create proc empty_branches_prevent_persisting_improvements()
begin
  declare a int;

  if 0 then
    -- empty
  else if 0 then
    set a := 1;
  else
    set a := 1;
  end if;
  let x0 := a;

  if 0 then
    set a := 1;
  else if 0 then
    -- empty
  else
    set a := 1;
  end if;
  let x1 := a;

  if 0 then
    set a := 1;
  else if 0 then
    set a := 1;
  else
    -- empty
  end if;
  let x2 := a;

  switch 0
  when 1 then
    -- empty
    leave;
  when 2 then
    set a := 1;
  else
    set a := 1;
  end;
  let x3 := a;

  switch 0
  when 1 then
    set a := 1;
  when 2 then
    set a := 1;
  else
    -- empty
    leave;
  end;
  let x4 := a;

  switch three_things.zero all values
  when three_things.zero then
    -- empty
    leave;
  when three_things.one then
    set a := 1;
  when three_things.two then
    set a := 1;
  end;
  let x5 := a;
end;

The empty branch had no data structure, but it was still a semantic case. I like the eventual abstraction because “empty branch” now has an explicit place in the flow protocol.

2. Schema migration and ordering form the hardest persistent subsystem

Schema migration was the most maddening subsystem in the history.

It had high coverage, including the 100% norm, and it was tested relentlessly. The implementation usually did exactly what we intended. The problem was that I did not fully understand all the constraints, so we verified the wrong model with great precision.

There were two related failure families:

1. version CRCs and housekeeping history could disagree with the actual SQLite schema;
2. drop, create, dependency, and recreate order were each individually plausible but not interchangeable.

The first family caused the more consequential operational failures. The second produced ordinary implementation bugs around comparators, loop direction, graph order, backend parity, and quoted names.

The question is which order applies. Schema work has several perfectly legitimate answers:

• declaration order;
• dependency order;
• reverse dependency order;
• drop order;
• create order;
• recreate-group order;
• primary-key column order;
• physical serialized-field order.

Each one is locally reasonable. Trouble starts when code reuses one where another is required.

The version-history model was wrong

The original theory treated schema versions as a durable history. Each version had a CRC, so parallel development could add different work to the same version and the upgrader could tell which exact version contents had already run.

That sounded reasonable and the generated code faithfully implemented it. It was still the wrong abstraction.

Schema regions meant that an old version could legitimately acquire new work when a product subscribed to another region. @unsub and @resub added more historical transitions. The stored facet could say that a later delete or column upgrade had already happened while the physical database had returned to an older state.

This produced “zombie” tables. A newly active old-version step could recreate a table while its later deletion facet remained clean. Commit c5b3bb92 moved tombstoned table drops out of individual versions and ran them whenever any upgrade occurred. The public historical account says that change reduced schema-upgrade failures by 90%.

We then found weird housekeeping states: operators could manually remove schema without tombstones, manually delete tables, or leave old version facets behind. Commit 89660734 deleted facets for version gaps. That reconciled one class of stale metadata, but it still repaired the history rather than asking what was physically present.

The test oracle was also getting stronger. In April 2021, 24410139 made the integration suite execute every nothing -> vj -> vi path and compare the result with a direct install of vi:

echo "Upgrade from nothing to v$j, then to v$i -- must match direct update to v$i"

The validator canonicalized the actual sqlite_master contents. This was much stronger than asking whether generated text matched a golden. At Meta we had the same idea first against real product schemas. The public regression made the principle general and durable.

The decisive change came after we had frequency data. The public history says roughly one database in 100,000 was missing columns even though the corresponding version CRC claimed the upgrade had run. We could see the physical failure, but we could not reproduce the complete historical path that created it.

Commit fe921eac stopped using per-version CRCs to decide whether normal tables and columns needed creation. In schematic form:

before: version CRC changed -> run the version's operations
after:  schema changed -> run idempotent operations guarded by physical existence

Versions became ordering labels. Actual table and column presence came from a cached dictionary built from sqlite_master. Each operation became locally idempotent:

bprintf(&preamble, "  LET result := cql_string_dictionary_find(%s_tables_dict_, table_);\n", global_proc_name);
bprintf(&preamble, "  SET exists_ := result IS NOT NULL and result IS NOT '';\n");

bprintf(&upgrade, "    IF NOT %s_table_exists(cql_compressed('%s')) THEN\n", global_proc_name, target_name);

The direct test was both simpler and faster. The public account reports a 13-15% improvement and a 2.2% reduction in upgrader size for the important case.

The final simplification was deletion. Commit 27f0cb64 removed @resub, its grammar and semantic machinery, historical transition matching, diagnostics, and dedicated tests: 2,136 deleted lines across 49 files. A table is unsubscribed or it is not. Removing @unsub is the resubscription operation.

At Meta the recreate-group policy hook went further and compared actual schema with desired schema before deciding whether destructive recreation was needed. It also had to cope with old orphaned tables, missing tombstones, and other states that should never have existed. The public runtime exposes this hook but deliberately implements the naive policy: it always performs full recreation.

The customer-visible symptom of the internal failures was a full schema reset and a full content download rather than preservation across upgrade. It was not fatal, but it was expensive and disruptive. Public Git records the aggregate failure-rate improvements; the reset and redownload behavior comes from my operational recollection.

This is the cleanest example in the study where 100% coverage did exactly what coverage promises while the system still failed. Every line of the wrong model could execute correctly. The missing oracle was the physical end state.

Drop order was not create order

In December 2021, recreate groups were sorted in the order needed to drop tables, and that same order was reused for creation. Foreign keys can require the reverse:

   CHARBUF_OPEN(recreate);
   CHARBUF_OPEN(update_tables);
+  CHARBUF_OPEN(pending_table_creates);

@@
     table_crc ^= crc_charbuf(&make_table);
-    bindent(&update_tables, &make_table, 4);

+    // Now we have to remember that the tables in the recreate annotations have been
+    // sorted by reverse ordinal, meaning they are in the correct order to DROP
+    // we emit the DROP statements first as normal but the creates have to be
+    // stashed in a pending buffer that accumulates in the reverse order.  When we're
+    // done with the group, then we emit the whole batch of creates in the natural order.
+    CHARBUF_OPEN(temp);
+    bindent(&temp, &make_table, 4);
+    bprintf(&temp, "%s", pending_table_creates.ptr);
+    bclear(&pending_table_creates);
+    bprintf(&pending_table_creates, "%s", temp.ptr);
+    CHARBUF_CLOSE(temp);

@@
     if (i + 1 < count && gname[0] && !Strcasecmp(gname, (note+1)->group_name)) {
       continue;
     }
+    bprintf(&update_tables, "%s", pending_table_creates.ptr);
+    bclear(&pending_table_creates);

@@
+  CHARBUF_CLOSE(pending_table_creates);
   CHARBUF_CLOSE(update_tables);
   CHARBUF_CLOSE(recreate);

Commit: 39551d179f77ce34c0a6678fa9e1a89ae14db156

The group-boundary flush is important: it emits the accumulated creates only after all members of that recreate group have been seen. This commit updated generated golden references, but added no executed foreign-key regression.

In October 2022, recreate-group comparator direction required another fix:

 // Sort the @recreate annotations in place: the order is:
-//  * group name
+//  * group ordinal
 //  * ordinal reversed
 //
+//  We use create order for groups so that we do the CRC check for the
+//  strongest tables first - wherein we recursively drop children in
+//  drop order and also drop any tables in the correct drop order via
+//  the call to cql_rebuild_recreate_group. We also dirty the CRC for
+//  any children we end up dropping, so we want the children (weak tables) to
+//  follow after the parent (strong table) so that we recreate them.
@@
-  // reverse ordinal order
+  // We need group ordinal to be in create order so that we blow away
+  // strongest groups (groups that other groups depend on) first
   if (a1->group_ordinal != a2->group_ordinal) {
-    return (a1->group_ordinal < a2->group_ordinal) ? 1 : -1;
+    return (a1->group_ordinal < a2->group_ordinal) ? -1 : 1;
   }

   // It can't be a tie! ordinal is unique!
   Invariant(a1->ordinal != a2->ordinal);

+  // reverse ordinal
   return (a1->ordinal < a2->ordinal) ? 1 : -1;

Commit: ddd4335d0cad57742814498d1ed5ebbb00e63627

In February 2023, @delete table processing similarly needed reverse order:

   cql_object_ref deleteList =
     _cql_create_upgrader_input_statement_list(deletes, "DROP ");

   cql_code rc = SQLITE_OK;
   // Execute all delete table drops
-  for (cql_int32 i = 0;
-       i < cql_string_list_get_count(deleteList); i++) {
+  // Note deleteList provides table drops in create order, so we must execute them in reverse.
+  for (cql_int32 i = cql_string_list_get_count(deleteList); --i >= 0; ) {
     cql_string_ref delete = cql_string_list_get_string(deleteList, i);
     rc = cql_exec_internal(db, delete);
     if (rc != SQLITE_OK) goto cleanup;
   }

Commit: 3b597671509042abf95c568848efd05fbb0659be

The expanded C test fixture supplied parent/child tables and a create-ordered delete list:

 cql_string_ref tables =
   cql_string_ref_new(
     " CREATE TABLE g1( id INTEGER PRIMARY KEY, name TEXT ); "
     "CREATE TABLE use_g1( id INTEGER PRIMARY KEY REFERENCES g1 (id), name2 TEXT); "
-    "CREATE TABLE foo(y text DEFAULT 'it''s, ('); ");
+    "CREATE TABLE foo(y text DEFAULT 'it''s, ('); "
+    "CREATE TABLE g2( id INTEGER PRIMARY KEY ); "
+    "CREATE TABLE use_g2( id INTEGER PRIMARY KEY REFERENCES g2 (id)); ");
 cql_string_ref indices =
   cql_string_ref_new("CREATE INDEX extra_index ON g1 (id); ");
-cql_string_ref deletes = cql_string_ref_new("DROP TABLE IF EXISTS g2; ");
+cql_string_ref deletes = cql_string_ref_new("DROP TABLE IF EXISTS g2; DROP TABLE IF EXISTS use_g2; ");
@@
 rc = cql_rebuild_recreate_group(db, tables, indices, deletes, &result);
 E(rc == SQLITE_OK, "expected succesful recreate group upgrade\n");

The commit did not complete the correction in Lua. It changed the Lua loop to this:

-  for i = 1, #deleteList do
+  for i = 1, #deleteList, -1 do

Because the start is below the positive-length end while the step is negative, that loop executes zero iterations. Commit 6f2c9498 completed the backend correction:

-  for i = 1, #deleteList, -1 do
+  -- these are deleted or unsubscribed tables
+  -- note that tables in the list are in create order, so we reverse to get drop order
+  for i = #deleteList, 1, -1 do
     rc = cql_exec(db, deleteList[i])
     if rc ~= sqlite3.OK then return rc end
   end

So 3b597671 fixed C drop order and expanded the C fixture, while introducing a zero-iteration Lua loop; 6f2c9498 completed the correction. The fixture checks successful execution, but it does not independently prove the drop order because the referenced tables are empty. This is a multi-backend correction sequence. I remember the drop-order failure as operational. Public Git proves the correction chain but does not independently classify it as a released incident. The Lua loop is an incomplete first fix, not a separately reported field defect.

These are different bugs, but they point to the same weak spot. “Order” lived implicitly in loops, comparators, and buffer accumulation instead of in named order domains.

Recreate groups show what happened after the model matured

I remembered recreate groups as the point where we finally had this under control. The Git history supports that conclusion, with an important qualification: the feature was not flawless on its first patch.

Vaishnavi Mantha’s first visible repository work in July 2022 was the runtime skeleton for cql_rebuild_recreate_group. She then added generated integration, compressed desired-schema inputs, the dependency graph, cycle rejection, partial rebuild status, and recursive dependent-group handling.

The runtime hook began before the decisive October create-plan simplification. The harder partial-rebuild and cross-group dependency work landed after that change, when physical state had become the ordinary create-plan authority.

This was harder than the original recreate feature. A runtime could preserve a group when the actual and desired schemas were compatible, or destructively recreate it otherwise. Foreign-key dependencies between groups meant that a parent recreation could force children to recreate as well.

The feature arrived after the integration suite had already spent more than a year requiring every staged upgrade path to converge on the same canonical physical schema. Dependency tests came with the implementation. They covered named groups, multiple referenced groups, same-group references, and cycle rejection.

Public Git does show several development corrections:

• the default rebuild result needed initialization;
• comparator direction needed a concrete parent/child counterexample;
• creation order had to become a true topological group order;
• dropping a child through its parent had to dirty the child’s saved state;
• migration procedures had to run only after destructive recreation;
• deleted-table drops needed reverse order in both C and Lua.

The topological fix in c5df0113 states the conceptual correction directly:

-// We use create order for groups
+// We use topological order for groups.
+// recreate_group_deps holds (parent->child) edges and has no cycles.

Its regression used the kind of interleaving that declaration order could not solve: one group contained a table that depended on another group, followed by another table from the first group.

Every recreate-group correction I found within this human-found window predates the first public version tag, v1.0.0, on February 25, 2024. I found no later functional recreate-group fix within the window and no public issue or commit describing a released recreate-group incident. That absence cannot prove private deployment history, but it corroborates my recollection that the issues were found before release and that the feature produced no customer incident.

By then we had an explicit dependency graph and tests that checked the physical schema end to end. We could fix each recreate-group failure and add a regression without redesigning the subsystem again.

Identifier correctness was distributed

The October 2023 quoted-identifier work, ead444bf , changed a major invariant: an internal name was no longer necessarily a ready-to-print SQL identifier. The compiler had to preserve quoted status, canonical identity, and target-language spelling across many consumers.

This fork-only feature was developed test-first, with an explicit burndown list of major identifier usages. It was well tested but lightly used. Those are compatible facts: even a systematic list depends on remembering all the places a name can appear.

Static SQL generation and dynamic SQL construction need the same identifier model and escaping helpers.

Subsystem tests can stop the exact bug from coming back while the same family continues in another emitter.

Rewrites compose badly unless state and syntax are explicit

The December 2024 work on backed tables, upsert, and RETURNING found several issues together:

• rowid reselection where it was not appropriate;
• a lost upsert WHERE;
• a missing ambiguity-avoiding WHERE 1;
• confusion between generated names and quoted user identifiers;
• nested rewrite state not restored correctly.

Commit: cf546d7ed13edb2642abb75addcd6d5b48479c53

Each rewrite makes sense on its own. Once combined, the tree carries the obligations of all of them, and that cross-product is where the failure lived.

The correction was spread across the composing rewrites. The generated insert adds SQLite’s documented ambiguity breaker only for upsert:

              new_ast_select_where(
-                NULL,
+                  in_upsert ? new_ast_opt_where( new_ast_num(NUM_INT, "1")) : NULL,
                new_ast_select_groupby(
                  NULL,
                  new_ast_select_having(
                    NULL,
                    NULL
                  )
                )
              )

The backed-table update stops synthesizing rowid reselection during upsert and preserves the upsert’s own WHERE, rewriting its backed-column references:

-  ast_node *select_stmt = rewrite_select_rowid(
-     backed_table_name,
-     opt_where,
-     opt_orderby,
-     opt_limit);
+  ast_node *select_stmt;
+
+  if (!in_upsert) {
+
+    // this generates the normal where clause for the update statement
+    // WHERE rowid in (SELECT rowid FROM backed) which is what you want
+    // for the pdate case
+
+    select_stmt = rewrite_select_rowid(
+      backed_table_name,
+      opt_where,
+      opt_orderby,
+      opt_limit);
+
+    // for debugging print just the select statement
+    // gen_stmt_list_to_stdout(new_ast_stmt_list(select_stmt, NULL));
+
+    // the new where clause for the update statement is going to be something like
+    // rowid IN (select rowid from selected rows)
+  }
+  else {
+    // in the upsert case, the rows in question are already selected by SQLite itself.
+    // We don't need to do the rowid selection, we just need to rewrite the
+    // update list to use the backing table columns directly and we'll select
+    // the rowid out of excluded.rowid
+    select_stmt = NULL;
+  }
@@
-  ast_node *new_opt_where =
-    new_ast_opt_where(
-     new_ast_in_pred(
-       new_ast_str("rowid"),
-       select_stmt
-     )
-  );
+  ast_node *new_opt_where = NULL;
+
+  if (!in_upsert) {
+    new_opt_where = new_ast_opt_where(
+     new_ast_in_pred(
+       new_ast_str("rowid"),
+       select_stmt
+     )
+    );
+  }
+  else {
+    // the upsert case can have a WHERE clause of its own, it stays as is
+    new_opt_where = opt_where;
+
+    if (opt_where) {
+      AST_REWRITE_INFO_SAVE();
+      AST_REWRITE_INFO_SET(opt_where->lineno, opt_where->filename);
+      rewrite_backed_column_references_in_ast(opt_where, backed_table);
+      AST_REWRITE_INFO_RESET();
+      AST_REWRITE_INFO_RESTORE();
+    }
+  }

The nested rewrite now preserves any already-active rewrite context, and CTE names use the normal quoted-name generator:

+  // this can be called from an existing rewrite in the upsert case, handle both cases
+  AST_REWRITE_INFO_SAVE();
   AST_REWRITE_INFO_SET(root->lineno, root->filename);
-
-  rewrite_blob_column_references(&info, root);
-
+    rewrite_blob_column_references(&info, root);
   AST_REWRITE_INFO_RESET();
+  AST_REWRITE_INFO_RESTORE();

-  EXTRACT_STRING(name, ast->left);
-  if (is_ast_star(ast->right)) {
-    // use the shorter syntax for (*) which is just the name
-    gen_printf("%s", name);
-  }
-  else {
-    gen_printf("%s (", name);
+  EXTRACT_ANY_NOTNULL(name_ast, ast->left);
+  gen_name(name_ast);
+  if (!is_ast_star(ast->right)) {
+    // skip this for foo(*), the shorter syntax is just the name
+    gen_printf(" (");
     gen_name_list(ast->right);
     gen_printf(")");
   }

The expanded runtime test proves that quoted backed-table names, insert/select, delete, update, upsert, WHERE, and RETURNING compose into an executable statement, then checks returned values and exhaustion:

proc make_jsonb_backed_schema()
begin
  [[backing_table]]
  [[jsonb]]
  create table `a backing table`(
    `the key` blob primary key,
    `the value` blob
  );
end;

[[backed_by=`a backing table`]]
create table `a table`(
  `col 1` int primary key,
  `col 2` int
);

TEST_MODERN_ONLY!(verify_returning,
begin
  make_jsonb_backed_schema();
    cursor C for
    with data(a, b) as (values
      (1, 1),
      (2, 4),
      (3, 9),
      (4, 16)
    )
    insert into `a table`(`col 1`, `col 2`)
      select * from data
      returning *;

    let i := 0;
    loop fetch C
    begin
      i += 1;
      EXPECT_EQ!(C.`col 1` * C.`col 1`, C.`col 2`);
    end;
    EXPECT_EQ!(i, 4);

    cursor D for
      delete from `a table` where `col 1` in (2, 3)
      returning *;

    fetch D;
    EXPECT_EQ!(D.`col 1`, 2);
    EXPECT_EQ!(D.`col 2`, 4);
    fetch D;
    EXPECT_EQ!(D.`col 1`, 3);
    EXPECT_EQ!(D.`col 2`, 9);
    fetch D;
    EXPECT!(not D);

    cursor E for select * from `a table`;
    i := 0;
    loop fetch E
    begin
      i += 1;
      EXPECT_EQ!(E.`col 1` * E.`col 1`, E.`col 2`);
    end;
    EXPECT_EQ!(i, 2);

    cursor F for
      update `a table` set `col 2` = `col 1` * 1000
      where `col 1` = 4
      returning *;

    fetch F;
    EXPECT_EQ!(F.`col 1`, 4);
    EXPECT_EQ!(F.`col 2`, 4000);
    fetch F;
    EXPECT!(not F);

    cursor G for
      insert into `a table`(`col 1`, `col 2`) values (1, 1), (5, 25)
      on conflict(`col 1`)
      do update set `col 2` = 1000 where `col 1` <= 4
      returning *;

    fetch G;
    EXPECT_EQ!(G.`col 1`, 1);
    EXPECT_EQ!(G.`col 2`, 1000);

    fetch G;
    EXPECT_EQ!(G.`col 1`, 5);
    EXPECT_EQ!(G.`col 2`, 25);
    fetch G;
    EXPECT!(not G);
end);

The production diff shows that the upsert WHERE is preserved and its backed-column references are rewritten. This runtime case does not independently exercise the predicate’s false branch: the only conflicting key satisfies the predicate, while key 5 is a non-conflicting insert.

How to think about the subsystem

From my chair, schema migration is a compiler inside the compiler. It has its own:

• dependency graph;
• ordering semantics;
• identifier language;
• generated program;
• runtime execution behavior;
• compatibility rules across versions.

That is why the end-to-end upgrade tests matter more than the large golden outputs. They execute the generated migration as a program instead of checking only its text.

3. Global state works better than normal advice predicts

Conventional advice says to avoid global mutable state and pass explicit contexts. CQL mostly keeps each compiler phase’s state in globals.

The phase files carry substantial file-static state, especially semantic analysis and the C and Lua generators.

There are real advantages:

• recursive visitors have small signatures;
• current procedure, scope, callbacks, and output buffers are immediately available;
• code for one compiler phase stays together;
• the implementation avoids passing a very large context through thousands of calls.

Recursion almost never needs all of that state. Most nested contexts save one or two symbol tables, a join scope, or a callback and output buffer. For example:

symtab *saved_locals = locals;
symtab *saved_globals = globals;

locals = globals = NULL;
PUSH_JOIN(expr_scope, jptr);
sem_root_expr(expr, SEM_EXPR_CONTEXT_CONSTRAINT);
POP_JOIN();

locals = saved_locals;
globals = saved_globals;

The generic SQL generator state is a complete three-pointer abstraction with paired save and restore operations:

typedef struct gen_sql_state {
  charbuf *_Nullable gen_output;
  gen_sql_callbacks *_Nullable gen_callbacks;
  symtab *_Nullable used_alias_syms;
} gen_sql_state;

cql_noexport void gen_get_state(gen_sql_state *state) {
  state->gen_output = gen_output;
  state->gen_callbacks = gen_callbacks;
  state->used_alias_syms = used_alias_syms;
}

cql_noexport void gen_set_state(gen_sql_state *state) {
  gen_output = state->gen_output;
  gen_callbacks = state->gen_callbacks;
  used_alias_syms = state->used_alias_syms;
}

That is the normal shape. The shared-fragment case below is exceptional: it re-enters SQL generation while fragment predicates, variable numbering, and fragment buffers are all in flight.

Nested code generation exposed the cost

In March 2025, evaluating a shared-fragment argument containing a nested SELECT re-entered SQL generation and clobbered outer generator state.

The code had already documented exactly what remained to be done:

// todo: if the evaluation has a nested select statement then we will have
// to re-enter all of this.  We can either ban that (which isn't insane
// really) or else we can save the codegen state like callbacks and such so
// that it can re-enter.  That's the desired path.

The save/restore work in the comment had not been implemented. The fix did exactly that:

gen_sql_state state;
gen_get_state(&state);

bytebuf sfs_saved;
memcpy(&sfs_saved, &shared_fragment_strings, sizeof(bytebuf));
memset(&shared_fragment_strings, 0, sizeof(bytebuf));

int cur_fragment_predicate_saved = cur_fragment_predicate;
int max_fragment_predicate_saved = max_fragment_predicate;
int prev_variable_count_saved = prev_variable_count;
int cur_variable_count_saved = cur_variable_count;
bool_t has_conditional_fragments_saved = has_conditional_fragments;
bool_t has_shared_fragments_saved = has_shared_fragments;
bool_t has_variables_saved = has_variables;

CG_PUSH_EVAL(expr, C_EXPR_PRI_ASSIGN);
cg_store(cg_main_output, alias_name, sem_type_var, sem_type_expr, expr_is_null.ptr, expr_value.ptr);
CG_POP_EVAL(expr);

gen_set_state(&state);
memcpy(&shared_fragment_strings, &sfs_saved, sizeof(bytebuf));
cur_fragment_predicate = cur_fragment_predicate_saved;
max_fragment_predicate = max_fragment_predicate_saved;
prev_variable_count = prev_variable_count_saved;
cur_variable_count = cur_variable_count_saved;
has_conditional_fragments = has_conditional_fragments_saved;
has_shared_fragments = has_shared_fragments_saved;
has_variables = has_variables_saved;

Commit: 7dadf83af8d84f3d5d7450769e5520b80efb65cf

The Lua backend received the corresponding state boundary around its actual nested evaluation:

gen_sql_state state;
gen_get_state(&state);
bytebuf sfs_saved;
memcpy(&sfs_saved, &lua_shared_fragment_strings, sizeof(bytebuf));
memset(&lua_shared_fragment_strings, 0, sizeof(bytebuf));

int cur_fragment_predicate_saved = lua_cur_fragment_predicate;
int max_fragment_predicate_saved = lua_max_fragment_predicate;
int prev_variable_count_saved = lua_prev_variable_count;
int cur_variable_count_saved = lua_cur_variable_count;
bool_t has_conditional_fragments_saved = lua_has_conditional_fragments;
bool_t has_shared_fragments_saved = lua_has_shared_fragments;
bool_t has_variables_saved = lua_has_variables;

CG_LUA_PUSH_EVAL(expr, LUA_EXPR_PRI_ASSIGN);
cg_lua_store(cg_main_output, alias_name, sem_type_var, sem_type_expr, expr_value.ptr);
CG_LUA_POP_EVAL(expr);

lua_cur_variable_count = cur_variable_count_saved;
lua_prev_variable_count = prev_variable_count_saved;
lua_cur_fragment_predicate = cur_fragment_predicate_saved;
lua_max_fragment_predicate = max_fragment_predicate_saved;
lua_has_conditional_fragments = has_conditional_fragments_saved;
lua_has_shared_fragments = has_shared_fragments_saved;
lua_has_variables = has_variables_saved;

gen_set_state(&state);
memcpy(&lua_shared_fragment_strings, &sfs_saved, sizeof(bytebuf));

The list is longer here because nested shared-fragment arguments can themselves contain shared fragments. This path has more live recursive context than the usual one or two symbol tables.

A helper dedicated to pushing and popping this shared-fragment generation context would be cleaner, and the manual C/Lua copies are maintenance debt. I never got around to making that helper. Still, this commit is better read as an unfinished, already-documented re-entry requirement than as evidence that all compiler global state needs to be snapshotted. The commit added unrelated Dirname coverage, but no focused regression source for nested shared-fragment argument evaluation.

Repeated compiler invocation has an unusually strong regression test

The amalgam test calls cql_main several times in one process:

const char *cql_success_file = (char *) argv[1];
const char *cql_semantic_error_file = (char *) argv[2];
const char *cql_syntax_error_file = (char *) argv[3];

const char *args_1[5] = {"cql", "--in", cql_success_file, "--sem", "--echo"};
if (cql_main(5, (char **)args_1) != 0) {
  fprintf(stderr, "%s reported an error: this is unexpected\n", cql_success_file);
  exit(1);
}

const char *args_2[5] = {"cql", "--in", cql_semantic_error_file, "--sem", "--echo"};
if (cql_main(5, (char **)args_2) == 0) {
  fprintf(stderr, "%s did not reported an error: this is unexpected\n", cql_semantic_error_file);
  exit(1);
}

const char *args_3[5] = {"cql", "--in", cql_syntax_error_file, "--sem", "--echo"};
if (cql_main(5, (char **)args_3) == 0) {
  fprintf(stderr, "%s did not reported an error: this is unexpected\n", cql_syntax_error_file);
  exit(1);
}

if (cql_main(5, (char **)args_1) != 0) {
  fprintf(stderr, "%s reported an error: this is unexpected\n", cql_success_file);
  exit(1);
}

Source: sources/test/amalgam_test.c

I like this better than a cleanup unit test. It verifies observable reentrancy after both semantic and syntax failure, which directly exercises the risk created by phase globals.

The global-state design is deliberate

Phase globals come with explicit cleanup entry points. For example, C generator initialization calls cleanup before rebuilding dispatch and state, and compiler orchestration cleans every phase before returning.

cql_noexport void cg_c_init(void) {
  cg_c_cleanup(); // reset globals/statics
  cg_common_init();

  Contract(!error_target_used);

  // one table for the whole translation unit
  Contract(!string_literals);
  string_literals = symtab_new_case_sens();

cg_c_cleanup();
sem_cleanup();
ast_cleanup();
gen_cleanup();
rt_cleanup();
parse_cleanup();
cql_cleanup_open_includes();
cql_cleanup_defines();

#ifdef CQL_AMALGAM
  // the variables need to be set back to zero so we can
  // be called again as though we were just loaded
  cql_reset_globals();
#endif

I read the history this way: top-level reset works, ordinary recursion saves little context, and shared-fragment generation is the exceptional wide re-entry case that still wants a focused helper.

4. Ownership helpers and bulk lifetimes are CQL’s strongest C-level design

C usually gives you leaks, use-after-free, double-free, dangling pointers, aliasing errors, buffer overruns, integer overflow, and undefined behavior from alignment or effective-type violations.

CQL removes much of the ordinary ownership burden by dividing memory into a few strong lifetime classes.

AST and durable compiler data use arenas

The minipool is intentionally simple. The exact excerpts below omit the remainder of the introductory commentary and minipool_open, which eagerly initializes the first block:

// This is the stupidest pool allocator ever, it's only useful for cases where
// everything lives until the end.  Currently it's used to hold ast nodes and
// duplicated strings.

cql_noexport void minipool_close(minipool **head) {
  minipool *pool = *head;
  while (pool) {
    minipool *next = pool->next;
    free(pool->bytes);
    free(pool);
    pool = next;
  }
  *head = NULL;
}

// Get needed memory; if the memory is not available then we allocate a new
// block and thread it into the list.  Note that this is different than the
// other CQL helper bytebuf. This allocation never moves bytes or copies bytes.
// Once you get memory it is valid until you close the buffer. Bytebuf cannot
// hold pointers because the structure moves when it grows and it is one
// continuous allocation so it's only for byte streams. Minipool doesn't move
// objects around so pointers remain valid.
cql_noexport void *minipool_alloc(minipool *pool, uint32_t needed) {
  void *result;

  // for alignment, all allocs will be on a 8 byte boundary
  needed += 7;
  needed &= u32_not(7);

  if (needed > pool->available) {
    // Make a copy of the most recent head node and link to it we have to do
    // this because the head of the pool never changes once it's created.  Note
    // that none of the byte buffers are copied, only the minipool struct.  Once
    // we've copied it, we re-initialize the minipool to a size that is at least
    // big enough for the next allocation.
    minipool *old = malloc(sizeof(minipool));
    *old = *pool;
    uint32_t blocksize = MAX(needed, MINIBLOCK);
    pool->bytes = malloc(blocksize);
    pool->current = pool->bytes;
    pool->available = blocksize;
    pool->next = old;
  }

  result = pool->current;
  pool->current += needed;
  pool->available -= needed;
  return result;
}

The key property is in the comment: allocations never move and remain valid until the whole pool closes. Compiler ASTs naturally live for the compilation, so individual free calls would add risk with very little useful memory recovery.

That runs directly against generic “free each allocation promptly” advice. For this workload, I think the bulk lifetime is the better engineering choice.

Reference ownership needed type-specific operations

In November 2024, generated code released a destination reference before evaluating a create call. That was wrong when the old destination value was also an argument to the call:

static void cg_copy_for_create(charbuf *output, CSTR var, sem_t sem_type_var, CSTR value) {
   if (is_text(sem_type_var)) {
     bprintf(cg_main_output, "%s(%s);\n", rt->cql_string_release, var);
   }
   else if (is_blob(sem_type_var)) {
     bprintf(output, "%s(%s);\n", rt->cql_blob_release, var);
   }
   else if (is_object(sem_type_var)) {
     bprintf(output, "%s(%s);\n", rt->cql_object_release, var);
   }
   bprintf(output, "%s = %s;\n", var, value);
}

The fix introduced explicit setters for incoming +1 references:

void cql_set_created_object_ref(
  cql_object_ref _Nullable *_Nonnull target,
  cql_object_ref _Nullable source)
{
  // no upcount, we were given an upcounted ref. We just release what we had and store
  cql_object_release(*target);
  *target = source;
}

The executable aliasing case removed its temporary workaround and passes the destination directly as an argument to the create function:

-let b_temp := my_blob; -- avoid bug in cg_copy_for_create
-my_blob := corrupt_blob_with_invalid_shenanigans(b_temp);
+my_blob := corrupt_blob_with_invalid_shenanigans(my_blob);

The generator now emits the type-specific created-reference operations:

if (is_text(sem_type_var)) {
  bprintf(output, "cql_set_created_string_ref(%s%s, %s);\n", prefix, var, value);
}
else if (is_blob(sem_type_var)) {
  bprintf(output, "cql_set_created_blob_ref(%s%s, %s);\n", prefix, var, value);
}
else if (is_object(sem_type_var)) {
  bprintf(output, "cql_set_created_object_ref(%s%s, %s);\n", prefix, var, value);
}

Selected exact cases from separate locations in cg_test.sql cover object, blob, string, and boxing paths:

-- + cql_set_created_object_ref(&obj_var, obj_func_create());
set obj_var := obj_func_create();

-- + cql_set_created_blob_ref(&blob_var, blob_func_create());
set blob_var := blob_func_create();

-- + cql_set_created_string_ref(&s, cql_cursor_format(&C_dyn));
PROC BigFormat ()
BEGIN
  DECLARE C CURSOR FOR SELECT * FROM big_data;
  LOOP FETCH C
  BEGIN
    LET s := cql_cursor_format(C);
  END;
END;

-- + cql_set_created_object_ref(&C_object_, cql_box_stmt(C_stmt));
-- + cql_set_object_ref(&box, C_object_);
proc call_in_loop_boxed()
begin
  declare i int;
  set i := 0;
  while i < 5
  begin
    set i := i + 1;
    declare C cursor for call simple_select();
    declare box object<C cursor>;
    set box from cursor C;
    declare D cursor for box;
    fetch D;
  end;
end;

Commit: 0001945b26ad1b50b80f439ce57f2b9fbe89f523

I much prefer this to merely reordering the release. The helper name encodes the ownership contract and reduces the legal spellings of a subtle operation.

Net reference balancing is checked for every runtime test

The runtime TEST! macro records outstanding references around each test:

start_refs := get_outstanding_refs();
@id("test_", name!)();
end_refs := get_outstanding_refs();

if start_refs != end_refs then
  printf("Test %s unbalanced refs.", @text(name!));
  printf("  Starting refs %d, ending refs %d.\n", start_refs, end_refs);
  fails := fails + 1;
end if;

Source: sources/test/run_test.sql

This is a powerful general invariant. It catches net leaks regardless of the feature a test was written to exercise. FWIW, the limit is just as clear: balanced counts cannot prove correct ordering, absence of temporary use-after-release, or correct ownership transfer. The created-reference bug survived until the abstraction itself got stronger.

Why this design holds up

CQL’s best C mitigations reduce the number of lifetimes a programmer has to reason about:

• AST objects: compilation arena lifetime.
• Interned strings: durable shared lifetime.
• output buffers: lexical open/close lifetime.
• generated runtime references: retain/release helper protocol.
• created runtime references: distinct set_created protocol.

The remaining ownership bugs tend to appear at a crossing between lifetime classes, or when code spells out the protocol by hand.

5. Representation boundaries expose hidden assumptions

The two public binary facilities started with different contracts. Both were lightly used.

Two serializers, two starting contracts

There are two binary serializer histories in this codebase and they should not be mashed together.

The original cursor serializer was designed to take arbitrary cursor-shaped data and turn it into a blob for storage or transport. From the beginning we expected version skew: the reader might be older or newer than the writer. We also expected the blob to be attacker controlled. The March 2022 introduction, b573687a , said so directly:

The blob reading code is hardened so that it assumes the content of the blob is hostile. There are tests for this.

cql_cursor_from_bytes still puts that contract next to the implementation:

// This is the inverse of cql_cursor_to_bytebuf, it takes a byte stream and
// reconstructs a dynamic cursor from it.  The byte stream is assumed to be
// hostile. It could be corrupted in any kind of way and this code is expected
// to handle that.
cql_code cql_cursor_from_bytes(
  cql_dynamic_cursor *_Nonnull dyn_cursor,
  const uint8_t *_Nonnull bytes,
  uint32_t size)

The type signature in the blob provides shape versioning. A newer reader can fill missing trailing nullable fields with NULL. Two days after the initial change, 0b3c88a6 completed the other direction so that an older reader can ignore extra trailing fields from a newer writer. The current comment is wonderfully specific:

// Extra fields do not have to match, the assumption is that this is a
// future version of the type talking to a past version.  The past version
// sees only what it expects to see.  However, we did have to compute the
// nullable_count and bool_count to get the bit vector size correct.

The first few changes already included malformed-input cases, exhaustive truncation, and mutation under sanitizers. The serializer still had bugs. The most informative one appeared when the Lua implementation was required to produce exactly the same bytes as C in d94eded9 . That comparison found two shared-format mistakes:

• nested blob lengths used a fixed native integer instead of a varint;
• a nullable Boolean that was NULL did not advance the Boolean bit index.

The second problem made nullable Booleans effectively unusable. Ordinary C round trips did not reveal either problem because the C encoder and decoder agreed with the same representation. A fixed byte sequence and an independent Lua implementation did.

The bgetkey/bcreatekey/bupdatekey family had a completely different origin. These are SQLite extensions for backed tables. The operation names are configurable because applications are allowed to provide whatever storage format they want.

At Meta we had a hardened implementation that depended on internal infrastructure and could not be open-sourced. I needed enough public implementation to run the compiler tests, so the first open-source version was deliberately a stub. The September 2022 source could hardly have been clearer:

// this is nothing like a generic blob format but it's enough to do some tests
// the blobs that are stored can only be integers of course but since the
// codegen is the same for all types it still makes for a useful test
typedef struct test_blob {
  int64_t type;
  int64_t int1;
  int64_t int2;
  int64_t int3;
} test_blob;

Over time the stub became useful enough that it started becoming real. The 2023 implementation added all the primitive types, sparse values, canonical keys, updates, and a versioned header. The comments openly tracked the unfinished parts:

// NOTE: THIS CODE DOES NOT HANDLE NULL COLUMNS (which are basically the same as missing)
void bcreateval(sqlite3_context *_Nonnull context, int32_t argc, sqlite3_value *_Nonnull *_Nonnull argv) {

// NOTE: THIS CODE DOES NOT HANDLE THE CASE WHERE NEW COLUMNS ARE ADDED
void bupdateval(sqlite3_context *_Nonnull context, int32_t argc, sqlite3_value *_Nonnull *_Nonnull argv) {

Those limitations were subsequently removed, but the input contract was still the old one. A later version said:

// not computed but that is of no import since we are not yet validating all
// the internal offsets (blobs are assumed to be well formed for now)

and its failure path returned SQL NULL with the comment:

// If anything goes wrong we just return a null blob
// We could probably do better than this.

At this endpoint the public backed serializer had grown well beyond its original test stub, but still carried a well-formed-input assumption. That is a different starting contract from the cursor serializer’s explicit hostile-input requirement.

The human-era defects already exposed representation obligations. In August 2023, 9e9a4e09 corrected variable payloads emitted in update order rather than canonical field order. Equivalent logical keys need equivalent bytes; a successful decode alone does not establish that property.

The nullable Boolean cursor defect is even stronger evidence of light use. An actual use-case for nullable Booleans would have exposed it quickly. Coverage and same-implementation round trips had not supplied that feedback.

Platform width assumptions appeared at the lexer boundary

The earliest public-history example is a November 2020 fix for systems where long is 32 bits:

-0x[0-9a-fA-F]+               { char *s = Strdup(yytext); yylval.sval = s; return strtol(s+2, NULL, 16) <= 0x7fffffff ? INTLIT: LONGLIT; }
+0x[0-9a-fA-F]+               { char *s = Strdup(yytext); yylval.sval = s; return strtoll(s+2, NULL, 16) <= 0x7fffffff ? INTLIT: LONGLIT; }
 0x[0-9a-fA-F]+L              { char *s = Strdup(yytext); s[strlen(s)-1] = 0; /* remove the L */ yylval.sval = s; return LONGLIT; }
-[0-9]+                       { char *s = Strdup(yytext); yylval.sval = s; return atol(s) <= 0x7fffffff ? INTLIT: LONGLIT; }
+[0-9]+                       { char *s = Strdup(yytext); yylval.sval = s; return atoll(s) <= 0x7fffffff ? INTLIT: LONGLIT; }
 [0-9]+L                      { char *s = Strdup(yytext); s[strlen(s)-1] = 0; /* remove the L */ yylval.sval = s; return LONGLIT; }

Commit: 36e3e3a7880407509205158c1a632a21f9840dac

This is ordinary C portability trouble in a compiler-specific location: literal classification. A host-width assumption changed the accepted source language. The commit added no regression test, so the portability effect is supported by the lexer threshold logic and commit intent rather than an executed cross-platform case.

6. Lexer and parser bugs were fewer, but boundary-sensitive

The lexer and parser account for fewer conservative candidates than semantics, runtime, or code generation. The fixes that do show up usually involve:

• token adjacency;
• escape handling;
• host integer width;
• recursion direction and parser stack use;
• the difference between language errors and internal contracts.

Keyword recognition consumed an identifier prefix

In January 2021:

- ELSE[ \t\n]IF
+ ELSE[ \t]+IF/[^A-Z_0-9]

Commit: fa6522e3ada6be01612f527315fe62b63c7d9050

The old rule could tokenize ELSE IFNULL(...) as ELSE_IF. The regression test is admirably small:

-- TEST: ensure that ifnull parses properly after else, it's not "else if"
-- + ELSE ifnull(x, y)
select case when 1 then 2 else ifnull(x, y) end;

This is where lexical bugs like to hide: at the boundary between a keyword phrase and a longer identifier.

Grammar direction was a resource-use property

In July 2021, statement lists changed from right recursion to left recursion to avoid exhausting the Bison stack on very large procedures:

-  stmt ';'  { $result = new_ast_stmt_list($stmt, NULL); $result->lineno = $stmt->lineno;}
-  | stmt ';' stmt_list[slist]  { $result = new_ast_stmt_list($stmt, $slist); $result->lineno = $stmt->lineno; }
+  stmt ';'  {
+     // We're going to do this cheesy thing with the stmt_list structures so that we can
+     // code the stmt_list rules using left recursion.  We're doing this because it's
+     // possible that there could be a LOT of statements and this minimizes the use
+     // of the bison stack because reductions happen sooner with this pattern.  It does
+     // mean we have to do some weird stuff because we need to build the list so that the
+     // tail is on the right.  To accomplish this we take advantage of the fact that the
+     // parent pointer of the statement list is meaningless while it is unrooted.  It
+     // would always be null.  We store the tail of the statement list there so we know
+     // where to add new nodes on the right.  When the statement list is put into the tree
+     // the parent node is set as usual so nobody will know we did this and we don't
+     // have to add anything to the node for this one case.
+     //
+     // With this done we can handle several thousand statements without using much stack space.
+     $result = new_ast_stmt_list($stmt, NULL);
+     $result->lineno = $stmt->lineno;
+
+     // set up the tail pointer invariant to use later
+     $result->parent = $result;
+     }
+  | stmt_list[slist] stmt ';'  {
+     ast_node *new_stmt = new_ast_stmt_list($stmt, NULL);
+     new_stmt->lineno = $stmt->lineno;
+
+     // use our tail pointer invariant so we can add at the tail without searching
+     ast_node *tail = $slist->parent;
+     ast_set_right(tail, new_stmt);
+
+     // re-establish the tail invariant per the above
+     $slist->parent = new_stmt;
+     $result = $slist;
+     }

Commit: 1c95bbdb8e20dbf6a02e479245dce03dbde95fd9

A conventional representation review might object because the field temporarily changes meaning. Here the invariant is local to parser construction, documented in detail, and erased when the list is rooted. Given the existing AST shape, I think that is a sound engineering trade. The commit updated grammar documentation but added no checked-in large-procedure stress regression; the stack-use claim rests on the recursion direction and commit rationale.

Parentheses require associativity, not only precedence

In November 2020, C generation checked only whether a child operator had weaker precedence. Equal-precedence right children of left-associative operators also needed parentheses:

// This tells us if a subtree should be wrapped in ()
// Basically we know the binding strength of the context (pri) and the current element (pri_new)
// Weaker contexts get parens.  Equal contexts get parens on the right side because all ops
// are left to right associtive in SQL. Stronger child contexts never need parens because
// the operator already binds tighter than its parent in the tree.
static bool_t needs_paren(ast_node *ast, int32_t pri_new, int32_t pri) {
  if (pri_new != pri) {
    return pri_new < pri;
  }

  return ast->parent->right == ast;
}

Commit: 47e78dd5e6a56c501b694030e0a15e92027be2fe

The helper replaced precedence-only checks at the actual emission point:

-if (pri_new < pri) {
+if (needs_paren(ast, pri_new, pri)) {
   bprintf(value, "(%s)", result.ptr);
 }
 else {
   bprintf(value, "%s", result.ptr);
 }

Paired tests distinguish equal-precedence left trees from right trees:

-- + x = 1 * (2 / 3);
SET x := 1 * (2 / 3);

-- + x = 1 * 2 / 3;
SET x := 1 * 2 / 3;

-- + x = 1 - 2 - (2 - 3);
SET x := 1 - 2 - (2 - 3);

-- + x = 1 / 2 / 3;
SET x := 1 / 2 / 3;

-- + x = 1 / (2 / 3);
SET x := 1 / (2 / 3);

This is a standard tree-to-text failure. The operator set survives, but the original tree does not, because precedence alone is an incomplete printing rule.

7. Outputs and examples can become infrastructure

JSON has its own expectation-migration story.

I remember it starting as a nice-to-have output. It became critical to the Meta CI/CD system because it was a much better boundary than adding another large C backend every time somebody needed a new integration. The missing axis is downstream usage: the generator was already capable at release, but only a small amount of the build depended on it.

JSON was present in the October 2020 open-source release. cg_json_schema.c arrived in the initial commit with 1,819 physical lines, nearly 1,000 lines of test input, and a 2,473-line golden. The standard suite already checked expected content, generic JSON validity, and conformance to the CQL JSON grammar. So it did not begin in public as an untested toy.

What changed after release was the ambition and importance of the output. The public history captures plenty of that transition: 80 later commits touched cg_json_schema.c, while its test input roughly doubled and its golden more than tripled. JSON accumulated views, virtual tables, attributes, interfaces, subscriptions, declarations, dependencies, canonical duplicate suppression, per-entity CRCs, source locations, schema text, and other facts needed by real downstream tools.

The 2021 changes show the center of gravity moving. JSON gained view, index, and trigger attributes; per-entity CRCs for cheap change detection in 8ae78ade ; and shared-fragment dependency traversal in 52b6f6bb . At the end of that year, c31e0053 added cqljson.py, making the format immediately useful for databases, diagrams, and other flexible consumers. That is the phase where JSON stopped being merely nice output and became an engineering boundary.

Downstream use kept growing. At Meta the build system eventually consumed JSON to phase the build and decide which secondary artifacts had to be generated. At that point JSON was no longer just metadata handed to optional tools. It was part of the build’s control plane.

The public history later shows the same direction. Commit e5e91286 added isEmitted and variable-group information so JSON consumers could tell which groups a translation unit actually emitted. That is build-planning information, not decorative schema output.

The important design choice was to make the JSON format the contract and let the consumers be ordinary programs. That contract steadily became richer:

• attributes and dependencies had to survive compiler abstractions;
• per-entity CRCs made schema sameness cheap to test;
• duplicate legal declarations had to collapse to one canonical entry;
• high-bit text and later general UTF-8 had to remain valid JSON;
• field naming changes became compatibility events rather than local pretty-printing choices.

This moved a lot of work out of wordy C and into flexible Python. The deleted history makes the scale visible.

In January 2023, a94c748c removed the old Java C backend and its runtime/test surface: 2,934 deleted lines across 64 files. A Python generator consuming JSON replaced the same kind of work with something much easier to customize.

The Objective-C transition was even more explicit. 00875ef5 said cqlobjc.py was a complete replacement for --rt objc. A week later, 003d37a9 deleted the C backend and 4,185 lines of Objective-C golden output. The commit message explains why:

ObjC is now on the same footing as all the other ABI wrappers and indeed the python is a lot easier for most people to work with.

That is exactly what happened internally at Meta a few years earlier. JSON made the extraction possible.

The Python tools do not all have the same status. cqljson.py still says that its diagrams and generated database are sample outputs while the JSON input is the stable contract:

# NB: This code should be considered SAMPLE code, not production code.
# Which is to say you can reasonably expect that the specifics of the diagrams
# and the database produced here are likely to change at whim.  If you need
# a particular output, you are enouraged to FORK this sample into something
# stable.  The JSON format itself is the contract and it evolves in a backwards
# compatible way.  This script is likely to change to make different pretty
# pictures at various times.

The Java, C#, full Objective-C, and SQLite-extension generators are useful examples that applications can fork. cqlobjc.py is different because it actually replaced an integrated compiler backend. Its copied sample-code header no longer describes its real role particularly well.

The quality standard changed as the role changed. ec3da9eb explicitly said that cqljson.py was sample code and manual testing was enough. That is a different testing obligation from an integrated compiler backend or a tool used in normal delivery. The status of the Python examples and their promotion into infrastructure must be made explicit.

A sample that becomes part of normal delivery needs stronger compatibility and safety guarantees. The standard of excellence moved because the tool became important.

When a Python consumer replaces a C backend, the old backend’s coverage should not simply disappear with the deleted code. The replacement needs its own direct regression surface. JSON provides an excellent intermediate contract, but the contract does not test every consumer by itself.

8. The tests are unusually layered; independence matters more than volume

The human-era endpoint’s corpus is large. These counts include nested files under sources/test:

CQL uses several different kinds of checks:

Layer 1: directive-level semantic expectations

cql-verify supports ordered positive matches, negative matches, exact counts, exactly-one checks, and same-line checks:

-- match and advance the current match pointer
-- + foo       --> match foo, searching forward from the last match with +

-- these forms do not change the current search position
-- +[0-9] foo  --> match foo anywhere, but demand exactly n matches
-- - foo       --> shorthand for +0 foo (demand no matches)
-- * foo       --> shorthand for +1 foo (demand 1 match, anywhere)
-- = foo       --> match foo on the same line as the last match with +

This fits semantic analysis well because one compiler invocation can continue after errors and validate thousands of decorated AST fragments.

Layer 2: whole-output goldens

Golden files detect broad collateral changes. The harness normalizes unstable line numbers. In non-interactive runs it fails rather than blessing a diff; in an interactive run it copies output only after explicit confirmation:

if [ "${NON_INTERACTIVE:-0}" == 1 ] || [ ! -t 0 ] || [ ! -t 1 ]; then
  echo "When running: diff $*"
  echo "The above differences were detected. If these are expected copy the test output to the reference."
  echo "You can also re-run the tests without specifying --non_interactive to affirm the updates."
  echo "Don't just accept the changes to make the error go away; you have to really understand the diff first!"
  echo " "
  failed
else
  read -rp "When running: diff $*
The above differences were detected. Is this expected?
Don't just accept to make the error go away; you have to really understand the diff first! (y/N) " ANS
  case $ANS in
  [Yy]*) cp "$2" "$1" 2>/dev/null ;;
  *)
    echo " "
    failed
    ;;
  esac
fi

Source: sources/common/test_helpers.sh

That social safeguard matters. Golden tests make it very easy to bless a self-consistent but incorrect new world.

Layer 3: generated-code compilation

The suite also compiles and runs the generated C code. Generated test-helper CQL is semantically analyzed, converted to C, compiled, and executed. That compositional path is one of the suite’s strongest features. The exact command/run pairs below omit only each case’s TEST_NAME and TEST_DESC metadata:

TEST_CMD="${CQL} --sem --ast --in \"$O/cg_test_test_helpers.out\""
run_test_expect_success

TEST_CMD="${CQL} --test --dev --cg \"$O/test_helpers.h\" \"$O/test_helpers.c\" --in \"$O/cg_test_test_helpers.out\""
run_test_expect_success

TEST_CMD="do_make test_helpers_test"
run_test_expect_success

TEST_CMD="./$O/test_helpers_test"
run_test_expect_success

Layer 4: runtime execution

The runtime framework evaluates operands once, reports both sides on failure, and checks reference balance:

let @tmp(a) := a!;
let @tmp(b) := b!;
expected := @tmp(a) IS @tmp(b);
error_check(expected, @text(a!, " == ", b!), @MACRO_LINE);
if not expected then
  printf("left: %s\n", @tmp(a):fmt);
  printf("right: %s\n", @tmp(b):fmt);
end if;

Layer 5: differential or external oracles

EXPECT_SQL_TOO! evaluates the same expression in generated CQL code and inside SQLite:

@MACRO(stmt_list) EXPECT_SQL_TOO!(pred! expr)
begin
  pred!:EXPECT!;
  (select pred!):EXPECT!;
end;

This is stronger than another expected-output file because SQLite independently evaluates the same expression.

The same run_test.sql serialization case runs through both backends and compares its bytes with one fixed hexadecimal result:

let b := ext_cursor:to_blob;

let actual_hex := hex(b);
let expected_hex :=
   "66696C64736266696C64736246494C44534266696C64736200C00F090280D0AC"
   "F30E0000000000000A406120737472696E670014666972737420626C6F62E807"
   "8080A2A9EAE801000000000080234061207365636F6E6420737472696E670016"
   "7365636F6E6420626C6F62616E20756E7573656420737472696E6700";

-- this is true on all platforms!
EXPECT_EQ!(actual_hex, expected_hex);

The Lua harness compiles and executes that same source:

out/cql --in test/run_test.sql --cg out/run_test_core.lua --rt lua --global_proc go
cat lua_demo/test_helpers.lua out/run_test_core.lua >out/run_test.lua
echo "go(sqlite3.open_memory())" >>out/run_test.lua
lua out/run_test.lua | tee out/run_test_lua.out

The C harness separately builds and runs against the system SQLite link and a checked-in modern SQLite:

do_make run_test
MAKE_ARGS_SAVED=${MAKE_ARGS}
MAKE_ARGS=$(echo SQLITE_PATH=./sqlite ${MAKE_ARGS})
do_make sqlite
do_make run_test_modern
MAKE_ARGS=${MAKE_ARGS_SAVED}

TEST_CMD="./$O/run_test"
run_test_expect_success

TEST_CMD="./$O/run_test_modern"
run_test_expect_success

The canonical self-consistent failure

In December 2024, CQL consistently emitted invalid SQLite RETURNING (x,y,z) syntax. Parser, semantic, C, Lua, JSON, and golden outputs all agreed with one another because they shared the same mistaken model. Execution against SQLite exposed it.

Commit: 3163e93e7f01

The grammar and all four DML emitters removed the invented parentheses:

-returning_suffix: RETURNING '(' select_expr_list ')' { $$ = $select_expr_list; }
+returning_suffix: RETURNING select_expr_list { $$ = $select_expr_list; }

@@ -3652,35 +3652,34 @@ static void gen_delete_returning_stmt(ast_node *ast) {
-  gen_printf("\n  RETURNING (");
+  gen_printf("\n  RETURNING ");
   gen_select_expr_list(ast->right);
-  gen_printf(")");

@@ -3814,35 +3813,34 @@ static void gen_update_returning_stmt(ast_node *ast) {
-  gen_printf("\n  RETURNING (");
+  gen_printf("\n  RETURNING ");
   gen_select_expr_list(ast->right);
-  gen_printf(")");

@@ -4020,35 +4018,34 @@ static void gen_insert_returning_stmt(ast_node *ast) {
-  gen_printf("\n  RETURNING (");
+  gen_printf("\n  RETURNING ");
   gen_select_expr_list(ast->right);
-  gen_printf(")");

@@ -5351,35 +5348,34 @@ static void gen_upsert_returning_stmt(ast_node *ast) {
-  gen_printf("\n  RETURNING (");
+  gen_printf("\n  RETURNING ");
   gen_select_expr_list(ast->right);
-  gen_printf(")");

The resulting test executes the insert through SQLite, consumes every returned row, checks its values, and checks the final row count:

TEST_MODERN_ONLY!(verify_insert_returning,
begin
  make_jsonb_backed_schema();
  cursor C for
    with data(a, b) as (values
      (1, 1),
      (2, 4),
      (3, 9),
      (4, 16)
    )
    insert into `a table`(`col 1`, `col 2`)
      select * from data
      returning(*);

  let i := 0;
  loop fetch C
  begin
    i += 1;
    EXPECT_EQ!(C.`col 1` * C.`col 1`, C.`col 2`);
  end;
  EXPECT_EQ!(i, 4);
end);

More golden output would have preserved this mistake. Executing the SQL through SQLite caught it.

Short-circuit logic needed more than a complete truth table

AND and OR had a similar history on a smaller scale.

The public import already contains the mature C design, so Git cannot show the many internal iterations that produced it. By October 2020 the suite covered all nine false/true/NULL combinations for each operator. It checked both the result and observable call counts, including the asymmetric cases:

• false AND NULL skips the right side;
• NULL AND false must evaluate the right side;
• true OR NULL skips the right side;
• NULL OR true must evaluate the right side.

It also used divide-by-zero expressions to prove that a skipped operand really was not evaluated:

EXPECT_SQL_TOO!(not (0 and 1/zero));
EXPECT_SQL_TOO!(1 or 1 / zero);

Those tests were comprehensive across the value and evaluation-count model. They were not comprehensive across every execution mechanism needed to implement that model.

In October 2021, c2cb6af4 found a temporary-lifetime dimension. The generator traversed the deferred right operand before the left operand but emitted the left first. Nested nullable logical expressions could therefore reuse a temporary that was still live.

The fix made traversal follow emitted evaluation order:

   CG_RESERVE_RESULT_VAR(ast, sem_type_result);
-
+  CG_PUSH_EVAL(l, pri_new);
   CHARBUF_OPEN(right_eval);

The added regressions used nested AND/OR directly as branch conditions and required every live Boolean temporary to remain distinct. The original 3x3 matrix could not reveal that failure because each case contained one top-level logical operation.

Lua added another missing dimension. The initial backend deferred a statement-producing right side inside an anonymous function. That preserved values and side-effect counts, and the golden output approved it. It did not preserve the error continuation when a procedure-as-function needed to transfer control to the enclosing procedure’s cleanup path.