Scaffold semantic analysis #14

Merged

crowmaki merged 9 commits from types into main

2026-08-24 02:26:41 +00:00

crowmaki commented

2026-08-24 02:26:02 +00:00

Owner

Step 00.

crowmaki added 9 commits

2026-08-24 02:26:02 +00:00

spec: literal bindings, overload resolution, and type tests afb01a5f61

Resolves three gaps where the spec said too little to implement, or said two
things. Each is written into docs/prd and bound to a fixture in the manifest.

Literal bindings (Type Inference > Literal Bindings). Basic Data Types said
`const a = 128; // u8` while Type Inference said `let x = 3; // s32`; both
could not be a rule about literals. It is a rule about bindings: a `const`
takes the smallest fitting type, since it can never be reassigned, and a `let`
widens to s32/f32, since a mutable u8 inferred from 128 would reject `x = 300`
for a reason invisible at the assignment. Every existing example and both
literal lex fixtures were already consistent with it; none needed editing.

Overload resolution (Functions > Overload Resolution). Arity filter, then
assignability filter, then a per-argument conversion rank (exact, promotion,
widening, union member, implicit generic, any). A candidate wins by ranking no
worse on every argument and better on at least one. A tie is an error naming
both candidates, never broken by declaration order. Return types take no part,
so two declarations differing only in return type are a redefinition error.
Compiler-provided functions are ordinary set members. An unannotated literal
argument ranks 0 against its default type and 1 against any other type it
fits, so adding an overload cannot change which candidate an existing call
already resolved to.

Type tests (Type Tests). `is` is a runtime test, but a struct carries no tag,
no vtable pointer and no RTTI, so it reads a discriminant that already exists
rather than asking a general question: its operand must be a union or an
interface reference, and testing a concretely typed value is refused rather
than folded to a constant. It narrows both branches, and narrowing an
interface reference yields *T, since the reference was a pointer and a table
and never a value.

Two consequences of that recorded where they belong. Unions carry a tag,
elided where an existing value serves as one, so ?*T keeps the pointer-sized
layout the spec already guarantees (Unions > Representation). And the global
implementation table survives into the running program, to answer whether the
concrete type behind a reference implements another interface - the only
runtime lookup in the language, paid for by a static per-type descriptor on
each interface table rather than by anything in the object (0-overview.md >
Object Model, which had called the registry compile-time-ish).

Five fixtures, three positive and two negative, all registered in the manifest.
All lex and parse; they fail the compile phase, which does not exist yet.

sema: decompose semantic analysis into atomic testable steps (docs/semantics/) 12c0fedd2b

The working map from "decide what a parsed Catlang program means" down to
RED -> GREEN -> VERIFY units, in the shape docs/parser/ used for the phase
before it. Eleven steps against the 21 fixtures under tests/fixtures/semantics
(18 positive, 3 negative), each step naming exactly which it completes.

Four decisions shape the phase and are recorded so a step can cite rather than
relitigate them.

The workspace is re-cut. ast is a module of parser and Span is a module of
lexer, so a type checker that wants to point at source would depend on a
scanner and a code generator would depend on a parser. Step 00 extracts source
(Span, Position, SourceBuffer, and a new shared Diagnostic) and ast, leaving
every dependency pointing at something more fundamental than itself. The moves
are mechanical and history-preserving, and land as their own commits ahead of
any content change.

Semantic analysis produces a typed HIR rather than an annotated AST. Every hir
expression carries a Type, not an Option<Type> and not a side table entry that
can be missing, so an untyped node after this phase is unrepresentable. It is
also where desugaring lands - a span subscript becomes a slice construction,
and every implicit conversion becomes an explicit node, so code generation
never re-derives the conversion rules in a second place that can disagree with
the first. What needs a temporary (?., ??, !) stays a node until a later
lowering.

Diagnostics accumulate rather than short-circuit. One error then stop is right
for a parser, whose next token is meaningless after a syntax error, and wrong
for a type checker, where a second unrelated type error is perfectly
diagnosable.

The // ERROR: marker becomes an assertion. AGENTS.md already requires a
negative fixture to mark its offending line; the fixture driver now reads it,
so a negative fixture rejected for the wrong reason or on the wrong line fails.

Two items stay open and are labeled with the step they gate: size_t has no
width because there is no target model (proposed: one TargetLayout constant),
and a bare type name in value position needs a typing rule written down. Two
more are deliberately out of phase and recorded so the omissions are not
mistaken for oversights: interface conformance checking, and the fact that
string interpolation holes are never parsed by anything today.

build(hook): compare the suite against .known-red, not against green 123bd2b2fb

A pipeline phase opens with its fixture driver failing every fixture. That is
the contract AGENTS.md sets - RED before GREEN - and it is what the semantics
phase is about to do, at 21 of 21. The pre-commit hook's third check demanded a
uniformly green `cargo test --all`, so a phase that follows the contract could
not be committed in steps at all.

Check 3 now asks a different question: are the failures exactly the ones
`.known-red` names? The comparison runs both ways, and the second direction is
the point. A test that fails without a line blocks the commit, as before. A test
that passes while still holding a line also blocks it, asking for the line to be
deleted, so an exemption cannot outlive the red it covers. Deleting the last
line restores the plain green-suite rule with no further change to the hook.

The suite runs with --no-fail-fast so every target reports, and a non-zero exit
with no failing test names is treated as a broken build rather than an empty
exemption set.

parser(step 00a): null is a base type name, not only a literal word 618cd34ffc

Spec phase: parse. Fixture: tests/fixtures/semantics/type-tests.cat.

The Unvalued table lists `null` as a type beside `c_void`, and `is` takes a
`type-atom`, so `u is null` - the only explicit spelling of a union's null test,
and the one the spec's own Narrowing section writes - has to parse. It did not:
`base_kw_spelling` omitted the keyword, so `null` was a word `primary` accepted
and nothing else. The fixture had the line removed to work around it.

One line of parser, the grammar's base-type-name list, and the fixture line
restored. The grammar note that explains why `const` appears twice now covers
`null` for the same reason: one keyword in two roles.

`?T` is deliberately left alone. The spec's claim that it is `T | null` is a
claim about types, and the semantics phase README already places that collapse
at lowering, so `?int` still parses to `(opt int)` and `int | null` to
`(| int null)`. A parser that unified them here would be deciding what a program
means. The plan's acceptance criteria asserted the opposite, and asserted `int`
would dump as `s32`; both are corrected in docs/semantics/00-scaffold.md with
the reasoning, because `base_kw_spelling` records a canonical spelling and `int`
and `s32` stay distinct TypeRefs until step 10 makes them one Type.

Also drops a stale reference to `is_base_kw`, a function that no longer exists,
from the doc comment this commit was already rewriting.

RED: null_is_a_type_atom, and the restored fixture line failing the parse driver
at type-tests.cat:12:24 with "expected a type".

sema(step 00b): extract source from lexer 1b59359a22

Spec phase: none. No fixture; this is scaffolding.

`Span` lived in `lexer`, so `ast::TypeRef` wrote `use lexer::source::Span` and
the syntax tree depended on the scanner for nothing but a pair of usizes. Left
alone, adding a type checker on top would make the dependency graph say a type
checker needs a tokenizer, and adding code generation would say code generation
needs a parser. Neither is true, and the cost of fixing it only grows, so it
happens before any sema code exists to be re-pointed.

`source` has no dependencies and never will. `span.rs` and `buffer.rs` are the
old `lexer/src/source.rs` split at the seam it already had, moved with history.
`lexer` and `parser` change only their `use` paths. Zero behaviour change, which
`cargo test --all` proves at the same 105 tests as before.

`Diagnostic` is the one piece of new content, and it is new rather than moved
because there is nothing to move: `LexError` and `ParseError` are separate ad
hoc structs and this phase needs a third. They are NOT rewritten in terms of it.
A scanner and a parser each report one error and stop, and `Result<_, Their>`
says so in the signature; semantic analysis has no such shape, because a second
unrelated type error is perfectly diagnosable. So the two convert instead, which
keeps their contracts intact and still gives the eventual CLI one printer.

Both conversions are tested for the thing that actually breaks in a From impl,
which is a dropped span.

sema(step 00c): extract ast from parser d65f1a0794

Spec phase: none. No fixture; this is scaffolding.

Verbatim, and git records it as a rename with a one-line diff: the `use` path
that 00b changed under it. The 10 golden dump tests move with the file and pass
unchanged, which is the evidence the move was mechanical.

The syntax tree is not the parser's private property. Semantic analysis reads it
and code generation will read what semantic analysis produces from it, so a tree
that lives inside its builder makes every later consumer depend on the builder.
It now depends on `source` and nothing else.

No `pub use ast;` re-export in `parser`. The plan called for one so existing
`parser::ast::Expr` paths would keep working, but there was exactly one such
path - parser.rs's own import - and it is migrated in this commit. A re-export
with no consumers is a compatibility shim for nobody, and `parse()` returning an
`ast::Program` is already the honest signal that a caller wanting to name that
type depends on `ast`, which is what `sema` will do.

sema(step 00d): split ast into one module per node family ad1726c19d

Spec phase: none. No fixture; this is scaffolding.

1343 lines in one lib.rs was reasonable as a module of a crate and is not
reasonable as a crate. The split follows the section comments the file already
carried, so the boundaries are the ones its author drew: name, ty, op, expr,
stmt, decl, item, and dump.

`dump` earns its own module for a different reason than the rest. `sexp` and the
sixteen list helpers were private free functions reached from every family,
which is exactly the shape that ends with two families printing the same node
differently. They are `pub(crate)` in one place now, and the golden tests move
with them, because the dump is the parser's test contract and this is where that
contract lives.

The tree stays flat at the crate root. Callers write `ast::Expr`, not
`ast::expr::Expr`, because which module a node lives in is this crate's business
and not theirs.

Pure moves plus `use` lines, proven rather than asserted: a line-set diff of the
pre-split file against the eight modules, ignoring comments and `use`
statements, is empty in both directions. The 10 golden tests still pass.

sema(step 00e): the sema crate and the compile-phase fixture driver (RED) 7d9b95b7d8

Spec phase: compile. Fixtures: all 21 under tests/fixtures/semantics/.

`check(&ast::Program, &SourceBuffer) -> Result<hir::Program, Vec<Diagnostic>>`,
with a body that returns one diagnostic saying semantic analysis is not
implemented. That is the whole crate. The signature is the deliverable: it says
semantics lowers into a typed HIR rather than annotating the AST in place, and
it says diagnostics accumulate rather than short-circuit, which are decisions 2
and 4 of the phase README. `hir::Program` is a stub carrying only its span;
step 20 declares the node families.

The driver asserts more than the lexer's and the parser's do, because a
semantics fixture is not only a program that must be accepted. Three of them are
refusals, and AGENTS.md already requires each to mark its offending line with a
trailing `// ERROR:` comment. That marker is now an assertion: an unmarked
fixture must type-check, and a marked one must be rejected on exactly the marked
lines. Without the second half, a negative fixture passes whenever the checker
rejects it at all, including for an unrelated reason on an unrelated line, which
is the failure a negative test exists to rule out. Only error-severity
diagnostics are compared, so a future warning on an unrelated line cannot break
a negative fixture.

RED: 21 of 21, every one of them "semantic analysis is not implemented". That
count is the phase's progress bar, so the driver is not #[ignore]d; `.known-red`
carries the exemption and will refuse to outlive it.

Non-vacuity, run by hand against three throwaway `check` bodies rather than one:

  always-Err on line 1  -> 21 fail; the 3 negatives fail on the distinct
                           line-mismatch message, not the generic one
  always-Ok             -> 18 pass; the 3 negatives fail with
                           "accepted, but line 3 is marked"
  reject-on-marker      -> 21 pass

So the all-green state is reachable, the Ok arms are exercised, and the line
mapping is right for owned-rebinding.cat, which is the one fixture with CRLF
endings.

build(hook): a known-red line is stale only if its test actually reported ok 3e5f44b072

The first version compared two sets of names and treated any listed test absent
from the failures as fixed. That is wrong for the case this repo hits on every
commit before the driver exists: a line naming a test that has not been written
yet, or one behind a target the run did not build, reported as "now passes" and
blocked the commit.

Absent and passing are different facts. The check now looks for the test
reporting `ok` in the run's own log, so a line goes stale only when its test
really went green. A line waiting for its test is left alone, and a test that
fails without a line is still caught by the other direction.

Found while checking out each commit of step 00 and running it, which is also
how 1b59359 turned out not to build: the hook validates the working tree, and
the working tree was at the finished state for all six commits. History stays as
it is - the branch squash-merges - but the gap is worth naming, because a hook
that only ever sees the tip cannot vouch for the commits under it.

crowmaki changed title from ~~types~~ to Scaffold semantic analysis

2026-08-24 02:26:34 +00:00

crowmaki merged commit c5a30503b7 into main

2026-08-24 02:26:41 +00:00

crowmaki deleted branch types

2026-08-24 02:26:41 +00:00

crowmaki referenced this pull request from a commit

2026-08-24 02:26:43 +00:00

Scaffold semantic analysis (#14)