Implement parser steps 00-20: crate scaffold, AST + dump, type grammar #6

Merged

crowmaki merged 5 commits from parser/implementation into main

2026-08-21 21:45:10 +00:00

crowbot commented

2026-08-21 21:42:30 +00:00

Collaborator

Parser implementation — steps 00, 10, 20

Implements the first three steps of the docs/parser/ decomposition on the parser/implementation branch. The parse phase's crate boundary, AST shape, golden-dump test contract, and the type-reference grammar are in place; statements/expressions/functions/user-types/top-level (steps 30–70) come next.

Commits

parser: scaffold the parser crate under the workspace (step 00) — new parser/ workspace member linking against lexer; parse(tokens) -> Result<Program, ParseError> entrypoint; empty-program test passes from day one.
parser: declare AST backbone + golden dump inspector (step 10) — full AST node families (decl/type/stmt/expr) as tagged structs, every node carrying its source Span, and the recursive dump() S-expression inspector that makes every later step's unit test a one-line golden assertion (the parser's equivalent of the lexer's check_kind).
parser: type-reference grammar over tokens (step 20) — parse_type_ref() and the binding chain behind every annotation: type unions (A | B | C, loosest), the ?/*/[] prefixes (* binds tighter than |), static [N]T vs dynamic []T/[..]T arrays, named/qualified types with optional <T, U> generics, tuple types (T1, U), and function types (T1,U) -> R.
parser: dump array types as ([] T) / ([N] T) — finalizes the array dump spelling.

Design decision (settled in review)

Span is a standard-library type name, not a reserved keyword. The old lexer reserved lowercase span — but the language's actual spelling (every fixture, the grammar's type production, the spec's Slices and Ranges) is Span, which arrived as an ordinary identifier anyway. So:

Dropped KeywordKind::Span and the lexer's "span" keyword mapping.
Span<T> / bare Span now parse as a plain Named type, exactly like string (also an un-reserved aliased type).
Identifiers are case-sensitive: lowercase span is a distinct name.
docs/prd/language-grammar.md and docs/parser/20-type-refs.md updated to record the decision.

Test contract

cargo test at the workspace root is green (9 lexer + 8 parser tests). Every type alternative in step 20 is pinned by a golden dump (int, *int, ?*int, []int, [..]int, [25]int, Span<u8>, unions, tuples, function types), following the RED → GREEN → VERIFY discipline from AGENTS.md.

Notes

The parse fixtures (32 in tests/fixtures/parse/) stay red until step 70 wires top-level assembly; steps 30–70 land in follow-up PRs.
No lexer behavioral changes beyond the Span de-reservation.

## Parser implementation — steps 00, 10, 20 Implements the first three steps of the `docs/parser/` decomposition on the `parser/implementation` branch. The parse phase's crate boundary, AST shape, golden-dump test contract, and the type-reference grammar are in place; statements/expressions/functions/user-types/top-level (steps 30–70) come next. ### Commits - **`parser: scaffold the parser crate under the workspace (step 00)`** — new `parser/` workspace member linking against `lexer`; `parse(tokens) -> Result<Program, ParseError>` entrypoint; empty-program test passes from day one. - **`parser: declare AST backbone + golden dump inspector (step 10)`** — full AST node families (decl/type/stmt/expr) as tagged structs, every node carrying its source `Span`, and the recursive `dump()` S-expression inspector that makes every later step's unit test a one-line golden assertion (the parser's equivalent of the lexer's `check_kind`). - **`parser: type-reference grammar over tokens (step 20)`** — `parse_type_ref()` and the binding chain behind every annotation: type unions (`A | B | C`, loosest), the `?`/`*`/`[]` prefixes (`*` binds tighter than `|`), static `[N]T` vs dynamic `[]T`/`[..]T` arrays, named/qualified types with optional `<T, U>` generics, tuple types `(T1, U)`, and function types `(T1,U) -> R`. - **`parser: dump array types as ([] T) / ([N] T)`** — finalizes the array dump spelling. ### Design decision (settled in review) **`Span` is a standard-library type name, not a reserved keyword.** The old lexer reserved lowercase `span` — but the language's actual spelling (every fixture, the grammar's type production, the spec's Slices and Ranges) is `Span`, which arrived as an ordinary identifier anyway. So: - Dropped `KeywordKind::Span` and the lexer's `"span"` keyword mapping. - `Span<T>` / bare `Span` now parse as a plain `Named` type, exactly like `string` (also an un-reserved aliased type). - Identifiers are case-sensitive: lowercase `span` is a distinct name. - `docs/prd/language-grammar.md` and `docs/parser/20-type-refs.md` updated to record the decision. ### Test contract `cargo test` at the workspace root is green (9 lexer + 8 parser tests). Every type alternative in step 20 is pinned by a golden dump (`int`, `*int`, `?*int`, `[]int`, `[..]int`, `[25]int`, `Span<u8>`, unions, tuples, function types), following the RED → GREEN → VERIFY discipline from `AGENTS.md`. ### Notes - The parse fixtures (32 in `tests/fixtures/parse/`) stay red until step 70 wires top-level assembly; steps 30–70 land in follow-up PRs. - No lexer behavioral changes beyond the `Span` de-reservation.

crowbot added 5 commits

2026-08-21 21:42:30 +00:00

test(parse): RED fixture for tuple args & explicit spread dca3e6eefc

Adds tests/fixtures/parse/tuple-spread.cat and a feature-manifest row
(phase: parse) for the tuple-vs-argument-list rule and explicit tuple
spread, per the TDD contract (no production code without a fixture).

RED: no parser exists yet; the fixture registers the expected syntax
per docs/prd/language-spec.md > Tuples > In argument lists.

parser: scaffold the parser crate under the workspace (step 00) 9d04825b60

Add parser/ as a new workspace member crate that links against lexer and
exposes the parse(tokens) -> Result<Program, ParseError> entrypoint. Step 00
only proves the crate builds and its golden test harness runs: parse returns
an empty Program for empty input, and a structured ParseError is in place for
the grammar steps to throw. docs/parser/00-scaffold.md.

parser: declare AST backbone + golden dump inspector (step 10) 6936c94e54

Populate parser/src/ast.rs with the full AST shape: the decl/type/stmt/expr
node families as tagged structs, every node carrying its source Span, and the
recursive dump() rendering each family to a canonical S-expression. This is
the parser's test contract: golden assertions become one-line string compares.
Also extend the test harness with dumps_empty_program/dumps_var_declaration
wrapping the program dump. No grammar yet (step 20 starts it).

parser: type-reference grammar over tokens (step 20) c95cc65c14

Add parse_type_ref() and the binding chain behind every annotation: type
unions (A | B | C, loosest), the ?/*/[] prefixes (opt/ptr/arr, * binds tighter
than |), static [N]T vs dynamic []T/[..]T arrays, named/qualified types with
optional <T, U> generic args, tuple types (T1, U), and function types
(T1,U) -> R.

The Cursor now keeps the source text so identifier names (the lexer keeps
only spans) can be recovered, and a lex_source helper feeds the golden tests;
each type alternative is pinned by a golden dump.

Also settles Span as a standard-library type name (user call), not a keyword:
- Remove KeywordKind::Span + the lexer's reserved lowercase 'span' mapping
  (it protected a spelling nothing wrote; the spec's actual type is 'Span').
- Span<T>/bare Span now parse as an ordinary Named type, like 'string'.
- Update docs/prd/language-grammar.md and docs/parser/20-type-refs.md to
  record the decision. Identifiers stay case-sensitive.

parser: dump array types as ([] T) / ([N] T) 98b7517499

Render []T/[..]T as ([] T) and static [N]T as ([N] T), matching the step-60
fixture spelling, instead of the old (arr T)/(arr N T). Updates the golden
unit tests and the step-20 doc example.