Shuffle answer options (#5) and standard 'None of the above' choice (#6) #11

Merged

crowmaki merged 3 commits from fix/issue-5-shuffle-choices into main

2026-08-29 06:17:07 +00:00

crowmaki commented

2026-08-29 06:00:25 +00:00

Owner

Fixes #5 and fixes #6.

#5 — answer options need to be shuffled

Models put the correct choice first most of the time, leaking the answer
key through the generated order. All three question generators (tutor
checks, assignment questions, diagnostic questions) now shuffle their
substantive choices once at normalisation time — before anything is
persisted — via a new shuffle_choices helper in agents/_common.py.
Grading is by choice_id, so the key is unaffected.

The permutation is deterministic (seeded from the question's own
content), so replays, tests, and re-served questions are stable.

#6 — "None of the above" standard on every option set

Every generated multiple-choice question now carries a pinned
__none_of_the_above__ / "None of the above" choice, giving learners an
honest answer when a generated set contains no correct answer.

Design decisions:

It may be the correct answer — that is what makes it help: a set
accidentally generated without a correct answer stays gradeable. The
__dont_know__ sentinel still never is.
Required at generation, lenient at rest. A new shared
validators.validate_choice_set requires exactly one NOTA choice with
the fixed text for every model-generated question (the generate-validate
loop retries until it complies), while schemas.question_key_errors
only validates NOTA if present so persisted courses/attempts from
before this rule still load.
Pinned tail order everywhere (backend shuffle + desktop rendering):
content choices → "None of the above" → "I don't know".

Changes

plugin/schemas.py: NONE_OF_THE_ABOVE constants + cross-field checks
(unique, fixed text, allowed as correct_choice_id)
plugin/validators.py: validate_choice_set helper; validate_question
enforces NOTA presence; scoring already grades by choice_id
plugin/agents/: prompt rules for assignments + diagnostics, tutor
fallback protocol and turn_defects (checks now validated too), the
pedagogy-tutoring skill §2, and shuffle_choices pinning
desktop/: pinned-tail rendering in attempts, checks, and diagnostics;
ensureDontKnow backfills both pinned choices for older persisted data
docs/: design.md question-shape rule updated

Testing

Python: 448 passed (5 pre-existing Windows environment failures on
main remain: test_install.py, one test_events.py)
New tests/test_agents_shuffle.py: permutation, sentinel pinning,
determinism, key integrity, position scrambling, NOTA pinning,
key-on-NOTA grading
Desktop: typecheck, bundle lint, and all a11y tests pass; dist rebuilt

Fixes #5 and fixes #6. ## #5 — answer options need to be shuffled Models put the correct choice first most of the time, leaking the answer key through the generated order. All three question generators (tutor checks, assignment questions, diagnostic questions) now shuffle their substantive choices once at normalisation time — before anything is persisted — via a new `shuffle_choices` helper in `agents/_common.py`. Grading is by `choice_id`, so the key is unaffected. The permutation is deterministic (seeded from the question's own content), so replays, tests, and re-served questions are stable. ## #6 — "None of the above" standard on every option set Every generated multiple-choice question now carries a pinned `__none_of_the_above__` / "None of the above" choice, giving learners an honest answer when a generated set contains no correct answer. Design decisions: - **It may be the correct answer** — that is what makes it help: a set accidentally generated without a correct answer stays gradeable. The `__dont_know__` sentinel still never is. - **Required at generation, lenient at rest.** A new shared `validators.validate_choice_set` requires exactly one NOTA choice with the fixed text for every model-generated question (the generate-validate loop retries until it complies), while `schemas.question_key_errors` only validates NOTA *if present* so persisted courses/attempts from before this rule still load. - **Pinned tail order everywhere** (backend shuffle + desktop rendering): content choices → "None of the above" → "I don't know". ## Changes - `plugin/schemas.py`: `NONE_OF_THE_ABOVE` constants + cross-field checks (unique, fixed text, allowed as `correct_choice_id`) - `plugin/validators.py`: `validate_choice_set` helper; `validate_question` enforces NOTA presence; scoring already grades by `choice_id` - `plugin/agents/`: prompt rules for assignments + diagnostics, tutor fallback protocol and `turn_defects` (checks now validated too), the pedagogy-tutoring skill §2, and `shuffle_choices` pinning - `desktop/`: pinned-tail rendering in attempts, checks, and diagnostics; `ensureDontKnow` backfills both pinned choices for older persisted data - `docs/`: design.md question-shape rule updated ## Testing - Python: 448 passed (5 pre-existing Windows environment failures on `main` remain: `test_install.py`, one `test_events.py`) - New `tests/test_agents_shuffle.py`: permutation, sentinel pinning, determinism, key integrity, position scrambling, NOTA pinning, key-on-NOTA grading - Desktop: typecheck, bundle lint, and all a11y tests pass; dist rebuilt

crowmaki added 2 commits

2026-08-29 06:00:25 +00:00

fix(agents): shuffle answer choices so the key isn't reliably first a7c0956217

Models place the correct choice first most of the time, leaking the
answer key through the generated order (issue #5). Substantive choices
are now shuffled once at normalisation time — in tutor turns, assignment
questions, and diagnostic questions — before anything is persisted.
Grading is by choice_id, so the key is unaffected.

The permutation is seeded from the question's own content, so replays
and re-served questions are stable. The sentinel choice stays last,
matching the desktop contract (content choices first, sentinel last).

feat(agents,desktop): standard 'None of the above' choice on every question (issue #6 ) 2d9f5bc543

Multiple-choice option sets now always include a pinned
__none_of_the_above__ / 'None of the above' choice, giving learners an
honest answer when a generated set contains no correct answer.

- schemas: NONE_OF_THE_ABOVE constants; question_key_errors validates the
  fixed text when present and allows it as correct_choice_id; presence
  stays lenient so persisted documents from before the rule still load
- validators: validate_choice_set helper; validate_question and the
  tutor check loop require exactly one NOTA choice; it may be the key,
  the DONT_KNOW sentinel still may not
- agents: prompt rules (assignments, diagnostics, tutor fallback
  protocol) and the pedagogy-tutoring skill describe the new choice
- shuffle_choices (issue #5) pins NOTA before the dont_know sentinel
- desktop: pinned-tail rendering (content choices, then None of the
  above, then I don't know) in attempts, checks, and diagnostics;
  ensureDontKnow backfills both pinned choices for older data
- docs: design.md question-shape rule updated

crowmaki referenced this pull request from a commit

2026-08-29 06:15:18 +00:00

fix(review): PR #11 follow-ups

crowmaki added 1 commit

2026-08-29 06:15:18 +00:00

fix(review): PR #11 follow-ups 3246dfb765

- agents/_common: remove duplicated docstring-only shuffle_choices stub
  left by a merge artifact (shadowed the real definition, F811)
- schemas: enforce >= 2 substantive choices in question_key_errors so the
  lenient NOTA-presence rule cannot reintroduce a one-substantive
  question on persisted documents (legacy docs with 2 substantive +
  sentinel still pass)
- desktop: ensureDontKnow -> ensurePinnedChoices; rebuild the pinned tail
  (content, None of the above, I don't know) instead of appending, so
  legacy checks keep the sentinel last; a11y fixtures now supply the
  NOTA choice so the backend-supplied (no-duplication) path is covered
- tests: drop duplicate factory constants and the tautological shuffle
  assertion (unsorted input makes the permutation check meaningful)

crowmaki commented

2026-08-29 06:15:51 +00:00

Author

Owner

All three findings fixed in 3246dfb:

Duplicate shuffle_choices stub - deleted; the real definition (with the pinned-tail docstring) is now the only one.
Substantive floor - question_key_errors now counts non-pinned choices and rejects < 2 with needs at least 2 substantive choices, found N. Verified: legacy [a, b, __dont_know__] still passes, [a, __none_of_the_above__, __dont_know__] is rejected. one_substantive moved back into schema_rejects in tests/test_validators.py, and test_question_with_one_substantive_choice_rejected now asserts the substantive floor (not minItems) does the rejecting.
ensureDontKnow -> ensurePinnedChoices - now rebuilds the tail the way orderedChoices/choicesWithSentinel do (filter both pinned ids from content, re-append NOTA then sentinel last), so legacy [substantive, __dont_know__] checks render with the sentinel last. TranscriptView updated. All three renderers now share the same ordering contract.

Minors: duplicate factory constants deduped; the tautological shuffle assertion replaced with an unsorted input so the permutation check is meaningful; a11y fixtures (transcript check, both attempt questions, diagnostic question) now supply the NOTA choice from the "backend", so the updated assertions exercise the normal no-duplication path.

Python: 448 passed (same 5 pre-existing env failures). Desktop: typecheck clean, 6/6 tests, dist rebuilt.

All three findings fixed in 3246dfb: 1. **Duplicate `shuffle_choices` stub** - deleted; the real definition (with the pinned-tail docstring) is now the only one. 2. **Substantive floor** - `question_key_errors` now counts non-pinned choices and rejects `< 2` with `needs at least 2 substantive choices, found N`. Verified: legacy `[a, b, __dont_know__]` still passes, `[a, __none_of_the_above__, __dont_know__]` is rejected. `one_substantive` moved back into `schema_rejects` in tests/test_validators.py, and `test_question_with_one_substantive_choice_rejected` now asserts the substantive floor (not minItems) does the rejecting. 3. **`ensureDontKnow` -> `ensurePinnedChoices`** - now rebuilds the tail the way `orderedChoices`/`choicesWithSentinel` do (filter both pinned ids from content, re-append NOTA then sentinel last), so legacy `[substantive, __dont_know__]` checks render with the sentinel last. TranscriptView updated. All three renderers now share the same ordering contract. Minors: duplicate factory constants deduped; the tautological shuffle assertion replaced with an unsorted input so the permutation check is meaningful; a11y fixtures (transcript check, both attempt questions, diagnostic question) now supply the NOTA choice from the "backend", so the updated assertions exercise the normal no-duplication path. Python: 448 passed (same 5 pre-existing env failures). Desktop: typecheck clean, 6/6 tests, dist rebuilt.

crowmaki merged commit 94f40969bb into main

2026-08-29 06:17:07 +00:00

crowmaki deleted branch fix/issue-5-shuffle-choices

2026-08-29 06:17:07 +00:00

crowmaki referenced this pull request from a commit

2026-08-29 06:17:09 +00:00

Shuffle answer options (#5) and standard 'None of the above' choice (#6) (#11)