Roundtrip Test Matrix

This document tracks the intended roundtrip-test surface for single-file fits. It is meant to answer three questions at a glance:

Which workflow API is under test?
Which backend expectation is under test?
Which supported model family is under test?

Project-level fitting and MCMC/parallel execution are important too, but they are tracked as secondary dimensions so the main matrix stays readable.

Axes

Workflow axis

B: File.fit_baseline()
Sp: File.fit_spectrum()
SbS: File.fit_slice_by_slice()
2D: File.fit_2d()

Scope axis

SF: single-file workflows
P: project-level workflows via Project.fit_2d()

Backend axis

M: force MCP with project.spec_fun_str = "fit_model_mcp" and recover truth
G: run default compiled dispatch and recover truth on the GIR path
C: assert delta(MCP, GIR) = 0

For C, prefer fit_model_compare when the workflow supports it. Otherwise use direct parity checks on residuals or evaluated outputs.

Cell notation

M/G/C: all three test types should exist for that workflow/model-family pair
-: not applicable for that pair

Execution-mode qualifiers

These are not a full extra matrix axis; they are required focused variants on top of the main matrix:

Opt: normal optimizer-based fit, no MCMC
MC1: MCMC enabled with workers=1
MC2: MCMC enabled with workers=2
W1: explicit serial execution for a workflow with worker support
W2: parallel execution with workers=2

Use 2 as the standard parallel test setting. We usually care about 1 versus >1, not about many-worker scaling in CI.

Canonical model families

Each row below should have one canonical fixture. Closely related variants can be parameterized under the same row instead of getting separate rows.

ID	Model family	Representative fixture(s)
`F1`	Plain energy model	`single_glp`, `glp_only`
`F2`	Static expressions in energy model: direct refs, fan-out, forward refs	`two_glp_expr_amplitude`, `expression_fan_out`, `energy_expression_forward_reference`, `glp_expression`
`F3`	Top-level standard dynamics	`single_glp` + `MonoExpPos`
`F4`	Top-level dynamics with IRF / convolution	`single_glp` + `MonoExpPosIRF` and other lowerable IRF kernels
`F5`	Top-level subcycle / multi-cycle dynamics	`single_glp` + `["ModelNone", "MonoExpNeg", "MonoExpPosExpr"]`, `frequency=10`
`F6`	Top-level profile only	`single_gauss` + `roundtrip_pLinear_x0` / `roundtrip_pExpDecay_A`
`F7`	Top-level profile plus separate top-level dynamics on another parameter	`single_gauss` + profile on `Gauss_01_A` + dynamics on `Gauss_01_x0`
`F8`	Profile-internal dynamics	`single_gauss` + profile on `Gauss_01_A` + dynamics on `Gauss_01_A_pExpDecay_01_A`
`F9`	Expression parameter referencing a top-level time-dependent base parameter	`two_glp_expr_amplitude` + dynamics on `GLP_01_A`
`F10`	Expression parameter referencing a top-level profiled base parameter	`two_glp_expr_amplitude` + profile on `GLP_01_A`
`F11`	Expression parameter referencing a profiled parameter whose profile internals are time-dependent	`two_glp_expr_amplitude` + profile on `GLP_01_A` + dynamics on `GLP_01_A_pExpDecay_01_A`
`F12`	Mixed expression referencing both profiled and time-dependent base parameters	`two_glp_mixed_profile_dynamics` with profile on `GLP_01_A` and dynamics on `GLP_01_x0`

Target matrix

Scope: SF (single-file)

Family	B	Sp	SbS	2D	Notes
`F1` Plain energy	`M/G/C`	`M/G/C`	`M/G/C`	`-`	Core 1D workflow coverage
`F2` Static expressions	`M/G/C`	`M/G/C`	`M/G/C`	`-`	Include at least one direct-ref case and one fan-out or forward-ref case
`F3` Standard dynamics	`-`	`-`	`-`	`M/G/C`	Core 2D dynamic family
`F4` IRF dynamics	`-`	`-`	`-`	`M/G/C`	One canonical roundtrip plus parametrized parity across kernels
`F5` Subcycle dynamics	`-`	`-`	`-`	`M/G/C`	Important for multi-cycle indexing and expression prefixing
`F6` Profile only	`M/G/C`	`M/G/C`	`M/G/C`	`-`	Covers aux-axis plumbing in 1D APIs
`F7` Profile + separate dynamics	`-`	`-`	`-`	`M/G/C`	Top-level mixed feature case
`F8` Profile-internal dynamics	`-`	`-`	`-`	`M/G/C`	Single-cycle only
`F9` Expr -> time-dependent base par	`-`	`-`	`-`	`M/G/C`	High-value bug class for update ordering and pickling
`F10` Expr -> profiled base par	`M/G/C`	`M/G/C`	`M/G/C`	`-`	Expression namespace must see profiled values
`F11` Expr -> profiled base par with profile-internal dynamics	`-`	`-`	`-`	`M/G/C`	Single-cycle only
`F12` Mixed expr(profile + dynamics refs)	`-`	`-`	`-`	`M/G/C`	Stress case for combined namespace resolution

Project-level matrix

Scope: P (Project.fit_2d())

Project-level fitting should be tracked separately because it is currently wired through fit_project_mcp, so the single-file GIR/MCP expectations do not apply cleanly yet.

Family	2D	Notes
`PF1` Shared plain dynamics across files	`M`	Current core project roundtrip surface
`PF2` Project-level expressions	`M`	Includes file/project prefix rewriting and shared refs
`PF3` Shared dynamics with IRF	`M`	Covered with `BiExpProject` + `gaussCONV`
`PF4` Shared subcycle dynamics	`M`	Add once project fixtures exist

Future:

if project-level GIR lands, upgrade applicable cells from M to M/G/C
until then, do not force fake GIR coverage into the project matrix

MCMC and worker policy

Yes: MCMC should be tracked.

Yes: worker mode should be tracked anywhere the code can execute differently between serial and parallel paths.

But neither should multiply every cell in the main matrix. Instead use focused requirements:

MCMC requirements

MCMC is a second-layer contract on top of the clean optimizer roundtrips.

Minimum MCMC set:

MC1: one canonical B test on F1
MC2: one canonical B test on F1
MC2: one expression-sensitive case, preferably F9 or F10
MC2: one 2D varying case, preferably F3 or F8

Rationale:

MC1 checks that MCMC itself still works
MC2 checks pickling / process-boundary behavior
expression-heavy and nested-model cases are the highest-value bug surfaces

SbS worker requirements

Yes, SbS should distinguish W1 and W2 because n_workers=1 uses the serial path and n_workers>1 crosses a process boundary.

Current status:

the main SbS matrix runs with n_workers=1
focused W2 tests cover F1 and profile-bearing F6 with n_workers=2

Future requirement if worker-specific risk grows:

add a more expression-heavy W2 SbS case, likely F2, if process-boundary risk shows up beyond the existing plain/profile cases

Project worker requirements

For project-level fits, add worker variants only when project execution gains a parallel path that is semantically different from serial execution.

Practical rule

Use this rule to decide whether a new dimension deserves explicit tracking:

add it as a full matrix axis only if it changes almost every cell
otherwise add it as a focused secondary requirement

By that rule:

project-level fits: yes, but separate matrix
MCMC: yes, as focused secondary coverage
workers=1 vs workers=2: yes, but only for APIs that actually expose worker-dependent behavior

Minimum test shape per cell

For each required cell, the minimum useful test is:

simulate noiseless data from a truth model
fit through the target workflow API
assert recovered non-expression parameters match truth
for C, assert MCP and GIR agree exactly or within a tight tolerance

For MCMC-focused cells, the minimum useful test is:

run the fit with mc_settings.use_mc=1
assert no crash for MC1
assert no crash and no pickling/serialization failure for MC2
when runtime allows, also assert basic parameter recovery or constraint preservation

Noisy roundtrip tests are still valuable, but should be a second layer. The clean matrix above is the baseline contract.

Current snapshot

This is the current high-level state of the suite, not a substitute for the table above.

Covered today:
- the full single-file clean matrix above for M/G/C
- F2 variants for direct, fan-out, and forward-reference expressions
- noisy second-layer checks for F3, F6, and F8 on the GIR path
- focused MCMC checks for MC1, MC2, expression-sensitive MC2, and 2D MC2
- focused W2 coverage for plain and profile-bearing fit_slice_by_slice()
- project-level M roundtrips for PF1, PF2, and PF3
Thin or missing today:
- project-level PF4 shared subcycle dynamics
- project-level G/C coverage, because project fitting is still MCP-only
- expression-heavy W2 coverage for fit_slice_by_slice()
- MCMC assertions beyond no-crash / process-boundary coverage
- exhaustive noisy coverage, intentionally kept out of the main matrix

Suggested implementation order

The original single-file matrix is implemented. Highest-value next steps:

Add PF4 once a shared project-subcycle fixture exists.
Add a focused expression-heavy W2 SbS test if process-boundary risk shows up beyond the existing plain/profile cases.
Add lightweight recovery or constraint-preservation assertions to focused MCMC tests when runtime allows.
Upgrade project-level cells from M to M/G/C if project-level GIR lands.

Non-goals

This matrix does not try to track:

invalid / explicitly unsupported model combinations
low-level evaluator unit tests
plotting-only behavior

Those should stay in their existing focused tests.