Capture real work
Good evaluation tasks begin with work that actually occurs. Merion captures a WorkEpisode: a versioned record connecting a request, starting state, execution context, trajectory evidence, and human-reviewed outcome.
Before work begins
Run:
Merion records the task before the answer is known and creates a content-addressed local snapshot for every configured source.
For Git sources, the snapshot records repository head, status, and diff statistics. For ordinary directories, it records a bounded manifest of paths, sizes, modification times, and local content digests.
The snapshot deliberately excludes .merion, .git, dependency trees, and common build outputs. It does not follow symlinks or copy source-file contents into the manifest.
During the work
Continue through Codex, OpenCode, a manual process, or another harness. Preserve evidence that helps explain the outcome:
- tool calls and external actions;
- corrections and retries;
- human interventions;
- relevant artifacts and changed state;
- runtime, cost, and latency when available.
The existence of a long trace does not make the episode valuable. The outcome must still be reviewable, representative, and tied to an economically meaningful task.
Close with human outcome evidence
Run:
The reviewer records:
- whether the result was accepted, corrected, rejected, or abandoned;
- observable success criteria;
- unacceptable outcomes;
- the corrections that were required;
- whether the episode represents normal work;
- recurrence and economic value.
Merion does not infer these decisions from a plausible-looking final answer. A domain expert remains accountable for defining what good and bad mean.
After review, run merion next. Merion asks for related examples before suggesting evaluation formation. The current workflow requires at least three reviewed episodes before opening the task-formation path.
Continue with forming an evaluation.Last modified on August 21, 2026
