Challenge Passing Agent Tests with a Missing Case

When an agent writes both code and tests, a green result answers a narrow question: did that implementation satisfy those assertions on that run? To judge the change, also ask whether those assertions would reject a plausible implementation that violates your requirement.

Start with a rule you can read independently of the code. Then choose an input on which a correct implementation and a plausible wrong one must disagree. The missing case is more useful than a general request for “better coverage.”

Write the rule before reading the tests

The fictional requirement LIST-3 formats a changed-file list for display:

• Input is an array of nonempty display-label strings without carriage returns or newlines.
• Each input produces exactly one line: dash, space, unchanged label, then LF.
• Preserve the supplied order and repeated labels. Do not sort, deduplicate or drop later items.
• An empty input produces the empty string, with no heading or blank line.

These are original teaching requirements, not rules inferred from a particular repository or filename convention. Arbitrary filesystem names and invalid input types are outside this example's input contract.

Here is the correct tiny implementation:

export function correct(files) {
  return files.map(name => `- ${name}\n`).join('');
}

Two distinct wrong alternatives are:

export function firstOnly(files) {
  return files.length ? `- ${files[0]}\n` : '';
}

export function sortedUnique(files) {
  return [...new Set(files)].sort().map(name => `- ${name}\n`).join('');
}

firstOnly drops every later entry. sortedUnique keeps a plausible-looking list but changes order and removes repeated entries. Their failures are visible from LIST-3 before any test execution.

Apply the same weak checks to all three

In this snippet, format is whichever implementation is under review. The local runner applies these same four checks to each one.

import assert from 'node:assert/strict';

// W1: empty output has no malformed bullet lines.
assert.ok(format([]).split('\n').filter(Boolean)
  .every(line => line.startsWith('- ')));

// W2: one familiar item works.
assert.equal(format(['src/app.js']), '- src/app.js\n');

// W3: a multi-item result contains the first familiar name.
assert.ok(format(['src/app.js', 'src/view.js'])
  .includes('src/app.js'));

// W4: a stub returns its hard-coded response.
const stub = () => '- src/app.js\n';
assert.equal(stub(['src/app.js']), '- src/app.js\n');

W1 is vacuous for an empty filtered list: there is no line that can violate the predicate. It does not establish the exact empty-output requirement. W2 never exercises a later item. W3 checks presence of one name, rather than the complete ordered result. W4 never calls format at all; a useful stub test elsewhere cannot certify this formatter merely by returning its own expected value.

Node's version-tagged assertion documentation describes strict assertion mode and equality checks. Strict comparison can enforce the expected string you provide; it cannot decide whether that expected string expresses LIST-3.

The weak checks were actually run using Node.js v24.18.0, with no network or installed test dependencies:

The green result is accurate. Its interpretation as evidence of order and duplicate preservation would be inaccurate.

Derive a missing case from the requirement

Before running stronger checks, write literal expected strings from LIST-3. In the table, \n represents one LF character; it is not two output characters.

C2 is the smallest useful challenge here. It includes a later item and starts in an order that sorting would change. C3 separately requires repetition to survive; C4 checks exact labels with spaces and non-ASCII text. C5 checks a longer sequence.

The complete data file, B84-02-cases.json, stores those literal expectations. The improved check for each case is:

for (const c of cases) {
  assert.equal(format(c.input), c.expected);
}

Do not generate c.expected by calling the implementation under test or copying its returned value into the fixture without checking the rule. That would remove the independent disagreement you are trying to expose.

Compare predictions with actual execution

The predicted result was written from the requirement and the three functions. Running the cases produced this same matrix:

For C2, firstOnly actually returns "- src/z.js\n"; sortedUnique returns "- src/a.js\n- src/z.js\n". Each contradicts a different part of the same expected string.

The local runner uses separate process invocations for each implementation. The two wrong implementations exit with code 1 against the improved suite. Their assertion failures were retained in the accompanying run logs; they were not removed to make the package appear universally green. The worked execution results records runtime, cases, exit codes and log filenames. These executions test the displayed functions and fixed cases, not a customer project or an exhaustive correctness claim.

Challenge the new assertion too

Suppose the agent proposes this additional test:

assert.equal(
  format(['src/z.js', 'src/a.js']),
  '- src/a.js\n- src/z.js\n'
);

It expects sorting, which LIST-3 forbids. Actual execution rejects correct and accepts sortedUnique on this case. A new failure is therefore not sufficient reason to change the implementation: the test's expected value needs a requirement source as well.

Remove or correct that assertion if LIST-3 remains the agreed requirement. If the user actually wants sorted output, record a changed requirement and update the relevant examples deliberately. Do not silently change the rule to obtain a green suite.

Copy this passing-suite review request

Review the fictional changed-file formatter against LIST-3, not against
the implementation's current output. Use B84-02-cases.json and the three
complete functions in format-implementations.mjs.

First identify which requirement each existing weak assertion can establish.
Explain why the empty every() assertion is vacuous, why a singleton cannot
show later-item retention, and why the stub-only assertion does not exercise
the target. Preserve legitimate checks rather than deleting them wholesale.

Before execution, derive literal outputs for C0-C5 from LIST-3.
Apply the same exact-output assertions to correct, firstOnly and sortedUnique.
Return the implementation-by-case matrix, actual outputs for disagreements,
commands, runtime and retained raw failures. Label unrun predictions as such.

Inspect the proposed sorted-output assertion independently. It contradicts
the current order-preservation requirement. Do not edit LIST-3 or bless that
expectation merely because an implementation currently produces it.

Limit work to these new teaching files. No dependency installation, project
test-suite changes, customer-code reads or production execution. Return a
bounded test proposal; independent review and acceptance remain separate.
Do not infer exhaustive coverage or a general mutation score from two wrong
alternatives and six named cases.

A useful review can now name the missing distinction: “Show that the second item survives in its original position.” That is a concrete next check, with an expected answer that comes from the requirement rather than the agent's implementation.