When Local Tests Pass but CI Fails: Give the Agent an Environment Comparison

When a local check succeeds and CI fails, give the coding agent a comparison of the two runs before asking for a fix. Start with the same checked-out commit and test selection, then compare the working directory, runtime, dependency state, inputs, and workflow event. Use the differences to choose one hypothesis that a focused check could disprove.

GitHub's workflow troubleshooting guide points to run logs, workflow triggers, execution conditions, and runners as investigation surfaces. The comparison below is an original way to organize those surfaces into an agent request. It does not establish the cause of a real CI failure.

Keep the records tied to the same code

The project, configuration, paths, run records, and observations below are fictional. The records are written examples, not captured logs or commands executed for this article. EXAMPLE-C17 is a symbolic commit label, not an actual Git object ID.

Suppose a repository contains two schema fixtures:

shop/
  fixtures/
    export-schema.json
  packages/
    exporter/
      fixtures/
        export-schema.json
      tests/
        test_schema.py
  ci/
    task-description.txt

The root fixture is an older example:

{"id_field": "orderId"}

The fixture inside packages/exporter/ is:

{"id_field": "order_id"}

The relevant part of packages/exporter/tests/test_schema.py is:

import json
from pathlib import Path
import unittest


class SchemaTests(unittest.TestCase):
    def test_export_id_field(self):
        schema = json.loads(
            Path("fixtures/export-schema.json").read_text(encoding="utf-8")
        )
        self.assertEqual(schema["id_field"], "order_id")


if __name__ == "__main__":
    unittest.main()

The assertion expresses the application's required field name. It does not establish which fixture file a particular run opened. The code uses a relative path, so the run's working directory is relevant to that question. For the test syntax and the meaning of assertEqual, see the Python 3.13 unittest documentation.

The provider-neutral CI task description says:

checkout: requested revision
restore cache: workspace-fixtures
working directory: checkout root
runtime: Python 3.13.2
command: python "$CHECKOUT/packages/exporter/tests/test_schema.py"

This is a fictional task description, not a GitHub Actions workflow you can deploy. Its cache step is intentionally underspecified: it does not tell us which files were restored.

Write down what was actually recorded

The fictional local record says:

Revision label: EXAMPLE-C17
CHECKOUT: /work/shop
Working directory: /work/shop/packages/exporter
Runtime reported: Python 3.13.2
Command: python "$CHECKOUT/packages/exporter/tests/test_schema.py"
Selected test: SchemaTests.test_export_id_field
Reported outcome: success
Fixture path and content captured during run: not recorded
Working-tree changes captured during run: not recorded

The fictional CI record says:

Checked-out revision label: EXAMPLE-C17
CHECKOUT: /work/shop
Working directory: /work/shop
Runtime reported: Python 3.13.2
Command: python "$CHECKOUT/packages/exporter/tests/test_schema.py"
Selected test: SchemaTests.test_export_id_field
Reported outcome: assertion mismatch, actual orderId, expected order_id
Event: pull request
Fixture path and content captured during run: not recorded
Post-cache working-tree changes captured during run: not recorded

The records support a specific lead: the working directories differ and the two supplied fixtures have different content. They do not prove what happened during either run. A cache or another preparation step could have replaced a fixture after checkout.

Fill the environment-difference table

The same revision is a starting condition. Generated files, restored artifacts, environment variables, and command wrappers can still make the effective inputs different. Avoid describing the runs as equivalent until those differences have been checked.

Choose between at least two causes

Hypothesis A: the working directory selects the wrong fixture. Under the displayed code, a run from the root would look for the root fixture, while a run from the exporter directory would look for the exporter fixture. The field names in the supplied examples fit the reported mismatch.

To test A, capture the resolved fixture path and the field value immediately before reading it in the failing environment, or use an equivalent narrowly scoped diagnostic. If the failing run actually opens the exporter fixture and that fixture still contains order_id, A does not explain the record. Do not “confirm” A merely by repeating the initial table.

Hypothesis B: restored or generated content differs from the tracked fixture. Both runs may check out the same commit while CI's fixture cache supplies different bytes afterward.

To test B, inspect the cache's path scope and compare the actual fixture content before and after that preparation step in a disposable diagnostic run. If the relevant content does not change and matches the tracked fixture, this specific cache-content hypothesis loses support. A cache hit alone is not proof of corruption.

Start with A because the current records show a directory difference and the focused path observation directly distinguishes the two known fixtures. If the observation contradicts A, investigate B next. Broad changes to the runtime, cache, and test in the same attempt would make the result harder to interpret.

Copy this investigation request

Investigate why SchemaTests.test_export_id_field is reported successful
locally and failing in CI for the same checked-out revision.

Evidence supplied:
The tree, two schema fixtures, test excerpt, provider-neutral task
description, and paired records above. These are the current evidence;
the root cause is UNKNOWN.

First establish the actual checked-out commit, test selection, working
directory at the test step, runtime executable, and relevant setup/cache
conditions. Do not compare a local branch tip with an unverified CI
merge revision.

First hypothesis:
Relative fixture lookup selects the root fixture in CI.
Propose a diagnostic that records the resolved fixture path and relevant
field at the failing step. Explain what observation would refute it.

Second hypothesis:
Cache or generation changes fixture content after checkout.
Inspect the paths and effective content if the first hypothesis fails.

Read source and existing logs first. Do not rerun workflows, change
configuration, or edit the test until the proposed diagnostic is selected.
Never dump all environment values or secrets. Keep evidence to the named
path, field, runtime, revision, and relevant setup conditions.

Do not remove the assertion, change order_id to match an unexplained
actual value, disable checks, broaden permissions, or upgrade dependencies
as a trial fix.

Return a comparison table, evidence for and against each hypothesis,
the smallest next check, and a bounded change proposal only if supported.
Separate existing run records, manual predictions, and new execution.

Let the observation determine the repair

If later evidence confirms that the repository intends the exporter fixture and the path is resolved from the wrong directory, a repair could make the intended path explicit or align the configured directory. Choose according to the repository's test conventions and scope. Verify that the check still enforces order_id afterward.

If cache restoration supplies the wrong content instead, investigate the restore scope and artifact identity. Changing the expected assertion would leave that input problem unexplained.

This exercise ends with an investigation request and two testable hypotheses. Its actual root cause remains unknown because the discriminating checks have not been executed. In your own case, add their results to the comparison before asking the agent to implement a fix.

For a CueSpec conversation, read the latest answer before replying with a new log excerpt, so the next request addresses the current hypothesis.