the stochastic evidence problem.

Hand my staff the same trace. Do they produce the same evidence?

The regulator's mental model of an audit is simple. Hand my staff the same trace you handed your pipeline, and the staff should produce the same evidence claims. If they do not, your evidence is not evidence, it is a probabilistic guess that happened to land somewhere defensible the day you ran it. The auditor does not need to be hostile to surface this failure mode. They only need a second machine and a Tuesday.

Most engineering effort in this category is spent shipping fast, not reproducible. The Anthropic SDK call returns a structured response, the prompt looked right in staging, the PDF rendered, ship it. A replay test was never wired up. The pipeline is one-shot, the trace logs are present, and any second run is presumed to behave identically because the inputs are the same. They do not behave identically. Two API calls into Claude with the same trace, the same prompt, and the same model id can and do return different bytes. Sometimes the difference is one whitespace token, sometimes it is one citation, sometimes it is one risk-tier classification flipped from limited to high.

The audit-replay test is the simplest possible discipline a regulated AI product can hold itself to. Take an attested package, ingest its trace through the pipeline a second time, and assert that the evidence-bearing artefact is byte-equivalent to the first run on every claim that does not declare itself probabilistic. Everything that follows is in service of one outcome: that the evidence record reduces to the same bytes on a second run. That is a property of the record, and it holds by specification. Whether the model stages replay identically is a separate and currently unmeasured question — the four moves narrow it, they do not close it, and no claim on this page should be read as saying they do.

Non-determinism comes from at least four places.

An LLM API call is not a function call. It is a network request to a fleet of inference servers, each running attention kernels on a non-deterministic GPU, behind a load balancer that picks a replica without telling you which one, with a prompt-cache layer that may or may not be warm. None of that is exposed in the SDK signature. The signature looks pure. The execution is not.

Here is the part that reorders the whole problem. Anthropic's current models do not expose the knobs an engineer reaches for first: temperature, top_p and top_k are rejected outright on Opus 4.7 and later — including claude-opus-5, which is what all four pipeline stages run — and the Messages API has no seed parameter at all, so there is nothing to pin. As of the Anthropic Messages API reference, 6 August 2026. An earlier version of this page described setting a temperature and fixing a seed; neither is possible on the system it was describing, and both have been removed rather than restated. Without engineering intervention two identical API calls can return different outputs, and the intervention cannot happen at the sampling call. It has to happen in the record.

The engineering question is not how do we make the LLM deterministic. The engineering question is given an LLM that is not deterministic, how do we produce evidence that is. The four moves below are the answer.

Canonical bytes are the precondition for everything.

Before any hashing or comparison happens, the JSON record has to land in a canonical form. The JSON Canonicalization Scheme defined in RFC 8785 is the answer the standards community has converged on. The opening of the spec is exact about what canonicalisation means.

JCS produces a unique deterministic byte serialisation of equivalent JSON values. Two documents that mean the same thing serialise to the same bytes; two documents that differ in any meaningful way serialise to different bytes. The properties JCS pins down are the ones that vary across naive encoders: IEEE 754 number form, member-name sorting at every nesting depth, escape-sequence canonicalisation, and whitespace stripping. One property it deliberately does not touch is Unicode normalisation. §3.1 is explicit: "Although the Unicode standard offers the possibility of rearranging certain character sequences, referred to as 'Unicode Normalization' [UCNORM], JCS-compliant string processing does not take this into consideration. That is, all components involved in a scheme depending on JCS MUST preserve Unicode string data 'as is'." So a precomposed é and a decomposed e + combining acute remain distinct under JCS, and a caller that needs them to compare equal has to normalise before canonicalising. That is a real constraint on the producer, not a feature of the scheme — and an earlier version of this page had it exactly backwards.

import rfc8785
import hashlib

# JCS canonicalisation: stable across Python versions, OS locale, dict-insertion
# order. Pins IEEE 754 number form and sorts member names by UTF-16 code unit
# at every nesting depth. RFC 8785 §1 notes that JWS took the other route —
# base64url per RFC 7515 — so that data need not stay JSON. JCS exists for
# the case where it must.
canonical_bytes = rfc8785.dumps(trace_dict)
trace_hash = hashlib.sha256(canonical_bytes).hexdigest()

# trace_hash is now a function of the JSON value, not of the encoding.
# Two engineers on different machines, different Python versions,
# different locales, get the same hash for the same logical trace.

The temptation, when an engineer first touches this problem, is to reach for json.dumps(trace, sort_keys=True). That is closer to canonical than the default, and it is not enough. It sorts by Python code point, where JCS §3.2.3 requires UTF-16 code units — the two orders disagree for any member name above the Basic Multilingual Plane. It does not pin float form, so a number can serialise differently depending on the interpreter's repr (the notation thresholds around 1e21 and 1e-7 are where this bites). Its escaping depends on the ensure_ascii flag rather than on a fixed minimal-escape rule. And none of its behaviour is specified as a stable contract, so it is a property of the interpreter rather than of a standard. Each of those is a divergence a regulator can find by accident, and each breaks the record's checkability downstream. Both encoders leave Unicode normalisation to the caller — that is the one line where they agree.

Everything downstream assumes a canonical byte string. The canonical form is what makes the evidence record independently verifiable without contacting Warrant, regardless of who reconstructs it. The companion pillar at /blog/four-layer-evidence-stack sits on the same assumption: that the canonical form is reproducible. JCS makes that assumption real.

The property is testable, and it is tested against the RFC rather than against our own expectations. The canonicaliser is gated by the worked example in RFC 8785 §3.2.3 — the spec's own input and its own expected output, byte for byte — plus a test for the case a naive implementation gets wrong: object keys whose UTF-16 order and code-point order disagree. That is the whole reason the sorting basis matters in practice, so it is worth showing rather than asserting.

# RFC 8785 conformance - the gate on the canonicaliser

def test_rfc8785_section_3_2_3_sample() -> None:
    """RFC 8785 section 3.2.3 worked example: its input, its expected output."""
    value = {
        "numbers": [333333333.33333329, 1e30, 4.50, 2e-3, 1e-27],
        "string": sample_string,
        "literals": [None, True, False],
    }
    expected = (
        '{"literals":[null,true,false],'
        '"numbers":[333333333.3333333,1e+30,4.5,0.002,1e-27],'
        '"string":' + expected_string_field + "}"
    ).encode()

    assert canonicalize(value) == expected


def test_object_keys_sort_by_utf16_code_unit_not_code_point() -> None:
    """The non-BMP case, which a naive sort_keys=True inverts.

    U+1F600 is UTF-16 D83D DE00, so it sorts BEFORE U+FB33 by code unit
    and AFTER it by code point. Python's own sort gets this backwards.
    """

One canonical form, infinite valid encodings before it, one byte string after. That is the move.

Pin what produced the verdict. Hash it into the record.

The next move is at the model boundary, and it is not the one this section used to describe. Sampling parameters are the part of the contract a caller would reach for if the caller could; on the models this pipeline runs, it cannot. What remains is provenance. Pin every input that can change a verdict, hash the whole set, and put the digest inside the signed bytes. A reader who cannot re-run the model can still tell whether the thing that produced the verdict changed between two packages — which is the question an auditor actually asks.

All four stages run the same model, claude-opus-5, pinned in one place. That single source of truth matters more than it sounds: before it was consolidated, a prose table said one thing, the code ran another, and the eval harness scored a third — an eval grading a model the pipeline did not run. A digest over the configuration is what turns that class of drift from an internal embarrassment into something visible on the artefact.

# The only source of truth for which model each stage runs.
STAGE_MODELS: dict[Stage, str] = {
    "stage1_classify":              "claude-opus-5",
    "stage2_extract_actions":       "claude-opus-5",
    "stage3_assess_authorization":  "claude-opus-5",
    "stage4_map_obligations":       "claude-opus-5",
}

# Prompt file per stage. Their CONTENTS are hashed, not their names — a
# prompt edit changes verdicts and must change the digest.

# pipeline_config_sha256 is a digest over everything that can change a
# verdict, and it sits inside the signed bytes:
canonical = json.dumps(
    {
        "models": models,
        "prompt_digests": prompt_digests,
        "regulations_corpus_sha256": regulations_corpus_sha256,
        "satisfied_confidence_floor": satisfied_confidence_floor,
        "hard_refusal_confidence_floor": hard_refusal_confidence_floor,
        "inference_provider": inference_provider,
        "inference_model": inference_model,
    },
    sort_keys=True,
)

# No temperature and no seed appear anywhere in this file, because
# neither is accepted by the model: temperature/top_p/top_k are rejected
# on Opus 4.7 and later, and the Messages API has no seed parameter.

Note what is bound and what is not. The digest covers the model ids, the exact bytes of each stage's prompt, the digest of the regulation corpus the verdict was mapped against, and the confidence floors that decide whether a finding is downgraded or refused. It does not cover the model's weights, which Warrant does not hold, or the provider's serving stack, which Warrant cannot see. So the digest answers "was this the same pipeline" and never "was this the same computation". Conflating those two is the mistake this section previously made.

One non-negotiable point: the model id is pinned to a specific version. claude-opus-5 is a versioned model, not an alias. A model change has to be a code change to STAGE_MODELS, and because those strings are hashed into the config digest that sits inside the signed bytes, the change is visible in the record rather than silent. Two packages that disagree on the digest were produced by different pipelines, and a reader can see that without being told. The shape of the upgrade is discussed in the companion routing pillar at /blog/multi-model-routing.

A regression set is the deterministic anchor. Ours is not built yet.

The argument in this section is sound and the implementation is not finished, so read it as design rather than as telemetry. An eval set anchors a stochastic pipeline because it converts "the model changed" into a number you can gate a merge on. Without one, a model-version bump is an unbounded change: the prompts are the same, the code is the same, and the behaviour is different in ways nobody has bounded.

What exists today is the structural half. The model id is pinned to a specific version, so an upgrade has to be a code change and cannot arrive by alias drift. Every citation in the corpus is checked to resolve to enacted text on the regulator's own domain, so an amended or moved regulator page shows up as a failure rather than as a quiet wrong citation. Both of those are real and both are narrow.

What does not exist is the measured half. There is no labelled regression set, no human-agreement score, and no accuracy floor at the merge gate. Stage 1 classification accuracy is not yet measured. A measured number with named failure classes — which inputs the classifier gets wrong, and how — is the work in progress, and this page will carry it when it is real rather than before.

The right-hand column is the target list, not a record of catches. The Map stage is the only one carrying any gate today, and note what that gate cannot see: it establishes that the cited text exists at the regulator's URL, not that the pipeline picked the clause the action actually triggers. Those are different properties, and only the first one is currently enforced.

import pytest
from warrant.corpus import resolve_all_citations

def test_every_citation_resolves_to_enacted_text():
    # this gate is live. it is structural, not an accuracy gate:
    # it proves the cited text is at the regulator's URL. it does
    # NOT prove the pipeline chose the right clause.
    unresolved = resolve_all_citations()
    assert not unresolved, (
        f"{len(unresolved)} citations no longer resolve; merge blocked"
    )

@pytest.mark.skip(reason="no labelled regression set exists yet")
def test_stage_accuracy_against_human_labels():
    # the accuracy gate is not implemented. there is no labelled
    # set to score against and therefore no floor to enforce.
    # building it — with named failure classes — is the open work.
    ...

An eval-set anchor is what would make the rest of the system safe to upgrade. The model is non-deterministic in the small; a regression set is how you find out whether the small variance produced a meaningful change in the large. Until that set exists, a model upgrade here is reviewed by reading diffs, which is a weaker control and should be described as one. The honest statement of the current position: the canonicalisation and decoding moves are implemented and checkable, the eval-anchoring move is not.

Refusal is a recorded outcome.

The regulator does not need a deterministic answer. The regulator needs an honest one, and needs to see where the honesty runs out. That means the record has to carry its own uncertainty on its face rather than presenting every finding at the same confidence and leaving the reader to guess which ones were close calls.

Here is what the published schema actually carries, which is narrower than an earlier version of this page claimed. Each Stage 3 authorization row has a confidence number between 0 and 1, a refusal boolean, and an optional refusal_reason, alongside the four judgement fields — whether the action was within purpose, whether preconditions were met, whether it was reversible, and whether human oversight was appropriate. The receipt then lifts the aggregate to the top level: uncertain_count for Stage 4 obligations marked uncertain, incomplete_oversight_count for actions where oversight was judged absent or uncertain, gaps_count, and a refusal_reason set when the pipeline halted fail-closed. There is no uncertainty enum, no top_alternatives field, and no attestation-incomplete package status. The Messages API returns no logprobs or alternative completions, so a top-five-completions field could not be populated even if the schema declared one.

// Stage 3 per-action authorization rows
"authorizations": {
  "type": "array",
  "items": {
    "type": "object",
    "required": ["action_id"],
    "additionalProperties": false,
    "properties": {
      "action_id":                  {"type": "string"},
      "within_purpose":             {"type": ["string", "null"]},
      "preconditions_met":          {"type": ["string", "null"]},
      "human_oversight_appropriate": {"type": ["string", "null"]},
      "reversible":                 {"type": ["string", "null"]},
      "justification":              {"type": "string"},
      "confidence":  {"type": "number", "minimum": 0, "maximum": 1},
      "refusal":     {"type": "boolean"},
      "refusal_reason": {"type": ["string", "null"]}
    }
  }
}

// and on the receipt, the aggregates a reader sees first:
//   uncertain_count            - Stage 4 compliance='uncertain' count
//   incomplete_oversight_count - oversight 'no' or 'uncertain' count
//   refusal_reason             - set when the pipeline halted fail-closed

A refusal is not an error. It is an outcome, and the package records it the way it records any other: the reason is stated, the actions that were assessable keep their findings, and the reader can see which is which. An honest gap is more defensible than a confident invention.

There is exactly one refusal in this system that is deterministic, and it is worth separating from the rest. A trace that exceeds the ingestion cap is refused before the model is called at all — the check is a pre-model gate, so the outcome does not depend on sampling, on the provider's stack, or on anything the model does. Every truncated trace refuses, every time. That is the only determinism claim about refusal this page makes.

Whether a trace that clears the gate is signed or refused is not deterministic, and saying otherwise would be the same failure the rest of this page is trying to avoid. An internal evaluation returned different outcomes across repeated runs against identical code, and a separate case was signed where refusal was the expected result. That is an open finding on the record, not a solved problem, and it is the reason no claim on this page describes the pipeline as deterministic end to end. Tracking a refusal rate over time and triggering a regression run on a spike would be a reasonable control to build; it does not exist today, and an earlier version of this page said it did.

Two of these are checkable today. The rest is design.

An earlier version of this page printed reproducibility and precision percentages here as if they came from a production run. They did not. There is no eval run behind them, so they have been removed rather than restated, and nothing replaces them until there is a run to report. What follows is the honest split between the properties a reader can check and the ones still open.

The first property is the strongest one on this page, and it is strong precisely because it does not depend on us. JCS does what RFC 8785 specifies. Two engineers on different operating systems, in different time zones, with different Python versions, reduce the same trace to the same canonical form. A reader who doubts it can implement the RFC and compare. That is what makes the evidence record independently verifiable without contacting Warrant.

The second is narrower than it sounds and the narrowness matters. A citation check establishes that the passage exists at the regulator's URL. It says nothing about whether the obligation cited is the one the agent's action actually triggered. Conflating those two is how a compliance product ends up confidently wrong, so the distinction is drawn here rather than left implied.

The third and fourth are open. Publishing an accuracy figure before it is measured would defeat the point of the whole post: an artefact a regulator can rely on is one whose claims she can check, and an unmeasured number fails that test no matter which direction it is wrong in.

Where this hooks into RMF MEASURE 2.5 and 2.13.

The NIST AI Risk Management Framework asks, after the MEASURE function is complete, that "objective, repeatable, or scalable test, evaluation, verification, and validation (TEVV) processes including metrics, methods, and methodologies are in place, followed, and documented". Two sub-categories carry the weight for this post. MEASURE 2.5: "The AI system to be deployed is demonstrated to be valid and reliable." And MEASURE 2.13: "Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated and documented." As of AI RMF 1.0, 26 January 2023. The four moves above are a partial TEVV instantiation against those two — partial because the measured half is the half that does not exist yet.

• Reliability (MEASURE 2.5) · the canonical form is stable by specification, and that is a statement about the record rather than a measured system property. There is no trace-hash stability metric behind it, because none is needed for a guarantee that comes from the RFC — and none exists for the part that would need measuring, which is the model stages.
• Validity (MEASURE 2.5) · this is the gap. 2.5 wants the deployed system demonstrated to be valid and reliable, and Warrant cannot demonstrate that yet: classification accuracy is not measured and no accuracy floor is enforced at the merge gate. The citation-resolution check is enforced, but it is a source-integrity control, not an accuracy control.
• Effectiveness of TEVV (MEASURE 2.13) · also a gap, and the more honest one to name. 2.13 asks whether the evaluation process itself is working. An eval that returns different verdicts across runs on identical code is exactly the kind of thing 2.13 exists to surface, and it did.
• Security and resilience (MEASURE 2.7) · this is what 2.7 actually covers, and it is not what this post addresses. Nothing above is a resilience control. An earlier version of this page mapped all four bullets to 2.7 and cited a dual-provider replay as the resilience statement; no dual-provider replay is built — the pipeline resolves one provider per run and nothing compares two — so that claim has been removed rather than restated.

The deeper hook is documented in the companion regulator pillar at /blog/nist-ai-rmf. The relevant point here is that the engineering moves are not auxiliary; they are where a TEVV process would live, expressed in code. Two of the four are built and checkable. The two that a NIST assessor would press hardest on — a demonstrated accuracy figure, and evidence that the evaluation process is itself effective — are the two that are not.

Where this hooks into ISO 42001 V&V.

ISO/IEC 42001 is the AI management-system standard, and one limit has to be stated before anything else in this section: it is a paywalled standard. The full text sits behind an ISO licence, so a reader cannot open this citation and check that we characterised it correctly. That fails the test every other citation on this site is held to, which is why paywalled standards are not part of the corpus Warrant maps against. This section cites Annex A.6, the AI system life cycle group, rather than a sub-control beneath it — a pinpoint into a document the reader cannot open is a claim that cannot be checked, whatever the disclosure says around it. Treat what follows as commentary on a standard you would have to buy, not as a verifiable mapping. On the substance: the canonicalisation and configuration-provenance moves are a partial V&V contribution. They are not a complete V&V package, because the measured half — evidence that the system performs to its stated specification under representative conditions — is exactly the part not yet built.

The relevant property for this post is that ISO 42001 V&V evidence is auditable at the management-system level, not just the system-instance level. An auditor reviewing the AIMS asks show me the V&V process, show me when it ran, show me what it caught, show me what was rolled back, show me how the rollback decision was made. Of those, Warrant can answer the model-version-pin history today — the configuration digest inside the signed bytes is exactly a record of what was in force when a package was issued. The rest, the V&V process an ISO auditor would actually ask to see, is the part not yet built, which is the same gap section 05 names.

Where this hooks into Article 12 logging.

Article 12 of the EU AI Act sets the logging duty in two layers, and the difference between them decides what actually applies here. Article 12(2) binds every high-risk system: logging capabilities must enable the recording of events relevant for identifying situations that may result in a risk within the meaning of Article 79(1) or in a substantial modification, for facilitating the post-market monitoring referred to in Article 72, and for monitoring the operation of systems referred to in Article 26(5). Article 12(3) then adds a minimum list — the period of each use, the reference database checked against, the input data for which the search led to a match, and the identification of the natural persons involved in the verification of the results under Article 14(5) — but only for the systems in point 1(a) of Annex III, which is remote biometric identification. A trace-attestation pipeline sits under 12(2), not 12(3), and three of the four items on that minimum list have no application here. An earlier version of this page presented the 12(3) list as Article 12's general requirements; that was an overbroad reading and it has been corrected.

The logging requirement is the part the four moves bear on. Without canonicalisation, two equivalent traces hash differently and the audit-replay test fails: a reader cannot tie a logged event to the canonical record. Without configuration provenance, the record does not say what produced a verdict, so a change in the pipeline is indistinguishable from a change in the facts. Without eval-anchoring, model-version drift silently changes categorisation outcomes. Without residual-uncertainty disclosure, a categorisation is presented with false confidence and the record misleads.

Two limits on that, both of which matter more than the mapping. Article 12 does not bite yet. Regulation (EU) 2026/1744, in force 27 July 2026, replaced Article 113's third paragraph point (c): Chapter III Section 2 — which is where Article 12 sits — applies from 2 December 2027 for systems classified as high-risk under Article 6(2) and Annex III. Building the logging capability now is a choice about lead time, not a response to a live deadline. And the duty is not Warrant's. Article 12(1) binds the provider of the high-risk system, not a third party recording evidence about it — the corpus ties Article 12 to the obligations of providers under Article 16. The four moves are how a provider's logs survive being turned into an artefact someone else can check; Warrant's side of that is that the record it produces does not drift, not that it discharges anyone's duty. The companion regulator pillar at /blog/eu-ai-act-article-12 walks the full article and the per-clause mapping.

Three things still imperfect today. And no ship date for them.

The four moves narrow the gap. They do not close it. Three sources of residual variance remain, and the honest framing is that each has a candidate mitigation rather than one that ships — an earlier version of this section described all three as shipping or in flight, which was not true of any of them.

What would independent deterministic replay actually take? Three things, and Warrant has the third. First, a model snapshot a third party can address and re-run — the pinned model id names a version, but a reader cannot re-run it without their own account, and no frozen-snapshot interface exists to point at. Whether one ever ships is a vendor roadmap question this page is not in a position to answer, and an earlier version of it asserted a ship window that was never announced. Second, a measured replay-stability figure, which does not exist. Third, a record of everything on Warrant's side that determined the verdict, which is what pipeline_config_sha256 is for. The third is real and the first two are not, so the honest description of the position is that one of three prerequisites is met.

Until that capability exists there is no published replay measurement, and this page will not imply one. /trust states the property the canonical record has by construction; it does not report a replay run, because there has not been one to report. The gap is documented, and it is not narrow.

The model is probabilistic. The artefact is not.

The closure is worth stating cleanly. The four moves do not make an LLM deterministic. They make the evidence deterministic given the LLM's outputs. The model can still be probabilistic. The artefact the regulator reads is reproducible.

Two engineers with the same trace and the same Warrant package_id can independently confirm the same canonical record, the same regulator citations, and the same configuration digest, without contacting Warrant. They cannot verify the same per-token attention activations, and they do not need to. The chain of trust is not I trust the kernel. It is I can reduce the trace to its canonical form, confirm the record is the one that was attested, and read what produced it. Those three steps are independently runnable and none depends on a non-reproducible execution. A fourth step — re-run the eval suite and confirm the system still passes its contract — is what would close the loop, and that is the step this page has spent four sections explaining is not built.

That is the shape of the closure between a stochastic model and a deterministic audit, and it is a partial shape. The model is one thing a regulator cannot run; the evidence record is something she can. Trace canonicalisation under RFC 8785 is the byte-form discipline, and it holds by specification. Configuration provenance is the model-boundary discipline, and it ships. Eval-set anchoring is the system-level discipline, and it is not built. Residual-uncertainty disclosure is the artefact-level discipline, and it is partial — the counts are real, the enum this page once described was not. Two of four, then, with the two that are missing being the two that would let anyone measure the other two. A record that can be reduced to canonical bytes and read three years later is worth having on its own. It is not the same as a record whose accuracy has been demonstrated, and this post would be doing the thing it warns against if it ended by conflating them.

Questions an architect asks first.