Typed AI Decisions Need an Engineering Contract

AI features become product infrastructure only when their outputs can enter an engineering contract.

A model can explain why a conversion chart changed and still leave the application with an unresolved question: which workflow should run next? Someone has to interpret the prose, choose an action, and establish whether the result helped.

OpenAI’s Decisions API offers a more direct interface for that boundary. As of October 10, 2026, the official guide describes it as a public beta that evaluates text and images and returns predicates, choices, or scores. OpenAI says these decision workloads run about 10× faster than its Responses API. That is a provider claim, rather than a latency result measured in my application. Decisions API guide

The architectural opportunity is the finite answer space. An application can connect a typed answer to a documented policy, preserve the evidence behind that answer, and measure what happens after the selected action.

For the web and analytics products I work on, the useful chain is evidence → decision → action → outcome. Each arrow needs an explicit boundary. A typed response helps define one of them; the product still has to define the rest.

Consider an analytics product that detects an unexpected decline in purchase events. The chart can already show the decline. The next task is to route an investigation toward the most useful first check.

The application assembles a bounded evidence snapshot: event counts for the affected source, delivery lag, recent schema changes, validation failures, and comparison periods. The model classifies the snapshot into one of four categories:

This decision routes an investigation. It does not establish the cause of a revenue decline or authorize a change to customer data. The labels should say what the evidence supports and what action is allowed, without implying more certainty than the workflow can provide.

Keep deterministic rules ahead of the model when they already settle the situation. If the source has no valid events because ingestion is explicitly paused, the application can show that state directly. A model call becomes useful where the evidence needs interpretation under a defined rubric.

The categories also need to cover ambiguity. Two problems can coexist: an instrumentation deployment and an ingestion delay may both affect the period. If the product requires several findings, design several questions or a different output contract. For this example, the choice identifies the first investigation route and leaves mixed evidence eligible for review.

I would begin with this sort of bounded, reversible workflow. Its outcomes can be reviewed, its routing errors can be corrected, and the application can compare it with the existing manual process.

An enum prevents arbitrary category names. The question and its rubric establish what each category means.

Define the evidence window, required fields, missing-data behavior, and the criteria for adjacent categories. A snapshot with delayed delivery should not be silently treated as a completed reporting period. A percentage change without the underlying counts should not carry the same evidential weight as a high-volume comparison.

For OpenAI’s interface, predicates return an estimated probability that a condition is true. Choice and score answers expose a probability distribution and a separate confidence field. Preserve those distinctions when normalizing the response. Interpreting Decisions API answers

The following is an application-owned decision record, rather than the provider’s request or response schema. Its numbers are illustrative:

{
  "decisionId": "decision-1042",
  "tenantId": "tenant-a",
  "sourceId": "store-1",
  "evidenceRef": "evidence-1042-v1",
  "questionVersion": "purchase-triage-v3",
  "model": "gpt-6-luna",
  "status": "answered",
  "choice": "delivery_delay",
  "probabilities": {
    "instrumentation": 0.07,
    "delivery_delay": 0.87,
    "behavior_change": 0.04,
    "insufficient_evidence": 0.02
  },
  "confidence": 0.82,
  "policyVersion": "triage-routing-v2",
  "selectedAction": "prepare_ingestion_diagnostics"
}

The selected choice’s probability and the separate confidence value are different fields. A typed schema does not establish that either one is calibrated for this product’s data. Nor does a correctly formatted choice establish that the category itself is correct.

Validate the response at the application boundary: known question identifiers, supported answer variants, allowed choices, finite numbers, and required fields. Refusal, timeout, unavailable service, and an invalid response are explicit states. They must not become a default category that appears to have been chosen by the model.

Store the question version, application policy version, provider request identifier where available, and the model information returned by the provider. If a model name is an alias, recording it alone may be insufficient to reproduce an older result. Preserve the original output and evaluation context rather than assuming a future call will return an identical answer.

The application decides whether an answer is usable and what it permits. A model’s confidence does not grant access to a tenant, widen the reporting scope, or authorize a mutation.

For this example, the policy automatically prepares a diagnostic view only when the evidence is fresh enough, the answer is valid, and the selected category passes the evaluated thresholds. Everything else goes to review. Source permissions are checked separately before the diagnostics are loaded.

type Choice =
  | "instrumentation"
  | "delivery_delay"
  | "behavior_change"
  | "insufficient_evidence"

type Decision =
  | {
      status: "answered"
      choice: Choice
      confidence: number
      probabilities: Record<Choice, number>
    }
  | { status: "refused" | "timeout" | "invalid" | "unavailable" }

type Policy = {
  maxEvidenceAgeMs: number
  minConfidence: number
  minSelectedProbability: number
}

function selectRoute(
  decision: Decision,
  evidenceAgeMs: number,
  policy: Policy,
): Choice | "review" {
  if (!Number.isFinite(evidenceAgeMs) || evidenceAgeMs < 0) return "review"
  if (evidenceAgeMs > policy.maxEvidenceAgeMs) return "review"
  if (decision.status !== "answered") return "review"
  if (decision.choice === "insufficient_evidence") return "review"
  if (decision.confidence < policy.minConfidence) return "review"
  if (decision.probabilities[decision.choice] < policy.minSelectedProbability) {
    return "review"
  }
  return decision.choice
}

This function assumes that schema and numeric validation have already succeeded, and that the policy configuration has been validated. The thresholds come from evaluation of this workflow; they are not universal confidence cutoffs.

Versioning the policy makes a routing change inspectable. If the model output stays the same while the threshold changes, the application can explain why an action changed. It also makes rollback possible without rewriting historical decisions.

If an action creates a durable task, connect it to the decision ID with an idempotency rule. Repeating a request after a lost response should not create another investigation task. Preserve the original decision for a retry; a deliberate reevaluation receives a new decision ID and a link to the earlier one. Designing reliable AI agents covers the runtime and recovery boundaries around such actions.

Before enabling automatic routing, collect representative evidence snapshots and label the appropriate investigation route. Include missing fields, small volumes, delayed data, contradictory signals, and cases where the correct answer is insufficient evidence.

Separate the data used to choose thresholds from the data used to assess the final policy. Otherwise, the policy can look reliable because it was tuned to the same cases used to report its performance.

Check whether high-confidence decisions are more often correct on held-out cases. Group them into confidence ranges and compare predicted confidence with observed correctness, while keeping the selected-choice probability separate. Inspect the same relationship by source type, evidence quality, and input language when those differences affect the workflow.

A useful evaluation reports both automatic-routing accuracy and coverage. A strict threshold may improve accuracy among automatically routed cases while sending nearly everything to review. That could still be the right trade-off, but the product needs to see both quantities and the cost of mistakes.

For example, routing an ingestion incident toward a behavioral comparison can delay the investigation. Sending an ambiguous case to review consumes time but keeps the uncertainty visible. The relative costs determine the policy; the numeric score alone does not.

Keep classification quality separate from action utility. A correctly labeled ingestion delay does not prove that the diagnostic view helped anyone resolve it. Measure whether the action exposed useful evidence, whether the investigator changed the route, and whether resolution time improved against an appropriate baseline.

Outcome labels can arrive later and remain uncertain themselves. A reviewer’s first choice may change after new evidence arrives. Store the label version and its basis rather than treating every immediate click as ground truth.

Observability should let an engineer reconstruct what the application knew, what it decided, what policy it applied, and what happened afterward.

Record enough input to investigate a decision within the product’s data-retention and access rules. A digest can establish whether two inputs matched; it cannot explain the contents of an input that no longer exists. A scoped snapshot reference is often more useful than copying a complete customer payload into every log line.

Carry the decision ID across traces, tasks, and outcome records. A successful model call can then be distinguished from a rejected policy, a failed action, or an action completed without a useful result. Observability is a product contract describes the same need for signals that support a concrete decision.

Track the distribution of routes and fallback reasons alongside latency and error rates. A sudden rise in insufficient_evidence may come from a missing input field. A rise in confident but overridden classifications may indicate a changed source population or a question whose categories no longer fit the product.

Measure end-to-end latency too. The user waits for evidence gathering, a model call, policy evaluation, and diagnostics to load. A faster decision endpoint improves only one part of that path. Compare p50 and p95 workflow latency under the product’s real payloads and failure conditions before treating the provider’s speed claim as a user-visible result.

Start in shadow mode: record proposed routes while the existing process continues to determine actions. Compare the proposals with reviewed outcomes before expanding automatic routing.

Then enable a narrow, reversible action for the cases where the evidence supports it. Keep fallback explicit and observable. A temporary provider failure might send the case to the established manual queue; it should not erase the incident or produce an invented classification.

Retain a control path during rollout. If the evidence schema, model behavior, or source population changes, the team needs a way to disable automatic routing while continuing the product workflow. Reliable AI Agents Need SLOs connects that recovery behavior to measurable reliability promises.

Test the deterministic boundaries directly: invalid variants go to review, stale evidence cannot trigger automatic routing, permissions are enforced, and retries preserve one durable action. Use labeled evaluations for semantic decision quality and fault injection for timeout and recovery behavior. Each check establishes a different part of the engineering contract.

“The model returned an answer” is not a product metric. A trustworthy AI feature needs a traceable chain from evidence to decision to action to outcome. Typed answers make that chain easier to build; explicit policies and measured results make it useful to the people relying on the product.