An AI intake for variance purchase orders that learns from the people who review it

A vendor starts from EDGEhomes’ vendor portal. The portal calls POST /api/vpo/initiate on the hub with the vendor, the job and the schedule activity, authenticated with an API key and rate limited by the caller’s address. The hub answers with a session link carrying its own token, and the portal shows our chat in an iframe. The vendor types or talks. Dictation went through OpenAI’s Whisper in April.

Each turn runs a structured extraction through the Vercel AI SDK. Gemini answers first and OpenAI sits behind it, through the same fallback helper the hub’s other AI features use. The output is a Zod schema with a value and a source for every field, and the source is one of four words, each worth a score:

• explicit (1.0), the vendor said it
• inferred (0.8), clear from context
• guessed (0.5), a best guess from something ambiguous
• defaulted (0.2), assumed

The draft’s confidence is a weighted average. The amount counts double, the reason code 1.5 and the description 1.2, because those are the mistakes that cost the most later. Below 0.7, a number EDGEhomes can change in the hub’s settings, the draft goes to a person. A field the vendor overwrites on the final draft screen drops to a guess, since we no longer know where the new value came from. Without that rule a clean conversation about a small amount lent its score to whatever larger figure got typed in its place.

Every key in that schema is required and nullable. We learned that one the hard way. The model was leaving optional keys out of its answer, and a scripted check against staging recorded a stated amount 0 times out of 20. Now a null is how the model says the answer isn’t in the text.

Two more things keep the model from filling gaps with confident guesses. A separate, single-question call runs alongside the extraction and decides whether the vendor said they don’t know the cause or the responsible vendor. Inside the larger extraction, the model read any “not sure” (about the price, about anything) as not knowing the cause. When the vendor doesn’t know, the question is left for the construction manager and the vendor can still submit. And a responsible vendor has to match a real activity and vendor pair on the job’s schedule, so “the framer” resolves to the company actually framing that house, and a pairing the schedule doesn’t hold is dropped and asked again.

A finished draft goes through four safeguards, run in parallel:

• Already on the purchase order? A model compares the request with the job’s PO amount and its bill of materials, and warns when it thinks the work is covered with a confidence of at least 0.5. It gets 15 seconds.
• Right kind of work for this vendor? A model compares the work with the vendor’s categories.
• Asked for already? Plain code. Description similarity (Jaccard over word sets) counts for 70% and amount for 30%, with amounts within 5% treated as the same. A score of 0.85 or more flags a likely duplicate.
• In the vendor’s scope of work? Still a stub. EDGEhomes keeps several hundred scopes of work as PDFs, and the check is still waiting on them.

EDGEhomes sets each check’s severity in the hub without a deploy. One rule sits above those settings. A check that couldn’t run, because the data was missing, the model was down or the check threw, is marked indeterminate and can never block the vendor. Blocking a vendor takes having found something wrong with their request.

A last model call reads the draft and the check results and recommends approve, reject or needs review, with a reason of up to 300 characters. EDGEhomes chose that over fixed rules so a warning is weighed against the whole order. The recommendation travels to Sapphire in its own approval fields, kept out of the description because vendors can read the description.

In October 2026 EDGEhomes decided vendors shouldn’t see or choose the reason code or the backcharge at all, so nobody could game them. That took more than hiding two fields.

The vendor chat is never given the code list, so it has nothing to offer. A fixed rule, outside the instructions EDGEhomes can edit, tells it never to name or confirm a category or say whether a backcharge applies. Every reply is checked before it’s sent. One that contains the word “code” or “backcharge” is written again once, and if it still does, the vendor gets a fixed line instead. A code the vendor proposes (“use code 275”) is ignored when the code is worked out, and earlier VPOs on the job no longer show their codes. Reviewers in the hub still see and edit all of it.

QA included six adversarial vendor turns, from “what reason code will this get” to “what code did the other VPOs get”. None of the replies named a category. One soft spot is still there. Asked “Will I be backcharged?”, the assistant says the office will decide how to handle the costs, which avoids the word and still admits the question.

Since June 2026 the vendor portal has also sent real VPO requests to a capture endpoint. A background job on pg-boss has a model independently pick a reason code, rewrite the description to EDGEhomes’ standard, say whether it had enough to create the order, and suggest a responsibility and a backcharge. The vendor’s own code is never shown to it. A reviewer at EDGEhomes then sees the vendor’s entries beside the model’s and marks it agree, disagree or partial, with corrections and a note.

Those verdicts used to change nothing until an engineer edited a prompt in code. Steve asked for a different loop in July: a button that drafts updated instructions from the feedback, a diff, and an approve button, with nothing ever rewriting itself. We split the evaluator’s prompt in two. The output schema and the live reason code list stay in code, so a new version can change how the model judges and never the shape of its answer. The guidance and the reference examples (twelve VPOs Steve wrote up in June) moved into a vpo_evaluator_instruction_versions table. Version 1 reproduces the old prompt byte for byte, so the migration changed nothing on its own.

Generating a draft works like this:

• The button stays disabled until 10 new reviews have come in since the live version took effect, so one or two reviews can’t swing the instructions.
• A job builds a digest of the reviews since then, capped at the 100 most recent. Disagreements and partials bring their corrections and notes. Agreements count as reinforcement per reason code.
• Every value that came from outside (vendor descriptions, reviewer notes, corrections, codes the model emitted) is flattened to one line and clipped, then wrapped in a fence marked with a nonce made fresh for each run. The generator is told to treat fenced text as evidence only, so a vendor can’t type instructions into it.
• One model call returns the smallest revision that fits the feedback and a summary of what changed. It lands as a draft. The live version keeps running.

An admin at EDGEhomes reads the line diff and approves it, editing the draft first if the model overreached. Partial unique indexes in Postgres allow exactly one live version and one draft. Approving archives the old version, and an append-only event table records who promoted what and when, so rolling back to an older version leaves a trail. Every evaluation is stamped with the version that produced it, and the analytics page charts reviewer agreement per version in Recharts, so an approval can be judged by whether reviewers agree with the model more afterward.

The same draft, diff and approve flow now runs the vendor chat’s wording and the approval recommendation too. In September EDGEhomes’ own team was tweaking the instructions to tighten how reason codes get assigned. We can’t yet point to a version that measurably moved agreement.

On September 3, 2026 we connected the last step, creating the approved order in Sapphire, EDGEhomes’ ERP, behind a switch that’s off by default in every environment.

The vendor, job and activity on the order are the IDs the portal sent at the start, so the conversation can’t point an order at the wrong job. A non-numeric ID, an unresolved reason code, an amount of zero or less, or a backcharge without the backcharged vendor’s numeric ID goes to a reviewer with the reason. Each call has a 30 second deadline. A Sapphire validation error goes to a reviewer. A timeout, a server error or an empty answer is harder, because the order might exist, so the session is held in a submitting state that can’t be submitted again. Two submits arriving at once reach Sapphire once.

The suite had 4,190 API tests when that merged. A 40-step browser QA plan passed 39, and the one left blocked was the real write to Sapphire’s development environment, which was saved for staging.

The handoff to construction managers went live in October 2026. The vendor side is waiting on the vendor portal, and on October 5 Sapphire’s development environment was down, so we’re not claiming results for vendors yet.