Shopify Order Checker
An agent that checks every new Shopify order's address, phone and risk before it ships, holds the ones that would come back, and drafts the message that saves them.
- Open source · sample data for Noor & Loom, a Lahore clothing brand
- Design & Engineering
- Shopify Order Agents · 2026
Overview
Stores that take cash on delivery lose money on parcels that were never going to arrive. The Order Checker runs on every new Shopify order: a model splits the address into the parts a courier needs, code checks the phone, the address, duplicates and fraud signals, and scores the order. Clear orders ship as normal. The rest are held in Shopify with a plain reason and a message asking the customer exactly what's missing. A person sends it, then releases the hold or cancels. The agent never cancels anything itself.
The Problem
When a cash-on-delivery parcel is refused or can't be delivered, the store pays the courier both ways and the stock is stuck in transit for days. Most of these orders show warning signs the moment they're placed, but staff check them by hand, inconsistently, and often after the parcel has already shipped.
- ▸Addresses like "near Jamia Masjid, Samanabad" have no house number or street for the courier to find.
- ▸Wrong numbers and landlines mean the courier can't call ahead.
- ▸The same customer orders twice, test orders get through, and customers who refused deliveries before order again.
The Solution
The model reads; code decides. The model only splits addresses into parts and writes the customer message. Every point, verdict and hold comes from plain, tested code, so the same order always gets the same answer and staff can see exactly why it was held.
- ✓The model splits the address into house or flat, street, block, area, landmark and city, in English, Roman Urdu or UK and UAE formats.
- ✓Code checks the phone for the country, postal code against city, duplicates within 48 hours, refused-delivery customers, junk names, throwaway emails and large first orders paid on delivery.
- ✓Points add up to a verdict: clear, confirm with the customer (25+) or high risk (60+). Risk counts half on prepaid orders.
- ✓Problem orders are tagged, held from fulfillment, noted, and get the full result in a metafield. Shopify is the only database.
Architecture
A small agent on top of a Shopify integration and a merchant dashboard, with models behind my open-source agent-handrail package.
Shopify layer
One store interface with two implementations: an Admin GraphQL client (orders, tags, notes, fulfillment holds, metafields, cancellation, webhook registration) and an in-memory demo store. Dev Dashboard apps sign in with the client credentials grant; tokens are cached and refreshed.
Order Checker
Phone rules, address parsing, checks and scoring, message drafting and Shopify actions as separate tested modules. A batch runner checks open orders two at a time and skips anything already checked.
Models
agent-handrail: a fallback chain from Claude Opus 5.5 to Gemini Flash-Lite, zod-validated structured output with automatic repair, retries with backoff, and typed events for the trace.
Webhook & dashboard
Next.js 16. The orders/create webhook verifies Shopify's signature on the raw body, replies in milliseconds and checks the order after responding. The dashboard streams each run over Server-Sent Events.
Key Features
Reasons you can read
Every verdict lists the checks that fired and their points, so staff see exactly why an order was held.
Messages in the customer's language
A WhatsApp or email in English or Roman Urdu that asks only for what's missing, checked in code for the order number, links and placeholders.
Real holds in Shopify
Problem orders get a fulfillment hold, so they can't be packed by mistake. One click releases it when the customer confirms.
Dry run first
On a real store it starts in dry run, showing every tag and hold it would apply without changing anything.
Technology Stack
Design & Engineering
Researched the problem, designed the checks, and built the agent, the Shopify integration, the dashboard and the demo store.
- ▪Wrote the country-aware phone and address rules and the scoring as tested code, keeping the model to reading addresses and writing messages.
- ▪Built the Shopify Admin GraphQL client: tags, notes, fulfillment holds, metafields, cancellation and webhook registration.
- ▪Made the webhook production-safe: signature check on the raw body, instant reply, de-duplicated retries and work after the response.
- ▪Designed the merchant dashboard and recorded the demo against fourteen realistic orders.
Product UI
Technical Challenges
Keeping the model out of the verdict
Challenge: A model that decides which orders to hold gives different answers on different runs and can't explain itself to staff.
Solution: The model only extracts address parts and writes the message. Points, verdicts, holds and cancellations are plain code covered by unit tests.
Addresses in two languages and many formats
Challenge: Pakistani addresses mix English and Roman Urdu and often lead with a landmark; UK addresses lead with the house number.
Solution: A prompt that knows both conventions, plus a code check that treats a leading number as the house number, so a correct UK address is never flagged.
A webhook Shopify can rely on
Challenge: Shopify expects a reply within five seconds and retries deliveries, but a check with model calls can take much longer.
Solution: Verify the signature on the raw body, reply at once, run the check after the response, and ignore repeated deliveries and orders already checked.
Model outages during a batch
Challenge: Providers rate-limit and go down, which would leave orders unchecked or shipped without a look.
Solution: Retries with backoff, then the next model in the chain. If every model is down, the code checks still run and the order is still held; only the message is left for a person.
Outcomes & Highlights
On the sample store the agent sorts fourteen new orders exactly as designed before any parcel leaves: eight ship as normal, four wait for a quick confirmation, and two are held as high risk.
- ✓Catches landmark-only addresses, missing house numbers, invalid phones, duplicate orders, refused-delivery customers and fake names.
- ✓32 tests cover the rules, scoring, message checks, Shopify client, webhook signatures and the batch runner.
- ✓Connects to a real store through a Shopify Dev Dashboard app and starts in dry run.
- ✓Rides out provider rate limits by falling back to the next model, visible in the run trace.
- 14/14
- Orders sorted as designed
- $539
- Held before shipping
- 0
- Automatic cancellations
8 clear, 4 confirm, 2 high risk
Six orders on the sample batch
A person always decides
