ShelfReady
Cleans up an online store's product details so AI shopping assistants can find, compare and buy its products.Makes a Shopify catalog clean and structured enough for AI shopping agents to find, compare and buy from.
In plain English
- Problem
- More people now ask an AI assistant to shop for them. When a product's details are vague or missing, the assistant skips it, and the store never knows why.
- Bet
- Check every product the way an AI shopper would, fix what's missing, and have a person approve each change. Then test real AI shopping before and after.
- Result
- Almost every product is now fully readable by an AI shopper: 99.3%, up from 78%. In 360 test shopping trips, the AI never bought the wrong item.
Customer & problem
A mid-size outdoor-gear brand hears that shoppers are starting to buy through AI assistants. Its catalog has vague titles, missing sizes and materials, and inconsistent variants, so agents skip its products. The brand doesn't know how big the gap is, or which fixes matter.
ShelfReady asks one question of every product: could an AI shopping agent, given only this product's data, confidently match it to a shopper's request and answer their follow-up questions? Then it closes the gap and measures whether agents actually shop better.
Strategy & scope
The MVP is four pieces: a readiness audit that scores each product 0–100, an AI fixer with a human review queue, an MCP server so agents can shop the store, and an eval that runs the same shopping tasks on the catalog before and after fixing.
What I chose not to build, and why:
- No payments in the agent path. The agent's last step returns a Shopify checkout link, and the shopper pays on Shopify. That keeps payment and personal data away from the agent entirely.
- No auto-applied fixes. The fixer proposes and a person approves. Wrong product data causes returns and brand damage, and every approval becomes labeled data for improving the fixer.
- One store, Shopify only, no app-install flow. Multiple stores, other platforms and OAuth install are out of scope for v1. They add surface area without changing whether the core idea works.
- $0 a month until it proves its value. That meant free tiers everywhere and Gemini as the default model, with a model comparison planned once there's an eval to compare on.
The demo store is seeded rather than scraped. A deterministic generator built 150 products and injected 8 kinds of defects into 50 of them, so every audit and eval is scored against an exact answer key.
What I built
Fix the catalog
Shopify store
150 products, 50 seeded with defects
Sync
Mirror into Postgres with a content hash
Audit
7 rubric checks + one grounded model call per product
Fixer
Rule fixes + model fixes that must quote the product's data
Human review
Approve, edit or reject each fix
Write-backlive Shopify
Approved fixes to Shopify, then re-sync and re-audit
Let agents shop
AI agent
e.g. Claude, via a custom connector
MCP server
4 tools on Vercel, rate-limited, every call logged
Search + details
Structured filters on the Postgres copy
Stock + cartlive Shopify
Live from Shopify's Storefront API
Checkout link
The shopper pays on Shopify; the agent never sees payment
- P0 Setup: Shopify auth with a cached, early-refreshed token; Postgres on Neon; a live
/statuspage that checks every connection. - P1 Catalog: the generator plus a ground-truth file, an idempotent and resumable seed, and a sync that mirrors the store into Postgres.
- P2 Audit: a rubric of seven checks, category detection from the product's own data (150/150 correct), one model call per product that must quote its evidence, and an
/auditpage. - P3 Fixer: rule-based and grounded model fixes, a
/reviewqueue (anyone can view, only an admin can change), write-back to Shopify, and a before/after view. - P4 MCP:
search_products,get_product,check_availabilityandcreate_carton a public, rate-limited endpoint, tested end to end with Claude. - P5 Eval + demo: 60 shopping tasks with verified answers, run 3 times per catalog by an agent using the same tools as the live server, with an
/evalpage showing every transcript. Plus a public/playgroundwhere anyone can chat with the shopping agent: rate-limited, capped at 200 messages a day, and falling back to recorded replays of real sessions instead of an error.
Key decisions & trade-offs
- Ground truth before measurement. A deterministic generator rather than model-written products. Trade-off: the copy reads more templated.
- Weak checks cap the band. The first audit ranked products correctly but called 89% of the store agent-ready, because strong checks hid fatal gaps. Any check below 0.5 now caps a product at Partial, and two or more make it Not ready. Trade-off: bands no longer follow from the score alone, so the UI shows which check capped each product. Raising the cutoffs gave a similar split, but with thresholds picked after seeing the data.
- Grounded fixes or no fixes. Model fixes need a verbatim quote from the product's own data. Titles may only use words already there, and descriptions only numbers already there. Anything else goes to the merchant. Trade-off: some reasonable rewrites get dropped.
- Duplicate variants go to the merchant. Deleting a variant moves inventory and can break orders, which isn't a call the fixer should make.
- Search on the synced copy, stock and carts live. Shopify's storefront search can't filter on the structured attributes the fixer improved. Trade-off: search is only as fresh as the last sync.
- A public MCP endpoint, rate-limited, with no login. Anyone can connect an agent to the demo. The tools are read-only except cart creation, which takes no payment or customer data.
- Fix the agent's tools, not just the data. The first fair eval run failed mostly on tool behavior, on both catalogs: search dropped numbers, so "6 person tent" returned the cheapest tents, and invented attribute names silently matched nothing. I fixed the tools mid-eval. Trade-off: every earlier run was discarded, so before and after always use the same tool version.
- Repeat every eval 3 times. At 40 tasks, one task is 2.5 points, and the same model varies run to run. Trade-off: 3× the cost, still only about $1 for a clean run.
- A live demo with a budget. The playground spends real money, so it has a hard daily ceiling (about $0.50 at worst) and falls back to recorded real sessions, so visitors never see an error.
Results
The headline: an AI agent bought 0 wrong products in 360 shopping sessions, while the catalog's readiness score went from 94.1 to 98.3. The eval ran 60 shopping tasks, 3 times each, on the original messy catalog and on the fixed one, using the same model (Gemini 3.5 Flash-Lite) and the same tools.
- Store readiness score
- from 94.1to 98.3
- Products agent-ready
- from 78%to 99.3%
- Products not ready
- from 11to 0
- Agent shopping success
- from 98.9%to 99.4%
- Wrong product bought
- from 0%to 0%
- Tokens per shopping task
- from 9,773to 9,370
The finding: agent-readiness is data and tools. Before I fixed the agent-facing search tool, the first fair run scored 95% on the messy catalog and 92.5% on the fixed one. Fixing it moved both catalogs to 97.5–100%. With good tools, a capable agent absorbed most of the catalog mess. The few remaining failures all trace to missing or damaged facts. One of them is a product whose weight is still a merchant gap, and the agent declined rather than guess.
What this doesn't show yet. The success rate barely moved (98.9% → 99.4%) because many tasks have 2–4 correct products, so an agent can route around one damaged listing. Single-answer tasks would separate the two catalogs more sharply. The clean products also score exactly 100 on the audit because the generator makes them perfect, which a real catalog won't.
In a live test through the MCP connector, Claude went from "waterproof hiking boots, men's 10, under $150" to a working checkout link in 7 tool calls. There was no boot under budget, so it offered the closest match: a waterproof hiking shoe at $115.95.
What broke
- A laptop freeze mid-seed. 54 of 150 products had landed. Because resume state lives on each Shopify product, the re-run skipped those and created the other 96 with 0 failures.
- A score that hid the defects. The first audit said 89% agent-ready and 0% not ready. It was fixed with the band gate described above.
- Free-tier quotas. An unpaced audit hit the per-minute limit after about 100 calls and saved a partial run. Calls are now spaced out, and partial runs are never saved.
- An "all-or-nothing" apply that wasn't. The first test landed 6 of 8 changes and marked all 8 failed. Shopify has no transaction across mutations, so each change now records its own result.
- A fixer running on a stale audit. It drafted 11 rewrites of descriptions that were already fixed. They were deleted before review, and the fixer now refuses stale products.
- An invisible store. Products were Active but not published to a sales channel, so the Storefront API saw nothing. Active isn't the same as published.
- "Lynx 30f." The fixer had treated the product's URL handle as evidence, so a bad spec looked grounded. Specs must now match the product's own text in number and unit. A re-check of all 50 live fixes found 3 wrong, and all 3 were corrected.
- An eval harness bug. Stand-in product ids for the messy catalog were longer than the tools allow, so on long-named products the agent found the item but couldn't add it to the cart. That unfairly lowered "before". I found it from a crash, fixed the ids, and discarded the flawed run.
- A model outage. Gemini's free tier returned "high demand" errors and 115-second calls for hours. Moving to the paid tier cost about $1.54 for the whole eval, including discarded runs.
Roadmap
- P6 Ship: record a Loom walkthrough and write the launch post.
- Second model: Groq's gpt-oss-120b is running the same eval on its free tier, and its results go on
/evalwhen it finishes. - Sharper tasks: add single-answer tasks so the eval can separate a messy catalog from a fixed one more clearly.
- Search sort option: questions like "warmest" or "lightest" need sorting, not just relevance. One recorded demo answer got this wrong and was replaced before it shipped.
- Known rough edges: drafted descriptions can only answer what the data covers, and some attribute gaps still need the merchant.
Build log
- Episode 00
Sep 24, 2026Before any code: six accounts, a $0 stack, and three gotchas - Episode 01
Sep 25, 2026A green status page before a single feature - Episode 02
Sep 27, 2026Seeding 150 products, a third of them broken on purpose - Episode 03
Sep 27, 2026My audit said 89% of the store was ready. That was the bug. - Episode 04
Sep 28, 2026101 fixes, one human reviewer, and a store score from 94.1 to 98.3 - Episode 05
Sep 28, 2026Claude shopped my store and came back with a checkout link - Episode 06
Sep 29, 2026360 agent shopping sessions, 0 wrong products, and a finding I didn't expect - Episode 07
Sep 30, 2026A public AI shopping demo that can't run up a bill or show an error