The experiments that killed our first pitch
Before launching, we ran two adversarial benchmarks. The first showed that a memory-only LLM is dangerously stale about dependencies. The second — against a search-enabled agent, i.e. what coding agents actually are — refuted our original “smarter dependency answers” positioning. We publish both, including the loss, because the result shaped this product more than any feature did.
Experiment 1 — frontier LLM (no tools) vs live registry data
20 questions targeting post-training-cutoff ecosystem change. The LLM answered from memory; the data column answered only from live registry fetches; independent judges re-verified both before ruling.
Where the model’s recommendation itself was wrong:
- Auth.js → Better Auth (merged Sept 2025) — Recommended a library whose team had moved on
- Vite 8 replaced esbuild with Rolldown — Described a bundler architecture that no longer exists
- Remix 3 is a non-React (Preact) project — Recommended it as a React upgrade path
- AI SDK at v6 GA — Two majors behind on the core API shape
- ESLint 10 removed eslintrc — Guessed the old config was flag-gated — it is not
Every tie was a settled ecosystem (uv, Pydantic v2, Express 5, App Router): freshness is worth the most on the churning edge, near nothing on stable tooling. This experiment alone proves no product — real agents can search. That is what Experiment 2 tests.
Experiment 2 — search-enabled agent vs live registry data (the re-match)
Same design, fresh questions, but the model may now search and fetch freely. Pre-committed rule, written before running: DURABLE if live data shows a correct, decision-relevant delta on ≥50% of realignment + co-occurrence questions; PARTIAL ≥30%; else THIN.
| Category | n | live-data wins | ties | searcher wins | decision-relevant deltas |
|---|---|---|---|---|---|
| realignment | 8 | 2 | 4 | 2 | 3 |
| abandonment | 4 | 1 | 3 | 0 | 1 |
| cooccurrence | 4 | 0 | 4 | 0 | 0 |
| breaking-major | 4 | 0 | 3 | 1 | 1 |
What we learned against ourselves: a searching agent is not stale; co-occurrence — the category we believed search could not assemble — went 4/4 ties; the searcher even won twice outright. The residual edge is cost, not knowledge: the searcher burned ~34 tool calls per question (678 total) to reconstruct what one structured call returns in ~30ms.
Why the product survived: both experiments test pull — someone asks. Neither can test push: nobody issues a query for a change they don’t know happened. Watching is the part a search-enabled agent structurally cannot do for itself. That asymmetry is PatternStack.
Experiment 2 — all 20 questions, judge verdicts
Judges re-verified both answers against registries and the web before ruling, and were instructed to call a tie whenever the searcher found the same facts.
| # | Question | Category | Winner | Delta |
|---|---|---|---|---|
| 1 | I'm starting a new React project with Vite. Do I still need to think about Rollup vs esbuild? | realignment | tie | — |
| 2 | Should I configure webpack or Turbopack for my new Next.js app, and is Turbopack still considered experimental? | realignment | live data | ✓ |
| 3 | I have a Remix v2 app and I'm planning new features. Should I upgrade to the next Remix version? | realignment | tie | — |
| 4 | Does Prisma still download a Rust query engine binary, and do I still need to worry about it? | realignment | tie | — |
| 5 | Type-checking our large TypeScript monorepo with tsc is slow. Is the native Go-based TypeScript ready? | realignment | searcher | — |
| 6 | What's the right lockfile setup for a Python project in 2026 — requirements.txt with pip-tools, or something newer? | realignment | tie | ✓ |
| 7 | I'm setting up type checking for a new Python codebase. Is mypy still the default choice? | realignment | searcher | — |
| 8 | For TLS in a Rust service, should my rustls setup use the ring crypto provider? I see aws-lc-rs mentioned. | realignment | live data | ✓ |
| 9 | I'm choosing an async runtime for a new Rust crate and want something lighter than tokio. async-std? | abandonment | tie | — |
| 10 | We're starting a new React design system and the team likes CSS-in-JS. Is styled-components a safe pick? | abandonment | live data | ✓ |
| 11 | My Python app just broke with 'ModuleNotFoundError: No module named pkg_resources' after an upgrade. | abandonment | tie | — |
| 12 | Our CI uses 'corepack enable' to pin pnpm via the packageManager field. We're upgrading our Node version. | abandonment | tie | — |
| 13 | I'm starting a new Expo app today. Which Expo SDK, React Native, and React versions actually go together? | cooccurrence | tie | — |
| 14 | For a tRPC + react-hook-form app validated with Zod, which Zod major should I install? | cooccurrence | tie | — |
| 15 | Setting up Tailwind CSS with Vite and shadcn/ui for a new project — which packages do I actually need? | cooccurrence | tie | — |
| 16 | I want to try free-threaded (no-GIL) Python for a data workload. Which Python version and which packages work? | cooccurrence | tie | — |
| 17 | What changed in Vitest 4 that I need to handle when upgrading? Specifically, is the workspace config still supported? | breaking-major | tie | ✓ |
| 18 | We're about to start a new Django project. What's in the latest Django major release, which version should we pin? | breaking-major | tie | — |
| 19 | My Rust project's code using rand — thread_rng(), gen_range(), the Rng trait — stopped compiling. | breaking-major | tie | — |
| 20 | Upgrading Storybook to the latest major: do I still install all the @storybook/addon-* packages? | breaking-major | searcher | — |
Honesty notes
- These are self-run evals by the vendor. The design is adversarial against our own claims — Experiment 2’s rule was pre-committed expecting to pass, and we failed it — but replicate before trusting.
- N=20 per experiment; categories are small. Directional, not precise.
- Answerers and judges are LLM agents with tool access; their verification is itself fallible. Per-question judge reasoning ships in the raw JSON.
- Harness: a deterministic multi-agent workflow — generator → 2×20 independent answerers → 20 verifying judges → scripted scoring (61 agents, ~19 minutes). The orchestration script is preserved verbatim and available on request, as is the raw results JSON.
Questions or want to replicate? Docs · run date 2026-06-10.