TMediaDevelopment TMediaDevelopment LLC

A Conflux case study

Finding the bugs your AI feature already shipped

I built a test suite around the decisions my own AI product makes, ran it against production, and fixed what it found. The same method is now a consulting engagement, in two packages you can judge for yourself.

See the packages Start an intake

Conflux is a multi-model AI chat app that I build and run. Before any model answers, the app has to decide what the user wants: a plain reply, a generated file, an image edit, a video, or a multi-model research job. When that decision is wrong, the user gets the wrong thing. They get a PDF they said they didn't want, an edit that never happens, an attachment that disappears, or a throwaway question that triggers an expensive job.

I built a test suite around those decisions and ran it against my own product first. It found bugs that were already live, and each one below ended in a shipped fix.

What it found

“Don’t make a PDF” made a PDF.

The message “Don’t make a PDF, just tell me the answer” still generated a file. The rule looked for a file type plus a verb like “make” and had no idea what “don’t” meant. The fix suppresses file generation when the verb is negated. “Make me a PDF but don’t include the appendix” still produces the PDF, and a test now holds both cases in place.

“Can you” blocked an edit that “Could you” allowed.

With a photo attached, “Can you remove the watermark from the bottom corner” was treated as a question about the image, so nothing was edited. The same request starting with “Could you” worked. One word in one pattern caused it. Removing that word fixed the edit path and the same flaw in the image-blending path.

A cost gate fired on small talk and missed real research.

One check runs on every message in the default mode. For paying users it escalates the message to a multi-model job. For free users it shows an upgrade prompt. Four casual test messages all triggered it, including a wardrobe question and a pizza-topping comparison. The same check wrongly excluded three serious research questions because of everyday words: “business class”, “margin of error” and “bed bug”. One tuning pass fixed the three exclusions and one of the four false triggers. The other three false triggers are still open, and the suite reports them as failures on every run until they are fixed.

Video: a baseline that flagged three bugs, fixed the next day.

On November 14, 2026, a 111-case suite locked down how the app handles video requests, including which model each one would call. All 111 cases passed. That is a baseline, not a quality score: the cases record what the app does today, including behavior that is wrong. Writing them flagged three bugs:

  • A three-image blend request fell through to plain chat.
  • An attached video was silently dropped in two modes.
  • An attached clip was ignored when the user asked to make a video from it.

All three were fixed within a day. Each follow-up fix was probed with 100 to 193 test prompts, which caught three regressions before they shipped and turned up three more edge cases that were fixed the same day.

How it works

  • Rule checks first, an AI judge only where judgment is needed. Wrong route, wrong model or a missing field is caught by code, with no judge involved. A stronger model grades only what needs a reader, such as whether an answer is grounded in its sources.
  • Outages never count as quality failures. A timeout or a provider error is logged as a test-run problem and kept out of the score, so the trend line reflects the product.
  • Known problems are recorded, not hidden. Each one is labeled as an accepted quirk, a documented gap or a bug awaiting a fix. When the behavior changes, the test flips and says so.
  • A failing run cannot look clean. The headline number is computed from each case’s real pass or fail.

The Conflux suite now covers 17 kinds of checks. They range from free rule checks that run in seconds to end-to-end tests of live research and multi-model answers.

What this looks like for your product

You give me a way to call your AI feature, a sample of real user messages, and the complaints you already know about. You get back:

  • a test suite built around the decisions and answers your users actually receive
  • a scorecard showing what fails, with the exact inputs that fail
  • before-and-after scores for every fix
  • the suite itself, ready to rerun after each prompt or model change

I don’t need your source code, and your data stays under NDA.

Packages

Two ways to start. You can tell from the descriptions whether you fit.

Baseline audit

$1,500–$2,500

I probe your live AI feature with a test-prompt suite, then report the bugs it already shipped, with ranked findings and fix suggestions.

Who it fits. You have a live AI feature and want a clear picture of the bugs it already shipped, without standing up a full eval practice yet.

Eval suite + regression tests

$3,500–$6,000

Everything in the audit, plus a reusable eval suite and regression checks you rerun before each release.

Who it fits. You want the audit, and also a suite you can rerun before each release so the same class of bugs does not come back.

Intake

Five questions. I use the answers to tell you whether a package is a fit, and which one.

1. Name and email
2. Product / URL and what the AI feature does
3. Which models or providers it uses
4. What’s going wrong, or what worries you
5. Which package interests you, and your timeline

Prefer email? Write to troyboymedia91@gmail.com