How do you build an A/B test significance calculator?

We built the free A/B Test Significance Calculator in a single Claude chat, then uploaded it to Toolcrow and had it live a few minutes later. No servers, no build tools, no deploy pipeline. This post is the whole story, including the exact prompt, so you can build your own the same way.

We build our free tools in the open. When we ship one, we also show you how it was made. This is another post in that series, after how we built the UTM Link Builder.

What we set out to build

The job was small and specific: tell someone whether their A/B test result is real. You enter the visitors and conversions for each variant, and the tool works out each conversion rate, the uplift, and whether the difference is statistically significant. Instead of a bare p-value, it gives a plain-English verdict: this variant wins, or not significant yet.

The harder requirement was what it should not do. No ads, no account wall, no cookie banner, and nothing sent to a server. Someone testing an unreleased page should be able to paste in their real numbers and trust that they never leave the browser. That constraint shaped the whole build.

We built it in one Claude chat

There was no special setup here. We opened a normal conversation with Claude and described the tool in plain English, the same way you would brief a developer. The one part that needed care was the statistics: we asked for a specific, standard method rather than letting the model improvise, and we avoided pulling in any external library so the whole thing stays a single file.

The clearer the brief, the less back-and-forth afterwards. Here is the prompt we used. We cleaned up the wording slightly for readability, but the substance is exactly what we asked for. Copy it and adapt it for your own tool:

Build a single-file A/B test significance calculator as one self-contained HTML
file with inline CSS and JavaScript. No external scripts, no fonts, no network
calls, no build step.

Inputs: for Variant A (control) and Variant B (test), a visitor count and a
conversion count. A confidence-level dropdown with 90%, 95% (default) and 99%.

Behaviour:
- Calculate live as the user types.
- Show each variant's conversion rate, the relative uplift of B over A, the
  p-value, and a plain-English verdict: which variant wins and whether the
  difference is statistically significant at the chosen confidence level.
- Use a two-proportion z-test (pooled variance, two-tailed) and get the p-value
  from a normal-distribution approximation. Do not pull in a stats library.
- Validate gently: if conversions are higher than visitors, show a note instead
  of a broken result.
- A Copy button that copies a plain-text summary, and a Clear button.

Constraints: everything runs client-side, nothing is stored or sent anywhere.
Clean, calm design, mobile friendly, one column on small screens.

The decisions behind the prompt

A prompt is really a list of decisions. A few of ours made the difference between a toy and a tool a marketer would actually trust:

• Name the method. We asked for a two-proportion z-test with pooled variance, the standard test for comparing two conversion rates. Naming it stops the model from inventing something plausible-looking but wrong, and it means the numbers match the professional tools people already use.
• Two-tailed, on purpose. We did not assume the new variant is better. A two-tailed test also catches the case where your change quietly made things worse, which is exactly when you most want to know.
• A verdict, not just a number. A raw p-value sends most people back to a search engine. The tool translates it: which variant wins, and whether it is significant at the level you picked.
• Guide, do not block. If conversions come out higher than visitors, the tool shows a calm note instead of a broken result. Nagging popups are the fastest way to ruin a simple tool.
• No server, on purpose. One file, no network calls. There is nothing to log because there is nowhere for the data to go, so you can trust it with results from private experiments.

Putting it live on Toolcrow

Once the file worked in the chat, going live was almost boring, which is the point. We copied the HTML, pasted it into a new Toolcrow drop, and Toolcrow detected the code type and published it to its own live URL straight away. We set it to open so anyone can use it, added the explainer and FAQ you see on the tool page, and that was it.

No server to rent, no deploy step, and no expiry date on the link. It stays up until we turn it off.

Try it, then build your own

Start with the A/B Test Significance Calculator. If it saves you a trip to a stats textbook, the bigger takeaway is that the tool itself is no longer the hard part. A clear prompt in a Claude chat gets you the code, and Toolcrow gets you the live link.

If you have already built something in a Claude chat, here is how to share a Claude artifact as a live link of your own.