Probability Amplification for Cheaper AI

It’s common sense that if you want a more reliable response from your AI, you should use a more powerful (and more expensive) model. Spend more money: get a better answer. Simple.

But that’s not always true.

Yes, for some kinds of problems, a reasoning task on a low-cost model will fail randomly. Give it the same input many times, and only some answers will be correct. Switch to a more expensive model, and the price will increase, but so will the accuracy.

In those cases, however, you can often get a better result for less money through probability amplification, a technique computer scientists have used for decades before the invention of LLMs. Majority voting orchestration uses this technique to improve responses by running the task multiple times and choosing the most common outcome. Doing this with a powerful model can be more accurate but is always more expensive.

Used properly, however, this technique can save money instead.

This blog post will walk you through how probability amplification works as a cost-saving tool. It will explain a real-world example and suggest the criteria you can use to determine when this method is right for you.

An Old Idea

The underlying concept here isn’t new. Using multiple, mostly reliable sources together to get a more reliable result, and specifically the math to prove it works, was invented way back in 1775 by the Marquis de Condorcet.

Condercet’s focus was on juror voting, but the idea was adopted by computer scientists in the twentieth century. Essentially, they found much quicker algorithms that could solve difficult problems at the cost of not being 100% accurate. Because these faster algorithms were randomly and independently inaccurate, they could repeat the calculations many times and take the most frequent result to improve accuracy. With enough repetition, the chance of errors became astronomically small. The new formulas were so much faster independently that they were still quicker even after many repetitions.

Modern machine learning uses the same idea across many areas, including bagging, random forests, and ensemble voting. AI architecture extends this across multiple agents, especially through majority voting workflows. Still, it’s easy to think of majority voting as a trick reserved for when the best model still isn’t good enough. In practice, it can sometimes help the cheap model do the same job for less.

A Real World Example

To demonstrate, I created a simple test scenario based on a real customer case. In it, the AI must read a PDF and reason over the contents. The reasoning is slightly complex, so accuracy varies with price. Running the same inputs repeatedly with basic, standard, and premium models (GPT-4.1 mini, GPT-4.1, and GPT-5.0 reasoning, respectively) in a Power Automate prompt, I get the following results and costs:

The more expensive models outperformed the basic version but at drastically higher prices.

What happens, however, when we use majority voting? Below, we see the cost and accuracy changes as I use the Basic model 3+ times and choose the most common response. I’ve stuck to odd numbers here for simplicity (the usual assumption when there is only one wrong and one right answer and it cannot be a tie).

As you can see, three generations are already enough to compete with the standard model’s accuracy at less than a third of the price, since the odds of a 10% failure repeating two out of three times were already very low. By the time I’d spend as much on basic model generations as on a single standard model (about 10 generations, 25 credits), it would become hard to tell the result apart from the 160-credit premium version.

Interestingly, this assumes a binary division, which is a worst-case scenario in terms of model outputs: i.e., there is only one wrong answer. If your solution has only one right answer but multiple possible wrong answers, the gains are drastically higher over 4+ generations. In other words, if your correct answer is “blue,” but the model may also sometimes say “red,” “green,” or “yellow,” getting “blue” from the majority sampling becomes much easier because the wrong votes are split.

For additional savings, you can monitor the results as you go and stop early if one answer reaches an early majority, which it usually will.

How to know when it works

Before you start redesigning your existing solutions, however, you must understand when this process works and when it doesn’t.

First, bear in mind that this only works for independent failures. In other words, it applies when the errors you’re resisting aren’t specific to certain inputs. In the example above, the model was tested multiple times on the same input. However, if the model was tested 10 times with different PDFs and the basic model was always wrong about the same PDF, then probability amplification cannot help. In that case, it isn’t really 90% accurate, per se. It’s 100% accurate on nine test cases and 0% accurate on one. Repeating that 0% accuracy case ten times won’t improve your outcomes.

Second, consider your exact costs and accuracy. If the basic model is only 60% accurate and the standard is 99% accurate on a binary decision, for example, you’ll need too many generations to bridge the accuracy gap, and the cost savings will vanish. Unfortunately, there’s no simple rule of thumb for deciding whether your odds are close enough to justify the cost. You’ll need to calculate it, and for that, you have fundamentally three choices.

• Do the math yourself: If you’re comfortable doing so, you can calculate the effect using Condorcet’s jury theorem.
• Use an online calculator: You can also use an online calculator. For example, with this one [link], you can enter the accuracy, number of generations, and cost per generation, and it will estimate your final accuracy and price. However, this won’t account for accuracy gains when the AI can be wrong in more than one way. Those gains are more difficult to calculate because you’ll need to know the odds of each type of error. Nonetheless, even in those situations, this is a good starting point, and for multi-error versions, net accuracy will be higher than the calculator suggests (at least for 4+ generations).
• Ask an AI: Artificial intelligence can make mistakes, as endless online disclaimers remind you, but I’ve found that many of them can help check the math with respectable accuracy. If you’re not comfortable with the other methods, your favorite chat assistant can usually help you figure out how much accuracy you’ll get over a given number of generations, if you can test how accurate single generations are first.

Final thoughts:

Often, optimizing an AI solution doesn’t require a completely new design. A clever application of traditional methods may be more than sufficient. In this blog, we reviewed probability amplification and its application to AI cost savings. Centuries-old math with a decades-long history in computer science can cut costs even in the most technologically advanced solutions of our day.

Nonetheless, probability amplification isn’t a magic fix for every problem. Good prompting and data quality are a must before you can even consider probability amplification. Without those, your errors will likely not be independent, rendering probability amplification irrelevant. Even when you have independent errors, probability amplification won’t be useful in cases where the accuracy of your cheaper models is too low (taking too many generations to close the gap) or too high (at which point, normal generation from the cheaper model is usually sufficient).

Still, the potential cost savings make probability amplification worth keeping in mind, especially if your task requires moving all the way up to a high-priced deep reasoning model. That percentage reduction will add up over high-volume workflows.