Fighting AI Entropy: Why Classic QA Matters More Than Ever
As non-deterministic code introduces randomness into the SDLC, consistent, rock-solid testing is our best line of defense.
If you’re a Quality Assurance or technology professional in 2026, you have undoubtedly been impacted by AI-assisted software development.
AI-assisted coding is not necessarily bad—in fact it has lowered the barrier to entry to individuals seeking to be get started in software engineering. Some small projects can be created completely by AI. I’ve not seen large and complex project projects make it that far and there are some reasons why.
The key issue is the non-deterministic nature of AI-produced code. Code and text produced by AI is randomly generated, and the results from Large Language Models (LLMs) vary from one run to another. Sometimes those variances don’t matter, sometimes they do. Consider these two sentences:
The quick red fox jumps over a lazy brown dog.
The quick red fox leaps over the lazy brown dog.
Semantically, those two sentences are the same. A fox propels itself into the air over a dog. However, this sentence is famous because it uses every character in the English alphabet. Changing the word in the sentence from “jumps” to “leaps” means that the second sentence no longer uses the letters ‘j’ or ‘m’. It can no longer be used to test lowercase letters in fonts or to teach typing.
Nuances like the sentence above crop up in ai-generated code all the time and they can be hard to catch if they pass syntax checkers, linters, and compilers. Knowing the importance of that nuance is the job of the QA professional, but if presented with 100s or thousands of them, what are we to do?
Can process help?
AI coding is new, but bugs are not new. Pre-AI, we knew that different people produced software at differing levels of quality, and the industry developed several practices that mitigated these differences. The waterfall software development lifecycle (SDLC) – Requirements, Design, Coding and Testing—served well to convert ideas into deterministic, reliable software, even for mission-critical and life-safety circumstances. The Agile Revolution compressed the waterfall SDLC into smaller, iterative phases, and initially that was a challenge for mission-critical projects, but organizations found that shoring up the requirements and testing phases in these situations was sufficient for creating and shipping solid code.
The SDLC process can’t exactly solve the problem AI-assisted coding introduces because AI-influenced production introduces entropy at all stages of development.
A product manager may create their requirements specifications with AI, programmers and testers may plan and implement their parts of the process with AI. As a consequence, subtle errors are introduced at all stages, and a product may deviate from the original idea in ways not always caught.
The image below shows this from signal processing perspective—the original signal (project intent) is created by AI f(x), but it has some errors e(x), which slightly changes the output. Maybe that matters, maybe it doesn’t.,
One way to stop AI Entropy
I would suggest fighting the problem at its source: The opposite of randomness is consistency. If we cannot rely on AIs to be consistent, we build systems of developing and testing that enforce stable output.
Imagine you have a system under test (SUT) that is being developed very quickly by developers using AI-assistance. If you have requirements, those can be a fundamental rock on which your testing takes a deterministic stance. This means the font package must have valid output for every character. The product must boot without error messages. The report must print on one page. You define a set of test cases for what must happen, no matter what, and this becomes the acceptance criteria. This is, in fact, classic quality assurance.
Scaling Classic QA with tools
The challenge turns out not to be re-inventing QA, but finding and making use of tools to make it possible in a vibe-coded era.
Tool 1: You can use AI to help with test plan creation. LLMs will gladly create test suites, but they often have holes because they don’t understand the product well. This is the subject of several papers at PNSQC this year.
Use your own expertise to define the test plan but take advantage of LLM dictation and review features to save time in writing it up. Anthropic, OpenAI, and Google all have voice-dictation features in their LLM products. If you aren’t working in a crowded environment, you can use dictation to get your ideas to paper faster. Microsoft Word also has a dictation feature if you don’t want to risk having your thoughts go to a chat AI before you can see and correct them.
LLM AIs are generally better at analysis than synthesis. You can take advantage of this by writing a test plan and feeding it and a description of the product under test (or codebase if you have access) and ask the LLM for missed cases and areas of refinement in your test plan.
Tool 2: Use AI to help with tooling and automation. Be careful with this if you’re not already practicing software development, but LLMs can generate software for your test automation suite. An example might be if you have a legacy environment that works with Selenium and WebDriver for web testing: most commercial AIs are trained on those technologies and can update a test set with additional cases. They can also troubleshoot minor issues. You should still “test the tests” that it generates, but the LLM generation saves time.
The key idea is that Software Quality Assurance is essentially the same practice, but we have to take advantage of different tools to ensure quality. Providing feedback on whether something works or not has to be based on evaluative systems or test cases that do not change. Modern LLM-based AI provides the means of creating these test quickly and creating vehicles for running them. Our ability to adapt to these new toolsets will determine how successful we are in the goal of quality assurance.
About the Author
Mitch White is an AI Engineer at Cornell University with over 25 years of experience spanning roles as a Quality Assurance SDET, software developer, manager, and Director of Software Development. He holds an MBA from the University of Denver and a undergraduate degree in Computer Science and Electrical Engineering from MIT. Remotely based in Beaverton, Oregon, Mitch enjoys exploring new technologies and spending time with his family.
 | A guest post by
|