The Horseshoe Paradigm
A New AI Risk Every Quality Engineer Needs to Test For
Our work as Quality Engineers has never been at a more polarizing time. While some despair that the very craft of QA is on the verge of disappearing, others like me see the adoption of AI as the next frontier of risk to be learned and understood.
Let’s talk about a new risk area I recently identified and want to share with you today. At a recent conference, I shared with the QA Engineers that AI is not so much a thinking technology as it is a matching technology. The very measurement of Tokens refers to the smallest possible match. At its core, AI frameworks have captured vast amounts of existing information primarily from the internet. When AI works with word structures such as “Once upon a …,” its algorithms refer to the volumes of content it has consumed and derive the next logical word as statistically likely to be “time”. An interesting reality of how AI pays attention to data and instructions is that it places more emphasis on the beginning and ending of content and less on the middle. I have coined this reality the Horseshoe Paradigm, and within it lies a new type of risk we need to be aware of and test for.
If the application we are testing has AI integrated, results derived from datasets are more likely to be incorrect in the middle regions. Incorporate techniques such as equivalence partitioning, statistical sampling, or anomaly detection to identify these inaccuracies, ensuring your testing strategy accounts for the Horseshoe Paradigm.
Use the following three assessments to determine if you should enhance your testing approach to target defects within the Horseshoe specifically.
First, we need to understand the design of the application under test. Is AI being used within the application to derive results? Was AI utilized in the development of that code?
Next, we need to understand that the middle of datasets and calculated results are much more likely to be inaccurate if AI is being used to derive those results. Pay particular attention to decimal numbers.
Lastly, we should understand the concepts of an “Oracle,” which is a human subject matter expert who provides certified data used in your testing validations. It’s critical not to rely on AI to generate this data due to the Horseshoe Paradigm and the higher risk of invalid calculations in the middle of large datasets.
Utilize these three questions to help you mature your testing approach as you adapt to the new technology benefits and challenges of AI. Realize that there are very few “experts” in the field of AI at this point. As we uncover new risks, such as the Horseshoe Paradigm, you will need to adapt your approach to account for them.
As AI becomes embedded in more of what we test and how we build, our job as Quality Engineers isn't shrinking — it's expanding into territory that didn't exist five years ago. The Horseshoe Paradigm is one piece of that map. For another, read my piece on Silent Confidence.
About the Author
Greg Paskal is a testing innovator with over 40 years of experience, known for pioneering sustainable and maintainable approaches to both manual and automated testing. He is the creator of METS (Minimal Essential Test Strategy) and the author of Test Automation in the Real World, available on Amazon. A dedicated mentor and lifelong learner, Greg shares his insights through writing and his YouTube channel, Craft of Testing.
Learn more at CraftOfTesting.com, METSTesting.com, and GregPaskal.com, or connect with him on LinkedIn.
 | A guest post by
|