My AI Test journey – the path less taken
A Line in the Shifting Sand: Navigating AI Testing with Standards
Conferences are filled with grand plans for AI. Billion-dollar companies pop into existence on AI bets. Papers and standards are being written on all aspects of AI, including using AI to drive testing and how AI should be tested (e.g. ISO). The future is so bright that I must wear shades, or is it so dark that I want to run and hide?
There are no clear answers or winners yet in AI. I have lived through history where computers appeared in businesses (1960-1970), personal computers became ubiquitous (late 1970s to 1980s), the internet was going to make us all rich (1980s to 1990s), the dot-com bubble (2000), the World Wide Web (WWW is everywhere all the time in the last few years), and now AI. There will always be winners and losers. My interest and this blog posting are about one small slice of AI, testing. This slice will not make you rich. The slice may help make AI-driven cyberism (look that one up) a good time. This post will at least give some references and maybe encourage you to help support the small test slice of the AI mega-pie. These are pointers to starting-point standards for testing, verifying, validating, and evaluating AI.
These AI efforts are done with the support of groups like IEEE and ISO (if you don’t know what those are, use your AI to look them up). There is already an ISO Joint Working Group 2 (designated JWG2) on AI testing. This team is a cross between AI supporters (ISO JTC1/SC 42) and test people (ISO SC7/WG2). You can follow along or even become involved in what is being developed and published by monitoring ISO.org websites or contacting me.
Currently, there are over 20 WG26 ISO/IEC/IEEE 29119 test standards that have either been produced or are in work. The standards provide specifics on the software testing process, documentation, techniques, and specialised areas, including automation, specific product domains, and performance. ISO 29119 Part 1 is freely available from ISO, while other ISO standards must be purchased. Two WG26 standards directly deal with AI. There is a standard’s lifecycle, and there are levels of maturity for these documents that I will not detail here, but again, you can use your AI to look up the details and numbers on specific standards. Some software testing standards are mature and published, while AI standards are in the process of becoming mature enough for release.
On the SC42/JWG2 side, there are 26 or more AI testing standards in progress. Again, maturity is being worked on. You can visit ISO.org and use its search feature to find relevant results.
You can become involved in these efforts through your company if it joins ISO or by going to the “Join IEEE” page. I am the IEEE/US representative for software and AI testing and can answer specific questions (feel free to contact me).
I know a lot of software people, AI supporters, and testers will argue with me and say that standards are complete nonsense, that as an industry, we are not mature enough, and that it’s a waste of time. To be fair, there are aspects of standards that definitely fit that description.
However, for me, standards provide a “line in the shifting sand” from which I can continue my learning and testing journey. Standards are not perfect, so thinking, engineering, and testing will always be needed. But I prefer the structure of standards over what I often see, particularly with some AI systems, which can be rather chaotic (though I do love me some chaos theory, at times).
I provide this blog so readers know about one of the many paths in the AI wilderness, particularly for AI software testing. As Robert Frost observed in his famous poem, as I stand at a crossroads, I will take the path less traveled.
Enjoy thinking, but use AI to help.
About the Author: Jon Duncan Hagar is a semi-retired systems software engineer and testing consultant specializing in software product integrity, verification, and validation, with a focus on embedded and mobile software systems.
With over forty years of experience in software engineering and testing, Jon has supported a diverse range of high-stakes projects, including:
Control Systems (avionics and automotive)
Spacecraft & Ground Systems (IT)
Mobile & Smart Devices
Attack Testing of new smartphones (with a class and book currently in development)
An active contributor to the engineering community, Jon has delivered over 50 presentations and authored two books spanning software testing, Agile methodologies, product integrity, and quality assurance. He also serves as an IEEE representative for the ISO 29119 software testing standards, the AI ISO JWG2 team, and numerous other IEEE standards committees.
