QA Testing
The QA/Developer Line Is Changing - Here's Who's Filling the Gap
Last updated:
8 min read

Who actually has the skills to test it?
A few weeks back I wrote about why testing AI systems breaks some of the assumptions behind traditional QA.
Fixed input. Expected output. Pass or fail.
That model doesn't always work when the same prompt can produce different, equally valid answers.
But once we understood what needed to be tested, another question became harder:
Who actually has the skills to test it?
On the same EdTech assessment platform I mentioned in my previous post, our existing QA team was experienced and strong at traditional testing.
The problem wasn't their testing ability.
The problem was that the system they were being asked to validate required a different set of skills.
There wasn't always one expected answer. Evaluating an output required understanding AI behavior, the assessment framework, context, data and what constituted an acceptable response.
For a while, developers ended up owning much of the testing and model-evaluation effort. Not because developers necessarily wanted the extra responsibility, but because they were the people who had the required AI and data skills.
This got me thinking less about “QA versus developers” and more about:
What skills does AI quality actually require?
The first is AI literacy.
People need to understand why non-determinism isn't necessarily a defect, how hallucinations happen, how grounding affects an answer, where RAG systems can fail and how an agent decides to use a tool.

Without that foundation, it's very easy to test an AI system as though it were deterministic software.
The second is Python.
AI evaluation quickly becomes data-driven. You need to process datasets, run repeated evaluations, compare outputs, calculate metrics and automate regression checks.
The third is SQL and data understanding.
This became particularly important with database agents. A generated SQL query can be syntactically correct and still answer the wrong question.
To validate that, you need to understand the underlying data and independently verify the result.
AI evaluation
Then comes AI evaluation itself: golden datasets, evaluation criteria, LLM-as-judge, human calibration, semantic relevance, groundedness and other measures of quality.
And with agentic systems, you also need to understand traces.
The final answer may look wrong, but the actual problem could have happened several steps earlier:
User request → agent decision → tool selection → retrieved context → database query → downstream agent → final response.
Understanding where the failure actually occurred becomes part of testing.
Security is another area that increasingly overlaps with AI quality: prompt injection, unauthorized data access, PII exposure, unsafe tool calls and adversarial inputs.
So where does this leave QA?
I don't think AI makes QA less important.
I think it makes QA more technically diverse.
The traditional strengths of QA: test design, exploratory testing, automation and thinking about failure modes , remain extremely valuable.
But increasingly, they need to be combined with AI literacy, Python, SQL, data analysis,
evaluation techniques, trace analysis and security awareness.
And developers don't get a free pass either
AI-assisted coding can now generate code, tests, documentation and even review suggestions.
That makes independent validation more important, not less.
The roles may remain separate on the organization chart, but the responsibility for quality is spreading across the entire AI lifecycle.
At BUSoft, we're treating this primarily as an upskilling challenge, not just a hiring challenge.
The path we're working toward is fairly simple:
AI literacy → Python → SQL → AI evaluation → trace analysis → security and production
monitoring.
The goal isn't to turn every QA engineer into an AI researcher.
It's to make sure that the people responsible for validating an AI system understand how that system can fail.
Because ultimately, AI quality isn't just about whether the model produces an impressive
answer.
It's about whether we can systematically determine:
When is the answer good enough?
When isn't it?
And why?
That's a very different definition of quality: and an important capability as AI moves from
experimentation into production.
If figuring that out is the kind of problem you want to work on: whichever hat you currently wear: we're hiring at BUSoft.
Tags
Predictive Analytics
Customer Experience
Data Strategy
Written by
Ram
Co-Founder & CTO





