QA Partner for AI Software Companies

AI software companies ship at a pace that legacy QA vendors were never designed for. Your product is probabilistic, your model outputs change without a code deploy, and the security attack surface introduced by LLMs — prompt injection, data exfiltration through system prompt leakage, jailbreaks — has no equivalent in traditional web development. If your QA partner is still writing Gherkin scenarios to test happy-path flows and calling it done, you are not getting tested. You are getting a false sense of coverage that will cost you in production.

This guide is written for AI startup founders, CTOs, and Heads of Engineering at AI software companies in Australia and the USA who are evaluating QA partners or reassessing their current testing approach. It covers why AI products fail differently, what a specialist QA partner actually does differently, the engagement models that suit AI-native companies at each funding stage, and a clear decision checklist to use when evaluating any QA vendor.

Why AI Software Companies Need a Specialised QA Partner

Traditional software either works or it does not. An AI product sits in a third category: it works sometimes, in some contexts, for some users, and fails in ways that are difficult to reproduce and impossible to enumerate with a finite test suite. This is not a solvable problem with more manual test cases. It requires a fundamentally different testing philosophy.

• Probabilistic outputs require statistical validation — a traditional functional test asserts that input X produces output Y. An LLM test must establish that input X produces outputs within an acceptable quality band across hundreds or thousands of runs, with a defined tolerance for edge-case degradation.
• Data drift breaks AI features without a code change — when the underlying model is updated by the provider (GPT-4o, Claude 3.7, Gemini 2.0 updates all change output characteristics), your product behaviour changes without your team touching a line of code.
• Prompt injection is an AI-specific security threat class — the OWASP LLM Top 10 documents attack vectors unique to LLM-powered applications. Prompt injection (LLM01), insecure output handling (LLM02), and supply chain vulnerabilities (LLM05) do not appear in OWASP Web Application Top 10.
• Agent failures cause real-world consequences — an AI agent that can read email, write code, or submit forms can cause real damage when it goes off-script. You must test what it does when it encounters unexpected states, ambiguous instructions, or malicious inputs.
• The cost of shipping a broken AI feature is disproportionate — a hallucination that provides medical advice, a chatbot that can be jailbroken to produce harmful content, or an AI agent that takes irreversible financial actions based on a prompt injection attack generates consequences far larger than a typical UI bug.

The 5 Mistakes AI Dev Teams Make When Managing QA In-House

Most AI development teams we engage have not failed at testing through lack of effort. They have failed because the testing instincts built through years of traditional software development are systematically wrong for AI systems.

• Testing only happy paths — the most dangerous AI failures do not occur when a user asks a clearly normal question. They occur at the boundaries: ambiguous questions, adversarial inputs, questions the model was not designed to answer, multi-turn conversations where the context window fills in unexpected ways.
• No adversarial testing programme — adversarial testing for AI includes red-teaming prompt libraries designed to elicit harmful outputs, testing across demographic groups to identify fairness failures, and systematic boundary probing. Most in-house QA teams do not have the adversarial prompt libraries, the tooling (Garak, PyRIT, custom harnesses), or the methodology to do this systematically.
• Ignoring OWASP LLM Top 10 — the OWASP LLM Top 10 is the closest thing the industry has to a standardised security checklist for LLM-powered applications. LLM01 (Prompt Injection), LLM06 (Excessive Agency), LLM08 (Vector and Embedding Weaknesses), and LLM09 (Misinformation) are all testable attack vectors with documented methodologies.
• No regression harness after model updates — without an automated evaluation harness running on a representative golden dataset, you are discovering model update regressions in production after users report them — typically days or weeks after the update.
• No performance baseline for AI infrastructure — without performance baselines, you cannot detect when a model version change or infrastructure change has degraded your P95 or P99 response times to the point of affecting user experience.

What to Look for in a QA Partner for AI Products

The QA market has fragmented in response to AI. Some vendors are marketing AI-flavoured versions of their existing functional testing practices. Others have built genuine AI-native testing capabilities. Here is how to tell the difference.

• AI-native testing experience, not AI-adjacent — ask whether the vendor has tested LLM-powered applications, RAG pipelines, and AI agents specifically. Request anonymised case studies for AI product engagements. A vendor who pivoted to AI testing in 2024 by rebranding their existing automation practice is a different proposition from one who has been testing generative AI products since 2022.
• LLM evaluation methodology and tooling — a credible AI QA partner can describe their approach to hallucination measurement (do they use LLM-as-judge scoring? BERTScore? RAGAS for RAG pipelines?), their adversarial prompt library, and their approach to multi-turn conversation testing.
• OWASP LLM Top 10 testing capability — the vendor should be able to map their security testing methodology to specific items in the OWASP LLM Top 10. Ask them to describe their approach to testing LLM01 (Prompt Injection) and at least three other items. Vague answers indicate they have not done this work.
• Australian and US market presence and time zone alignment — for ongoing retainer engagements, time zone overlap is a practical requirement. A QA partner with engineers in Sydney, Melbourne, or AEST-compatible locations provides same-day communication cycles.
• Compliance knowledge relevant to your industry — for Australian AI companies: OAIC Privacy Act 1988, APRA CPS 234, TGA SaMD requirements, Australia's AI Ethics Framework. For US AI companies: SOC 2 Type II, HIPAA, NIST AI RMF, and emerging state laws including California AB 2013 and Colorado SB 205.

QA Engagement Models for AI Companies

The right engagement model depends on where you are in your funding journey, how much internal QA capability you have, and how frequently you ship.

• Embedded QA team (seed to Series A) — a dedicated QA engineer or small team working inside your sprint cycles, attending standups, writing test cases alongside feature development, and managing your evaluation harness. Cost: an embedded senior AI QA engineer through KiwiQA typically ranges from $8,000–$14,000 AUD/month (USD $6,000–$10,000/month) — significantly less than the fully-loaded cost of a senior in-house QA hire at $130,000–$180,000 AUD salary plus on-costs.
• Project-based testing sprints (pre-launch or major release) — a structured 4–8 week engagement focused on validating a specific release, new AI feature set, or compliance checkpoint. Cost: project-based AI testing engagements typically range from $25,000–$80,000 AUD depending on scope, with fixed-price deliverables agreed at project initiation.
• Continuous QA retainer (Series B and enterprise) — a standing engagement where KiwiQA owns the continuous evaluation harness, model update regression testing, and monthly security testing against the OWASP LLM Top 10. Cost: continuous retainer engagements typically range from $20,000–$45,000 AUD/month depending on the number of AI systems in scope and compliance reporting depth.

How KiwiQA Tests AI Applications: The AI Assurance Framework

KiwiQA's AI Assurance framework — Knowledge Validation, Adversarial Testing, Safety Checks, Compliance, Integration Testing — is our proprietary methodology for testing AI applications. Each phase produces structured deliverables that create a defensible testing record for compliance, enterprise sales, and internal governance.

• Knowledge Validation (K) — evaluating the factual accuracy, groundedness, and consistency of AI outputs against defined knowledge sources. For a RAG-powered product, this includes retrieval accuracy testing, citation fidelity, and hallucination rate measurement across a golden evaluation dataset. Example: for a legal AI product, we built a 2,400-question golden dataset across 12 practice areas and established a hallucination rate ceiling of <2% at P90 as the production acceptance criterion.
• Adversarial Testing (A) — systematic probing of the AI system's failure envelope using adversarial prompt libraries, jailbreak attempts, multi-turn manipulation sequences, and out-of-distribution inputs. We use Garak, PyRIT, and custom harnesses. Example: for an AI customer service agent, we ran 1,800 adversarial prompts across 9 attack categories and identified 23 guardrail bypasses that were remediated before launch.
• Safety Checks (S) — validating that safety controls — content filters, output classifiers, toxic content detection, PII detection, rate limiting, and session isolation — are functioning correctly and cannot be bypassed, while also testing for over-filtering that creates poor user experience.
• Compliance (C) — building the testing evidence portfolio required for regulatory compliance, enterprise due diligence, and internal governance. Includes OWASP LLM Top 10 coverage mapping, bias and fairness testing, NIST AI RMF alignment documentation, and audit-ready deliverables for APRA, TGA, and ISO 42001.
• Integration Testing (I) — validating the AI system's behaviour within its broader technical context: API contract testing, RAG pipeline component testing, tool/function call validation for agentic systems, and end-to-end workflow testing across multi-system sequences.

Testing Across the AI Stack: APIs, RAG Pipelines, Agent Workflows

AI products are not monolithic applications. They typically combine a model API, a retrieval layer, application logic, guardrail systems, and integration with external tools or data sources. Each layer introduces its own failure modes, and a testing approach that treats the whole stack as a black box will miss the majority of them.

• API layer testing — testing the model API integration for contract correctness, error handling, and latency SLA compliance. For production AI applications, API-level performance testing under load is essential before any high-traffic launch.
• RAG pipeline testing with RAGAS — we test both retrieval failures and generation failures independently and in combination using RAGAS metrics: faithfulness, answer relevancy, context recall, context precision, and context entity recall.
• Agent action validation — for AI agents with tool use capabilities, we test that the agent correctly selects tools, passes correct parameters, handles tool call failures gracefully, and does not take irreversible actions without appropriate confirmation steps.
• Multi-turn conversation testing — automated multi-turn test sequences that probe context handling, session isolation, and instruction persistence across long conversations where context drift and memory contamination failure modes emerge.
• Guardrail verification — guardrails tested for both effectiveness (adversarial inputs designed to evade detection) and false positive rate (benign inputs that should not trigger guardrails). Guardrails that block 15% of legitimate user queries are not an acceptable safety mechanism.

Australian AI Companies: Compliance and Testing Considerations

Australia's regulatory framework for AI is evolving rapidly. The key compliance obligations that shape testing requirements for Australian AI companies are distinct from US and EU requirements.

• OAIC Privacy Act 1988 and the Australian Privacy Principles — the Privacy Act's APP 11 (security of personal information) applies directly to AI systems that process personal data. The 2022 Privacy Act Review recommendations, currently being legislated, will introduce a 'fair and reasonable' test for data processing with direct implications for AI training data use.
• APRA CPS 234 and CPS 230 for fintech and healthtech AI — APRA-regulated entities building AI products face dual obligations: CPS 234 information security requirements and CPS 230 operational risk management requirements. For AI startups selling into Australian banking, insurance, or superannuation, demonstrating a CPS 234-aligned security testing programme is effectively a commercial prerequisite.
• TGA Software as a Medical Device for health AI — if your AI product makes or assists in clinical decisions, it is likely regulated by the TGA as a Software as a Medical Device (SaMD). Class IIb and Class III SaMD require documented verification and validation testing meeting IEC 62304 software lifecycle requirements.
• Australia's AI Ethics Framework — the Australian Government's AI Ethics Framework (AGAIEF) establishes the eight principles that government procurement and public sector AI deployments are evaluated against. AI companies selling into Australian government must demonstrate compliance with AGAIEF principles including fairness, transparency, and accountability.
• State vs federal considerations — Australia's regulatory landscape involves both federal frameworks (Privacy Act, TGA, APRA) and emerging state-level obligations. Victoria's public sector AI guidance, Queensland's AI strategy, and NSW's Customer Information Security Policy all set compliance expectations for government contracts.

US AI Companies: Compliance and QA Considerations

The United States AI regulatory environment is fragmented across federal guidance, agency rules, and a growing body of state legislation. Compliance testing obligations depend heavily on your industry vertical and the states where you operate.

• FTC AI guidance and unfair practices risk — the FTC's 2023 AI guidance and enforcement actions in 2024–2025 make clear that AI products making false or misleading claims about their capabilities, AI systems that perpetuate discriminatory outcomes, and AI that collects consumer data without adequate disclosure are all within the FTC's enforcement scope.
• NIST AI Risk Management Framework (AI RMF) — the NIST AI RMF (released 2023, updated 2024) is the most widely adopted voluntary framework for AI risk management in the US market. Enterprise customers, federal procurement, and insurance underwriters are increasingly asking AI vendors to demonstrate NIST AI RMF alignment.
• SOC 2 Type II for enterprise sales — the most common enterprise due diligence requirement for US AI software companies. For AI-specific controls, SOC 2 Type II increasingly requires evidence of AI system testing, model performance monitoring, and adversarial input handling.
• HIPAA for health AI — AI products that handle Protected Health Information are subject to HIPAA's Security Rule, which requires documented risk analysis for electronic PHI processing systems. This includes testing that the system does not generate or leak PHI in model outputs.
• California AB 2013 and Colorado SB 205 — California AB 2013 (effective 2026) requires developers of AI systems to publish training data summaries. Colorado SB 205 (effective 2026) requires algorithmic impact assessments for high-risk AI decisions. Both create testing obligations for bias testing across protected classes.

How to Evaluate Your AI QA Partner: A Decision Checklist

Use these ten questions to evaluate any QA partner you are considering for an AI product engagement. Strong answers indicate genuine AI-native capability; weak or vague answers indicate a traditional QA vendor who has added AI language to their marketing materials.

• Q1: Show me an example AI testing engagement you have delivered. What was the AI system type? What evaluation metrics did you use? A credible AI QA partner can describe a specific past engagement with concrete metrics and deliverables.
• Q2: How do you measure hallucination rate? The answer should include a specific methodology: golden dataset construction, LLM-as-judge scoring frameworks, or metrics like BERTScore and RAGAS faithfulness.
• Q3: Walk me through your OWASP LLM Top 10 testing approach. The vendor should describe specific test techniques for LLM01 (Prompt Injection) and at least three other items.
• Q4: How do you test AI agents with tool use capabilities? The answer should cover sandboxed tool mock environments, happy path and error condition testing, and irreversible action gating.
• Q5: How do you handle model update regression testing? The vendor should describe an automated evaluation harness, golden dataset versioning, and a process for comparing baseline metrics against post-update metrics within a defined SLA.
• Q6: What compliance frameworks do you have experience testing against? For Australian companies: OAIC Privacy Act, APRA CPS 234, TGA SaMD, AGAIEF. For US companies: NIST AI RMF, SOC 2 Type II, HIPAA.
• Q7: What AI testing tools do you use and why? Expect to hear about specific AI evaluation tools: Garak or PyRIT for security testing, RAGAS for RAG pipeline evaluation, DeepEval for LLM evaluation.
• Q8: How do you test for bias and fairness in AI outputs? The answer should describe demographic parity analysis and counterfactual fairness testing across protected attributes.
• Q9: What does your testing evidence look like for enterprise due diligence? Ask to see a sample deliverable structure — OWASP coverage matrices, metric dashboards, NIST AI RMF alignment documentation.
• Q10: How quickly can you start? KiwiQA can mobilise an embedded AI QA team within 5 business days, covering architecture briefing, evaluation dataset scoping, tooling setup, and first testing sprint.