Chaos Engineering for QA Teams: System Resilience

The hardest category of production bugs to find in pre-production is the class of failures that only occur when things go wrong at the infrastructure level: a database replica falls behind, a downstream API starts timing out at 30% of requests, a Kubernetes node evicts pods under memory pressure, a cloud region degrades. Your functional tests pass. Your load tests pass. Your integration tests pass. And then an infrastructure event your test suite never considered causes an outage that costs real users and real revenue.

Chaos engineering is the discipline of proactively introducing controlled failures into a system to discover how it behaves under adverse conditions — before those conditions occur naturally in production. The term was coined by Netflix's engineering team, who published their Simian Army approach in 2011 after discovering that the only way to build genuine confidence in a distributed system's resilience was to continuously verify it could survive realistic failure scenarios. The discipline has since matured from a Netflix-specific practice into a standard engineering methodology supported by a rich tooling ecosystem.

The Core Hypothesis-Driven Approach

Chaos engineering is not random destruction — it is a scientific method applied to system reliability. Each chaos experiment follows the same structure: define the steady state (what does normal look like? latency p95, error rate, throughput), form a hypothesis (if we inject X failure, the system will continue to maintain steady state because Y), run the experiment with the smallest possible blast radius, observe what actually happens, and compare it to the hypothesis. If the system behaves as expected, you have increased confidence. If it doesn't, you have discovered a real failure mode that needs to be fixed before production teaches you about it the hard way.

The emphasis on steady-state metrics before running any experiment is what separates rigorous chaos engineering from ad-hoc failure injection. Without a defined baseline, you cannot objectively evaluate whether an experiment affected the system. Steady-state metrics should come from your observability stack — latency percentiles, error rates, successful transaction rates, queue depths — and should be specific enough that a 10% degradation is detectable. Dashboards that show 'roughly normal' traffic are insufficient; you need numerical thresholds.

The goal of chaos engineering is not to cause chaos — it's to prove that your system is resilient to chaos. Every experiment that passes increases deployment confidence. Every experiment that fails increases system quality. Both outcomes are wins.

Common Failure Scenarios to Test

• Network failures — packet loss, latency injection (adding 200–500ms to specific service calls), connection drops between services
• Dependency timeouts — downstream APIs that start responding slowly or not at all; do circuit breakers engage correctly?
• Resource exhaustion — CPU stress, memory pressure, disk I/O saturation on individual nodes or containers
• Instance/pod termination — randomly terminating healthy instances to verify autoscaling and failover work as designed
• Database failures — primary/replica failover, connection pool exhaustion, slow query injection
• DNS resolution failures — verifying services handle DNS lookup failures gracefully rather than crashing
• Region or availability zone degradation — verifying multi-AZ or multi-region architecture actually survives the failure it was designed for

Tooling: Chaos Monkey, LitmusChaos, and AWS Fault Injection Service

The chaos engineering tooling ecosystem has matured significantly. Chaos Monkey (Netflix's original tool, now open source) randomly terminates EC2 instances in production — it remains the simplest way to verify autoscaling and recovery. LitmusChaos is a CNCF project purpose-built for Kubernetes, providing a rich library of chaos experiments including pod deletion, node drain, CPU/memory stress, network partition, and disk fill — all declaratively defined as Kubernetes custom resources and executable via a web UI or CI pipeline. AWS Fault Injection Service (FIS) is AWS's managed chaos engineering service, providing experiment templates for EC2, ECS, EKS, RDS and Lambda with fine-grained controls, rollback conditions, and CloudWatch integration. Gremlin is the leading commercial platform, offering a comprehensive catalogue of failure types with team collaboration features, automated scheduling, and detailed reporting.

Tool selection should follow infrastructure: teams on Kubernetes typically start with LitmusChaos for its native integration and zero additional cost. Teams heavily invested in AWS benefit from FIS's managed service and native observability integration. Teams running multi-cloud or hybrid infrastructure, or those needing enterprise governance features, typically evaluate Gremlin. All of these tools support the same core workflow: define experiment, set steady-state thresholds and rollback conditions, run in a controlled environment, observe, and analyse.

Starting Safely: The Minimum Viable Chaos Programme

The most common barrier to starting chaos engineering is the perception that it requires a mature, Netflix-scale engineering organisation. It doesn't. A minimum viable chaos programme for a team that has never run chaos experiments has four steps: 1. Instrument first. You cannot run a useful chaos experiment without observability. Before any failure injection, ensure your system emits latency, error rate, and throughput metrics, and that you have a dashboard showing steady state. 2. Start in non-production. Run your first experiments in staging or a production-equivalent environment, not in live production. This validates your experiment design before real users are affected. 3. Start with known failure modes. Your first experiments should be failure scenarios you already know your system handles — instance termination when autoscaling is configured, failover when you have a read replica. Building confidence with predictable experiments before probing unknown territory is the correct progression. 4. Automate and schedule. One-off chaos experiments provide one-off assurance. The value compounds when experiments run continuously on a schedule, verifying that system changes haven't regressed resilience properties that previously passed.

How QA Teams Lead Resilience Testing

Chaos engineering has historically lived in SRE or platform engineering teams, but QA teams are well-positioned to lead the programme — particularly the hypothesis formation, experiment design, and results analysis phases that benefit from structured testing methodology. QA teams already own performance test environments, understand how to define acceptance criteria, and have the process discipline to run experiments rigorously rather than ad-hoc. The most effective chaos programmes KiwiQA has implemented involve QA engineers owning the experiment catalogue and cadence, with SRE or platform teams providing the infrastructure access and tooling configuration.

KiwiQA's performance and resilience testing practice delivers chaos engineering engagements as a structured service: steady-state baseline definition, experiment catalogue design for your specific architecture, execution in a controlled environment, findings report with prioritised remediation recommendations, and implementation guidance for a continuous chaos programme post-engagement. For organisations preparing for high-traffic events, regulatory audits, or cloud migration, a pre-launch resilience test engagement provides concrete evidence of system stability under adverse conditions. Contact KiwiQA's performance testing practice to discuss a resilience testing engagement tailored to your infrastructure.