Flaky tests get blamed on timing issues. On race conditions. On test data that wasn't cleaned up properly. On CI infrastructure that behaved differently at 2 am than it did at noon.
The dependency is rarely the first suspect, and that's the problem.
What flakiness actually looks like when a dependency is causing it
A test passes locally. It fails in CI. It passes again on retry. Nobody changes the code and it starts failing two weeks later. The team marks it quarantined, writes it off as environmental, and moves on.
What's usually happening: the test is hitting a live upstream service, or a simulation that's accumulated enough drift from the real service that its responses are no longer consistent with what the application code expects. The failure isn't random. It's a contract mismatch showing up as noise.
The difference between a genuine race condition and a dependency-caused failure is nearly impossible to diagnose from a CI log. Both look like intermittent failures. Both retry-pass sometimes. Both disappear and reappear without code changes. Teams reach for the easier explanation - timing, infrastructure because the dependency isn't supposed to be the problem. The mock was written to handle this.
Where the simulation breaks down
Hand-written mocks have a maintenance problem that's invisible until it isn't. The mock was written once, against the upstream as it existed at the time. The upstream kept deploying. The mock didn't update.
What the simulation returns and what the real service returns have been diverging ever since - but no test is failing yet, so nobody knows. The divergence accumulates across response fields, error formats, pagination behavior, header values the application has started depending on. Eventually something in the application code exercises a path that depends on a field the mock stopped returning accurately, and the test starts behaving unpredictably.
This isn't a flakiness problem. It's a contract accuracy problem that looks exactly like a flakiness problem.
Service virtualization built from observed traffic doesn't start from what a developer believed the upstream returned. It starts from what the upstream actually returned during a recording session against the real service. The simulation reflects observed behavior - including response structures, edge cases, and fields that documentation never mentioned because they were just there.
When the upstream changes, re-recording produces a diff. The team sees which fields shifted. The simulation gets updated before those changes surface as test failures that take a week to trace back to a stale mock.
Why retry logic makes this harder to see
Most CI configurations retry failed tests automatically. The rationale is reasonable - infrastructure blips happen, and a single retry filters out the noise.
What retry logic also does: it masks dependency-caused failures that are intermittent rather than consistent. The test fails because the simulation returned something the application code didn't expect. The retry runs against the same simulation. If the failure is tied to a specific code path that isn't always exercised in the same way, the retry passes. The failure gets swallowed. The underlying contract mismatch stays invisible.
Over time, the test suite's flakiness rate quietly climbs. The team adds more retry logic to compensate. The signal-to-noise ratio keeps falling. By the time someone investigates seriously, the stale simulation has been causing intermittent failures for months.
The CI determinism that service virtualization actually provides
Replacing the live call entirely is what breaks the feedback loop. The test hits the virtualized service. It gets back a recorded response. The result reflects whether the application code handled that response correctly - not whether the upstream was available, not whether a rate limit fired, not whether a response field that the simulation hasn't been updated to reflect was missing.
Keploy captures real HTTP traffic at the kernel level through eBPF, which means re-recording against a staging environment produces an updated simulation from actual observed upstream behavior. The diff between the previous recording and the current one is explicit before any test runs against the stale version - not discovered through a flaky test report two weeks later.
The flakiness that comes from dependency drift isn't random. It has a source, and the source is that the simulation stopped reflecting reality at some point and nobody had a mechanism to find out. Service virtualization doesn't eliminate every source of test flakiness. But it removes the one that's hardest to diagnose because it hides behind symptoms that point everywhere except where the problem actually is.
What changes when the dependency is no longer a variable
When the virtualized service returns consistent, accurate, recorded responses on every run, flakiness patterns that were attributed to infrastructure or timing become attributable to actual code behavior. A test that fails consistently against a current simulation is telling you something real. A test that was failing intermittently against a stale mock was telling you nothing useful.
The diagnostic value of the test suite goes up not because more tests are passing, but because the ones that fail are failing for reasons that can be fixed.
Sign in to leave a comment.