I run a nightly pipeline that re-scores around 1,000 instruments.
One of the model personas uses company fundamentals. For crypto, it should simply abstain, because there are no company fundamentals to evaluate.
But for SOL, it wasn't.
The provider was being called with only the ticker symbol, and SOL is also the ticker of a NYSE-listed solar company. So the system was scoring a crypto pair using a solar company's P/E, ROE, and other fundamentals.
I added an asset-class check and tested it against live data. Crypto returned nothing, equities still returned the correct fundamentals, and the tests passed. I deployed it and rebuilt everything.
Then the nightly pipeline ran and still wrote all seven persona results for SOL, including the two that should have abstained.
At first this made no sense. The fix was correct. The test was correct. The new code was deployed.
The cache was keeping itself alive
The scores are cached for 26 hours so the dashboard does not recompute the full universe every time somebody opens it. The nightly job runs every 24 hours.
That small two-hour overlap was enough to break the whole assumption.
Each nightly run was reading the previous night's cache entry before it expired. It then used that old result and wrote a new cache entry, which was still there for the next run. So the stale result was basically keeping itself alive.
I already had a cache version, but the comment said to bump it when the response dictionary gains or loses a key. That is also how I had always used it. In this case the structure did not change. Only the meaning of one result changed, so I did not think about bumping the cache version.
Four explanations that looked reasonable
I also wasted time checking the wrong Redis database, reading an old TTL as if the entry had just been written, and wondering whether the scheduler was using a different code path.
Each explanation looked reasonable until I checked it properly.
What finally exposed the problem was looking at the actual production data after the nightly run and asking a very simple question: did the output change in the way this fix should have changed it?
That is the part I had not verified.
What I'd take from it
Testing a fix proves the code works. It does not prove the running system is actually using the new behaviour.
Caches, stale workers, old images, feature flags, bind mounts and similar things all sit in that gap. The check that closed it was comparing production output against what the fix should have produced — not against the absence of errors, and not against the test suite.