> "But we run the tests on every commit! If they fail it must be the latest change."
I really want to share a great failure here I've seen:
This was in a django web app, nearly a decade ago. First team I was on professionally, and took around a week to figure out (no one else cared because it was so rare, so I was the one looking for it): memcached would occasionally return corrupted data, that python couldn't un-pickle into an object, because the class attributes didn't match.
Turned out to be because of how our django settings were organized, a large collection of shared settings that no one ever looked in because they were set up once and then ignored, a set of overrides for live, a set for beta, and each dev got their own. The trick was that, to make it easy to set up, the memcached host/port were in the shared, live, and beta files - but not overridden in dev, so all the devs and the CI suite shared the same cache.
This meant that when one dev was changing the structure of one of these cached objects, before committing it all tests relating to it could fail for all devs and in CI.
I really want to share a great failure here I've seen:
This was in a django web app, nearly a decade ago. First team I was on professionally, and took around a week to figure out (no one else cared because it was so rare, so I was the one looking for it): memcached would occasionally return corrupted data, that python couldn't un-pickle into an object, because the class attributes didn't match.
Turned out to be because of how our django settings were organized, a large collection of shared settings that no one ever looked in because they were set up once and then ignored, a set of overrides for live, a set for beta, and each dev got their own. The trick was that, to make it easy to set up, the memcached host/port were in the shared, live, and beta files - but not overridden in dev, so all the devs and the CI suite shared the same cache.
This meant that when one dev was changing the structure of one of these cached objects, before committing it all tests relating to it could fail for all devs and in CI.