I code at the interface between ops teams (on the business side) of companies and dev teams (on the IT side).
One of the things I've realized is that in most unregulated companies (read: non-healthcare/financial) the business side of the house is used to having little or no lower lifecycle.
If they want to make a process change, they make it on production work.
Granted, they have change control approvals, etc. etc., but the whole dev-test-prod cycle looks extremely different for them, because you can't do certain things without lower environments.
This hasn't been my experience. I think it depends on how business-critical the application is.
I worked at a home remodeling company. Revenue was several million dollars a day. App handled sales, scheduling, logistics, everything. Breaking production was a big deal, it cost us millions per day and created logjams.
I would think that most online applications are the same. Even if a simple online web shop goes down you are costing money.
What kinds of experiences have you had where testing in production was the norm?
because you can't do certain things without lower environments.
I agree that this is something many shops REALLY struggle with.
One of the most challenging things is exporting or creating some kind of realistic data set for local development use. I think 99% of companies struggle with this.
Well, the lucky part is the important one. What it boils down is that such system is either self-contained or has rigidly defined outside interfaces. For anything that deals with a physical reality outside of the pure computational realm this tends to be impossible. You are not going to build an entire warehouse to serve as the physical part of testing environment and even if you did so it will not be really useful, because the thing will be different than the production one due to who knows what tolerances involved in building a physical things. In same vein if you interact with external services you can either mock them or use whatever testing environment the communication partner provides, in both cases it is bound to not behave the same way as the actual production environment.
