I worked for a loooong time in a medium size development company (about 200 developers, mostly doing large web portals). My team was some kind of central DevOps in charge of architectures, cloud, technology stacks… we were ALWAYS involved in EVERY deployment, and we were directly in full charge of the big ones.
After many years of constant work alongside the DEV/QA teams my team had gotten REALLY good doing deployments (we mostly sailed on each of them, since all was well tested, prepared and automated), and the project leaders simply trusted us. In the scarce occasions we said “sorry, this is not ready for prod” they knew it was true and didn’t pressured us. And our customers were happy, since needing a rollback was EXTREMELY rare.
One of the most important things we managed to agreed with all the team leaders:
- Fridays are read only.
- No, that doesn’t means we all can go home: Friday is now “Documentation Day”.
- Of course, if shit hits the fan, we are ALWAYS ready to deploy fixes.
I think in about 10 years I only had one call on a weekend.
That “wet dream” was only possible after several years of hard work of dozens of people, and only because many other small pieces fitted in their proper place.
Hi, manager! You said you want environments for your developers without needing my intervention in every step? Of course, here you have infra and config automation: this is how you can create (and backup! and restore!! and destroy!!!) DEV, TEST, QA, PRE and DEMO environments, all ready with your specific stack and versions and code and data. And this is how much will it cost to you, and this is how you define a budget limit in case something gets out of control. Everything is repeatable and 100% reproducible in seconds, so please do not hesitate to test and test and test. (And no, sorry, I won’t let you touch PRO on your own, because that can cost a lot of money and we need to keep proper security).
So, you are asking me if we have heard about code versioning? Yes, of course! Here is a proper git structure, with predefined branches, segregated groups and permissions, and strict (and automated) revision requirements for every PR. I own the organization, you own the repo, QA owns the tests, and your developers own their branches and are self sufficient. Oh, and please remember we freeze the main branch 48h before the deployment, and time only begins counting after all the automated tests have passed and QA has given their final approval! No cheating!!
Oh, you have a picky customer who wants a guaranteed instant recovery in case THE WORST happens? Here you are, a highly available blue/green deployment, so you can deploy the new version without touching the old and only switch when everyone gives the final OK. And please remember to warn them it does cost DOUBLE the money!
Believe me, is not a wet dream, is just a lot of initial effort and A LOT of trust and confidence in the work of those around you.
And you have no idea how satisfying was begin work a Tuesday at 9AM sending a message “Hi, we are starting deployment in PRO” and then less than 5 minutes later reply saying “Hi, all is finished and checked OK from all parts, thanks to everyone and see you next week”.