Reducing 2AM headaches Part 3: Resiliency
The title of this series underscores our motivation for building a toolbox for system management, silencing the pager. In the first part of the series, we discussed the importance of standardization. We then talked about automation. As we conclude the series, we turn our focus on resiliency.
Reducing 2AM headaches part 2: Automate
Last time we talked about standard operating environments and how they can reduce stress and give back time. Now we'll talk about managing not only systems but the SOEs themselves. Repeatable, reproducible, and remote are the watchwords for managing our environments.
Reducing 2AM headaches part 1: Standardize
One of the most effective ways to reduce fire fighting in daily administration is by standardizing the operating environments and automating deployment and configuration. A standard operating environment (SOE) that can support multiple use cases is a more robust and tested platform to build upon. It provides a uniform environment for troubleshooting when something goes awry. Reducing the differences in your operational environment to critical changes also reduces the overall complexity in multi-tier environments. Using centralized automation tools to define, build, and deploy these standards streamlines the process even more.
Standardization is not a new concept, nor is it disruptive way of thought. The industrial revolution owes part of its existence to standardization. Computers and gadgets get reviewed and reviled based on adhering to standardized parts and ports. Yet for some reason, every environment I've worked in has one-off, bespoke systems to one degree or another. Some had admins who thought it was easier, better, smarter, more secure to build custom environments. Some had admins who wrote wrapper scripts around standard UNIX utilities because they didn't like the way a particular error was handled. The only real outcome was increasing difficulty of maintaining and replicating the systems. While bespoke suits will fit better at first, you'd best be prepared to work hard to exactly maintain your shape otherwise you're in for regular and expensive tailoring.