Modern Software Development: A Continuous Journey, Not A Delivery Destination

My wife and I recently hiked the Stony Man Trail in the Shenandoah National Park. It was so much fun that we hiked it again a few weeks later with our college-aged kids and close friends. The four-mile hike leads to a spectacular summit, but also provides beautiful views of different vistas along the way. Our group hiked at different paces, so my wife and I suggested short goals for everyone to pause at trail crossings so we did not have large gaps in our convoy. We took pictures along the way and at the summit, then proceeded back down the mountain to our cars... then to a local restaurant for well-deserved burgers and beers.

As I hiked, it occurred to me that the hike can be thought of as a metaphor for software development. I have witnessed changing approaches to delivering quality software products, and more importantly, the varied perspectives as to what constitutes “done” in terms of an application. The waterfall approach used to be the de facto standard as customers demanded control over the entire development process and much effort was spent upfront defining what was to be delivered with a very specific set of capabilities. From a developer's perspective, the hand-off to the Operations team at the end of user acceptance testing constituted “done” as it was now in maintenance mode. This approach was appropriate given customers expected their software to be designed, constructed, tested, and delivered in a disciplined, documented, and standard way so that they could govern the process and ensure their money was spent for the product they intended. The software was delivered per schedule come hell or high water, the last few weeks or months on a delivery project were typically a “death march.” If you kept the development team intact for the initial release that meant there was most likely construction still needed to be done or there were significant defects in the software that required correction prior to customer acceptance. If activities were on schedule, then development team members were usually shifted to other construction activities on other projects.

Then and Now

Along came Agile methods like XP, Lean, Scrum, etc. that promoted a more iterative approach that emphasized feedback loops, working software delivered in increments, less upfront documentation, incremental acceptance of change or adjustments as more information is uncovered, and a more communicative and cooperative culture that includes all stakeholders associated with the software product. Agile works well for developers but there was still a missing delivery phase to get that software to end users in a continuous fashion working the same in pre-production environments as it would in production. Now, the DevOps method expands that sphere of stakeholders to include Operations staff that constantly accept stable versions of the application into production and maintain its infrastructure. This cycle repeats until there are no more application enhancements, defects, or financial backing to continue. The new “done” is a misnomer as the application is never “done” until it no longer serves a purpose, has no users, or so much technical debt has accumulated that is more cost effective to replace it. The new “done” means the application is decommissioned and removed from all environments.

The new software development paradigm is to develop an application in stages, implementing prioritized features as necessary to provide continuous value, removing technical debt to ensure reliability, and maintaining a cross functional team of knowledgeable resources through its life cycle until it reaches the end of its usefulness. The private sector lives with this mindset – focusing resources on the most valuable services – because it can't afford to maintain custom applications that provide no value to its end users and still remain competitive. The public sector is looking more and more at how the private sector performs software development and while it has different constraints and challenges, it cannot afford to continue to deliver applications as a goal in and of itself. It must look at the entire life cycle from womb to tomb.

I recently attended a technical conference where a large military department conveyed their needs for a more flexible, reactive, scalable platform for their application development, testing, and production use. Government representatives said they want their platform to allow development systems to “go away” once they move into testing, and to allow testing systems to “go away” once they move into production. They also indicated that they are trying to save money on “pay for use” systems – bringing them down at night when they are not active and starting them again the morning when users need them to do their jobs. While I understand and appreciate as a tax payer the goal of cost savings and not paying resources when they are not in use, this mindset completely misses the context of modern application development approaches. Whether it be a form of Agile or all the way to a DevOps approach, a modern application “lives” and needs continuous improvement until its value is no longer needed by the organization. It did not surprise me that this organization has over 8000 applications that need to be inventoried and evaluated to determine their need and purpose.

“End of the Journey”

This brings me back to my hiking experience. If we set the goal as reaching the summit and paid no attention to getting back to our cars (and eventually to our burgers and beers), we may have expended all our energy to attain that goal – with no means to get back down the mountain. One absurd scenario would require a rescue helicopter to get us back down, diverting lots of resources for our myopic adventure. A more morbid scenario is we are all orphaned at the top of the mountain until someone finds our corpses. Celebrating a milestone is encouraged from a team building and morale perspective but an “end of the journey” theme is usually conveyed to the development team upon initial software delivery. Spiking a football when achieving a first down on your own 40 yard line seems ridiculous but I've seen many major delivery “win” parties interrupted by an emergency outage from the newly released software, and I'm a bit jaded. After adopting an iterative, thoroughly tested, and well-coordinated delivery of production-ready code, our development and operations team's sleep was restful and our celebrations were enjoyable. Just like setting periodic goals for my hiking group when the trails changed, we regrouped to ensure everyone was still feeling fine and could attempt the next milestone. There were some detours along the way but we all remained together. The summit was a significant milestone but not the ultimate goal, we had more hiking to do.