The Road to DevOps Transformation

Tom Whiston Jun 15, 2018
The Road to DevOps Transformation

For anyone that has been following the Agile and Dev(Sec)Ops movements, the growing number of success stories you hear about adopting this model can be truly inspiring. However, like all things of value, it is a hard journey to really embrace DevOps and bring it in to the heart of your companies values. In this post I will be looking at some things to consider, and common pitfalls, from both a management and developer perspective, before you start your journey towards thousands of daily pushes to production.



Would you like to know more about the three DevOps principles?

Then we recommend the following blog article
«The Three Ways - the DevOps Principles and Why Other Teams in a Company Could Benefit from Them».

Read the article now


Devops from the Leadership Perspective

"And for my next number, I'd like to return to the classics."

It’s been said before but it bears saying again until you can recite it by heart; DevOps is not a prescribed set of tools and processes that you can apply for success. There are, of course, certain patterns that are common but the success or failure of your DevOps transformation will live or die in the mindset of the company. The most important role of the leadership during a DevOps transformation is to ‘get out of the way’ and invest trust in your teams to make the right choices….

Lead Don’t Manage

Although there is often buy-in to the idea of DevOps from the engineering teams in reality DevOps involves a radical change to the way that you work across a company and this needs to be reflected in the top level culture. The 2017 State of Devops report [1] states that there are 5 characteristics of transformational leadership - vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition. Gartner also predicts that by 2020 “50% of the CIO’s that have not transformed their capabilities will be displaced from the digital leadership team.[2]

Managers need to become leaders, giving up the idea of making all the decisions and instead placing trust in their employees, as subject experts, to find and pursue the right course of action. Building this high level of trust and creating a ‘Zero Blame Culture’[3] is never easy, and if you have ever worked in a large and bureaucratic company maybe you know how truly difficult this change can be to make, but this is the absolute most important thing that the leadership can do to support becoming more DevOps oriented.

DevOps Is Not a New Team

If your plan is to make a new DevOps team you are doing it wrong! DevOps mindset and practices should be adopted across the whole company to gain maximum benefit. Therefore introducing a new team simply shifts the problem on to a different set of responsible people. You should embed DevOps oriented people in to each team, allowing the team to learn and grow the skills needed to become self sufficient and high performing. But bear in mind that “There’s no such thing as a DevOps Hero[4] and your DevOps specialists should be aiming to share knowledge and help grow a team that can deal with challenges rather than taking them all on their shoulders.

Don’t Accept Contracts That Don’t Share Your Values

We all need to make a living, however you can guarantee that if you announce a new DevOps or agile transformation and the next day present a classic waterfall project, with long feedback loops, you are not going to generate trust or respect from your teams. Steve Parks of Convivio put it best in his 2016 Drupalcon presentation [5], when asked what to do about customers who don’t want to work with you in an agile manner he replied “I would get a different client!”. As you become more skilled in DevOps flow you might be able to bring companies in to your methodology and cadence, but when initially attempting to introduce DevOps you should be looking to find customers that reflect and reinforce your values and practices.

Internal Improvements Create Indirect Value

It has been recognised that there is a period where once the transformation towards becoming more DevOps oriented begins a slowdown appears to occur [6]. This is often because when undertaking the DevOps journey your teams will start to really bring in to focus where there are issues in workflow and where tools and processes need to be improved. This “mountain of technical debt” [7] will need to be paid back so that the team can become high performing and it is extremely important that the management supports these actions.

Even if they do not directly bring value today try to take the bigger picture in to account. For example: spending some time investing in building the correct monitoring and alert systems today could save hundreds of man hours in future and even help to avoid developer burnout. Management and teams must embrace the idea of Kaizen [8], and constantly iterate over tools and processes to solve problems in a small and sustainable manner. Encouraging this not only leads to a better daily work environment but empowers your teams to define roadmaps themselves, grow their problem solving skills and feel like they are making a real contribution to the company.

Devops From the Developer Perspective

For a developer a DevOps transformation means building new tools and processes, breaking down silos, working more closely with people from other disciplines and taking ownership of your work. By paying back the trust placed in you, continually improving your working environment and striving to shorten the feedback loop between you and the customer you can help to create and maintain the flow needed to produce quality fast.

Get Used to Customer Interaction

Our goal is to have every business metric be actionable - these top metrics should help inform how to change our product and be amenable to experimentation and A/B testing."[9]

Making the value chain shorter means that you need to really understand what the customer wants and how to process their feedback into actionable work. This means that you need to get used to communicating with the customer (be they internal or external to your company). I have seen quality programming teams stumble with this step, either being unwilling to show or discuss their work to the customer, finding the feedback unwelcome, or just not listening to it at all. This is a mindset that needs to be rooted out and destroyed. Bringing value to the customer is the main priority of any team, however beautiful and clean and well tested your code is if it does nothing that the customer wants it’s worthless to everyone except your ego.

Make Software Releases Boring

If you have ever worked on delivering a clients application in a traditional development approach you will probably know the fear of an upcoming release, how long will it take for this to be approved by the change board? Will all the deployment steps be undertaken successfully? What actually are the deployment steps needed? Has the ops team prepared everything needed for the deployment to be successful? Did the development team communicate successfully to the ops team all the required changes? How much time to we need to budget in case things go wrong? How much money will we need to spend on pizza and beer to keep the team happy while they pull an all nighter fixing things?

These sorts of releases create a dangerous situation in any company, they risk your reputation with the client, they risk inter-team relations as everyone tries to shift the blame, they risk your employees questioning their own skills, and in the end they lead to a situation where neither the customer or the developers actually want their code to go live. This is a spiral of doom as the key goal of any software development process is to deliver value to the stakeholders and you can’t do this without releasing your work. Therefore you should focus on making the release process as boring, automated and pedestrian as possible.

Bring Quality to Production Faster

Puppet 2017 State of Devops report found that:

“In 2017 (...) high performers have:
- 46 times more frequent code deployments

- 440 times faster lead time from commit to deploy

- 96 times faster mean time to recover from downtime

- 5 times lower change failure rate (changes are 1/5 as likely to fail)


All of these are only possible if you can get code to production in a fast and safe manner. We need to put in to place deployment practices that support easy deployments and rollbacks in case of issues. In a practical sense this means:

  • Immutable environments / Atomic Changes - Once something is deployed you cannot change it without going through another deploy cycle, this prevents configuration drift and manual changes that might be forgotten or erased by the next push.

  • Infrastructure as code - Changes to your environments should be delivered through an automated system and should be tracked via version control.

  • Thorough QA process - The four eyes principle[10] in concert with automated CI pipelines can help to build quality in to your applications and can be used as a gatekeeper for releases.

  • Practice failure regularly - Make sure it is easy to roll back your code, push new fixes and return to known good states. Having clear, easy and (preferably) automated routes back to working states ameliorates a large part of the risk associated with a high velocity of pushes to production.


Test in Production (With Good Metrics)

One of the most critical items that you should have in place before attempting to move fully in to a DevOps workflow is good instrumentation, logging and metrics of your application. As you increase the pace to market you must ensure that all errors become visible, and that the team is empowered to spend their time working on them. For some companies, web agencies in particular, this change may be hard. The traditional model suggests that once a feature or project is handed off to a customer the team moves on to the next item, never looking backwards.

This is a foolhardy approach to take, it is almost impossible to really test you application with certainty before it hits production, opening your application to wild network requests and having people really use the application is guaranteed to results in unforeseen errors (if you don’t believe this statement just think about how many errors, large and small, you have seen on the websites of the largest and most well resources tech companies) and although this in no way decreases the importance of unit or integration testing your application it does mean that you need to consider an additional layer of monitoring to ensure that your applications behave in the wild[11].

Antithetical to this is the simple fact that nobody likes to be woken up at 2am (and I will guarantee that you won’t be happy if your monitoring system is waking you up to tell you something trivial happened), so use your monitoring systems to build real actionable alerts. There is a whole ecosystem of companies growing to meet this need now, doing high cardinality short term storage of events and alerting[12] or using ML to help filter out false positives[13].

Invest Time in Improvements

As the workers ‘at the coal face’ of development you will be the people who see and feel the pain points in your daily workflow. You should ensure that you budget enough time in your planning to deal with these issues, be they technical debt, small fixes or greenfields projects. If you see something that slows you, and everyone else, down then fix it. Don’t live with a state of ‘just about works’ or even ‘good enough’, push to make your processes as automated, simple and foolproof as possible, because everyone will benefit.

Conclusion

Inevitably one article cannot hope to cover all the concerns and edge cases that may arise when trying to move to a DevOps model, however I hope that at least when considering the above broad points it is abundantly clear that DevOps in more than just a simple set of rules to apply. You must be prepared to take a deep dive in to your tools, methods and culture, and take the time to improve and refine them to become a successful high performing organization. DevOps methodology has a proven advantage but the road there is not easy and takes time and effort across the whole of an organization to implement properly.



Are you looking for an approach for more effective and efficient collaboration between development and operations? We are looking forward to supporting you with our DevOps Readiness Assessment. Get more information here.


[1] https://puppet.com/system/files/2017-10/2017-state-of-DevOps-report-puppet-dora.pdf
[2] https://www.gartner.com/binaries/content/assets/events/keywords/infrastructure-operations-management/iome5/gartner-predicts-for-it-infrastructure-and-operations.pdf
[3] https://codeascraft.com/2012/05/22/blameless-postmortems/
[4] https://www.nwcadence.com/blog/devops-culture-not-team
[5] https://www.youtube.com/watch?v=_FsZoez6U_M&index=45&list=PLpeDXSh4nHjQAM63-KW_8x_qEB-dio1tA&t=0s 
[6] https://puppet.com/system/files/2017-10/2017-state-of-devops-report-puppet-dora.pdf page 28
[7] http://samuelmullen.com/articles/the-high-cost-of-technical-debt/ 
[8] https://en.wikipedia.org/wiki/Kaizen
[9] DevOps Handbook
[10] https://blog.titansoft.com.sg/2015/07/08/two-man-four-eyes-principle/
[11]  I would strongly recommend this article for more insight in to this https://opensource.com/article/17/8/testing-production
[12] https://honeycomb.io
[13] https://unomaly.com

Tom Whiston

Strategic & Agile Consultant @ Nine
Find me on Github