Making the world a better place – Marc Cluet – Build Quality In

Making the world a better place - Marc Cluet

Marc on Twitter: @lynxman - Marc’s blog: devroot.org

Various companies, from startups to large enterprises

Timeline: 2009 to 2014

Context

One of my main purposes at work during the last five years has been to spread and help companies embrace a DevOps culture. Often this was as a consultant in smaller companies where the culture was already built and prepared for disruption (which makes things easier), but most recently I was responsible for introducing and nurturing a DevOps culture within larger, more complex organisations (banks, media giants, big software companies) which was a whole new challenge. I will describe the different challenges faced by companies of different sizes when they begin to embrace DevOps.

High level DevOps

DevOps is an evolution of our software industry in order to make projects more successful (this is clearly a very broad statement but fortunately there is data to back this up). In the Chaos Manifesto it is affirmed that DevOps projects have a higher level of success and a lower level of challenges, which from my experience is true: DevOps projects embrace the path of least resistance while at the same time having just enough structure surrounding them to make sure that everything is measured and accounted for.

A good friend of mine (with a heavy Project Management background) said that DevOps is all about the PPT, which stands for People, Processes, and Tools. Any good working culture (as DevOps is) should be able to cover all of those with a high degree of success and still embrace the path of least resistance to make things as natural as possible. Any kind of change to any of these three factors will act like a rubber band and add tension to the other two, you try to advance Tools only and that will create tension with Processes and People and so on. This is one of the main mistakes that organisations make when embracing DevOps, as it is widely thought that DevOps is all about the Tools when in reality it is heavily based on Culture (which embraces People and Processes); the Tools are just a means to an end (improvement, better software systems, happier customers) and are the catalyst of those Process and People changes.

Is it all about ‘the Cloud’?

The main driver for many companies to embrace change for DevOps is the evergreen promise of ‘the Cloud’: Converting CapEx (Capital Expenses) to OpEx (Operational Expenses) and only pay for what you use in an utility model pricing. Many organisations have invested heavily in the past in their own server infrastructure and even in their own datacenters, and most of them have discovered that these added huge capital costs are not in line with the primary line of business.

However, having adopted Cloud technologies, most organisations realise with time that so-called Public Cloud is more expensive in the long term, hence the huge price reductions for Public Cloud seen in 2014 by Amazon Web Services (AWS), Google Computing Engine, and Windows Azure (apart from the natural competition to embrace the market). Running cloud server instances 24x7 is usually more costly than renting server infrastructure (the model that Rackspace and other hosting providers have) and it is not a good fit for all the workloads that you might need, as shown by all the tutorials trying to run Hadoop or Cassandra in the cloud (it is a risky business). Public Cloud providers will sell you on-demand computing and help you cut dramatically the cost and time for provisioning new servers, which is extremely useful for peaky traffic sites as news agencies and ecommerce sites during sales. The emerging trend of “Own the base, rent the spike” is becoming more and more popular as organisations grow, as it is an economically savvy way to be able to cope with all your incoming traffic while at the same time keeping your operational costs to the minimum possible. DevOps fits incredibly well with this economic model; being able to embrace the maxims of DevOps will help you be prepared for that kind of infrastructure modularity.

There are some companies such as Etsy that have never invested heavily in Cloud or Virtualisation and run their entire production infrastructure directly on physical hardware, and yet are leaders in DevOps practices. Other companies such as Netflix have very different models of ‘peaky’ traffic, and have found that having everything in Public Cloud is a huge ‘cost buster’. In short, embracing the Cloud should not immediately lead either to renting hardware or discarding your own; the truth lies inbetween.

Introducing DevOps

Culturally, IT embraces change at different speeds and rates; this means that for people that live at the cutting edge of technology like myself, more than 95% of the organisations we meet are living in some version of the past, even the ones that come to all the DevOps meetups and have all the cool stickers on their laptops. Organisations with varying internal structures have different histories and different needs to embrace change, but nonetheless it seems that most organisations follow broadly the same steps to get up to speed with DevOps. I call this the Chain of Causality, which is tightly integrated around the PPT model (People, Processes and Tools).

The Chain of Causality

The Chain of Causality is in effect the order in which most organisations end up changing things to enable and support a DevOps model. There have been some times when I have suggested these steps down the chain in a different order due to the product or current organisation of the company. It is important not only to be able to help organisations to evolve and change for their own benefit but also to undertake that change in the right context: I always made sure about this when analysing how to approach this transformation scenario.

1. Automation

Automation is the first and important step towards a DevOps transformation, and is important for several different reasons:

  • Reproduce the same conditions across your production platform
  • Quickly send change to a group of servers
  • Be able to create ‘very close to production’ development environments quickly

Most of the next steps in the chain depend heavily on automation, which makes this a first natural step and the one with the most immediate ‘bang’.

2. Provisioning

In this context we will consider that provisoning is any action that will create a new computing resource that did not exist previously under our control. Once we have most of our basic automation ‘roles’ it will feel very logical to start creating and destroying instances (or physical servers), especially for development environments; the added flexibility to do so will help accelerate development. One very popular way to do so is to use tools like Vagrant to be able to create working images that can be used in any modern desktop or laptop computer.

3. Deployment

Deploying an application is always a stressful situation, and making deployments quicker and safer has always been something difficult especially with servers that needed to be maintained or upgraded at the same time. Thanks to Automation and Provisioning, Deployment would be the third natural step in this; we can easily automate provisioning in conjunction with Automation tools, and even create new servers with the latest version of our software. I have found that automating deployment is normally the second most time consuming task in a DevOps transformation; on the other hand, this process will also open the doors to better communication between dev and ops teams which is part of the cultural transformation that we will talk about in the last point of the chain.

4. Continuous Integration Testing

When the bridges of trust between dev and ops have started to be created it is time to start integrating the DevOps culture into the development, one of my personal favourite things is to introduce Test Driven Development (TDD) to an organisation in combination with Continuous Integration (CI), as the results show very rapidly. A good CI framework will help not only make developers more productive but also development iterations faster and higher in quality. In my experience this step requires a very close coordination with the development team and a lot of the ‘human touch’ to make certain that things are always clear and agreed by all parties.

5. Continuous Delivery for bug releases

Continuous Delivery and deployment to production of bug releases is one of the key business advantages integrated in DevOps: being able to be responsive to users about mistakes and bugs in the platform is a healthy part of keeping the competitive edge. I always found that organisations are challenged by this the first weeks they do Continuous Delivery as it feels “unnatural” to let go of the deployment steering wheel. Making sure that the organisation feels comfortable about this and doing it in different phases (maybe starting by a piece of code that is not key to the organisation) will make sure that the trust is built in the process.

6. Cultural change

When all the other pieces of the chain are in place Cultural Change is already happening: each piece of the chain will bring with it a small bit of DevOps Culture, methodology, and communication. I always left culture as the last part to consolidate, because the culture needs to be adapted and ‘made to measure’ for every organisation at the beginning. Once an organisation has been running in a DevOps way for several months I always prefer to go back and reevaluate the culture to see where it needs tweaking; as with any methodology and culture associated with it continuous improvement is important to keep things healthy.

Working with startups

In my experience startups are the easiest organisations to help transform, as it is not really a transformation per se, but more of a ‘getting up to speed’ task; many startups will be already doing DevOps in one way or another. The startups that I have met are at most some months in the past, due to the challenge of being operational while changing or upgrading a key piece of technology. Still, it is very important to be able to embrace that change as startups feed on the ability to keep innovating; trying new technologies needs to always be ‘in their DNA’, which might lead to refactoring of chunks of code or services, or being able to pivot their business model. One of the most difficult things with startups is trying to beat the emerging status quo; there are people more used to change than others and that is important to keep in mind. Also, it can be difficult for a startup to dump its business model, even when that’s necessary for survival.

Embracing small businesses

A small business that wants to embrace DevOps is a difficult cookie. There are not two small business that I have found that have the same constraints or are ready to put in the required effort to go all the way. One of the main limitations that I have found is the lack of resource - both human and capital - within small businesses. It does not help that small businesses will only embrace change when they are ‘looking at the abyss’; they will resist changing their ways while they generate revenue until it is slightly too late or the competition is getting very though. This also becomes worse due to the lack of knowledge about new technologies and options. The Engineers that I have seen that break this mould do so in their own personal time. Small business will be very flexible to embrace change once they have set up their mind on it, but they will need a lot of external help, that is where contractors or training can definitely help.

Help enable medium businesses

Medium businesses are very fun to work with, at least in my experience! They are normally two to three years behind the leading organisations as they are not big enough to get everyone up to speed together. Also, the pace at which they can implement organisational change is slightly slower than a small business, so that is one of the tricky parts to address. A medium sized business will transform itself in order to give it the edge against the competition, but will not stay that way for long if it is not able to compete properly. Due to having more resources than small businesses they are able to refactor chunks of their code to embrace the latest and greatest technologies and techniques. The main DevOps blocker in this kind of business is normally that mid-level management do not know how to embrace the new culture, allowing and enabling Engineering teams to try new things and learn.

Accepting the challenge of corporations

Corporations measure their survival capability in risk taking. Risk is one of the things that any big business will analyse and try to avoid. Corporations are therefore by far the most challenging organisations to transform. Being up to speed with new technologies and embracing change is very risk prone so most corporations are set in the past in a safe three to five years to be able to run well tested code with bugs that have been documented and patched; the same applies to the software they generate (unless they are an Engineer driven organisation). On top of that there are also the capital investments already made by the company which have an amortisation time and need to be used up to the time they are expected to (at least in countries such as USA, UK, and India). The corporations that I have interacted with are (in 2014) only just starting to embrace automation as the new futuristic thing; corporations expect this automation to come with commercial support so they can scale their issues quickly and push the risk to the providing company, which means that adding other changes to the mix have been so far difficult to justify. The best method that I have found to work with corporations is to create a ‘testing’ or ‘prototyping’ team which will use new methodologies with my support to be able to fully embrace DevOps; once this has been proven as a success, other small teams are able to adopt the same methods. I have not personally seen any big corporation completely transformed using DevOps, and it is a scenario that I find very difficult to see in the next years as DevOps goes against the risk-adverse way of doing business favoured by most corporations.

Teaching an old dog new tricks

Helping an organisation embrace the DevOps cultural change is no easy feat, and I have been personally burned by trying to do so in organisations that were definitely not ready for it. Before embracing a DevOps cultural transformation you should ask yourself the following questions.

  • Will this improve the competitiveness of the organisation?
  • Will this reduce operational and capital costs?
  • Do we have the right Engineers to do this?
  • Is there backing by all the C-level including the CTO/CIO, the CEO, and the CFO?
  • What parts of my work culture am sacrificing for this change and do I need to compensate for them?

Especially important is the amount of pain and risk that an organisation is ready to take in order to help this change, as this will dictate not only the speed of change but also what kind of liability and risk this produce.

Typical Organisations mapped to the risk/pain quadrant

Some examples of the pain and risk ‘profiles’ for different kinds of organisations are:

  • Financial organisations (very risk averse, low pain threshold)
  • Media organisations (risk prone but low pain threshold)
  • Marketing organisations (risk prone and high pain threshold)
  • Software organisations (risk averse, high pain threshold)

Any transformation will only be successful if the amount of pain generated by it is less of what an organisation can tolerate (as any transformation will have pain), the measurements in the previous graph should be a guidance towards that. The amount of risk an organisation will embrace will help to speed up the transformation process while the pain inflicted is tolerable.

People management

One of the most difficult things that any cultural transformation will face is the current status quo and people’s fears and worries. People in general are very change averse, but as Engineers we often do not take this into account; this is the main reason why a DevOps transformation might fail (as I discovered several times!). I was exposed to a lot of challenges from different groups due to fears of becoming obsolete, lack of expertise to embrace the new technologies and changing culture. This resistance can even drive a rejection of any effort to change. Unfortunately, this can evetually lead to an organisation going out of business: as W. Edwards Deming said “You Don’t Need to Change. Survival is Optional”. Preparations to face these challenges to DevOps are what occupy most of my time at the beginning of any transformation. In several organisations, I have been bombarded with all kind of complaints such as “this means I have no work anymore” and “my manager does not agree with this change”, which makes it critical to have the backing of the C-level and upper management, who can be used time and again to push change forward. Winning hearts and minds is 50% of the transformation challenge, there is a very good guide that I follow from Jesse Robbins which gives a very good checklist to prepare for this.

In Conclusion

Due to the pressures of the market, organisations will have to ‘transform or die’, as has happened all through our recent history. Who does not remember the demise of Kodak or the UK motor industry, victims of what happens when organisations do not embrace change. Embracing DevOps is the way to be able to be competitive in the future. In 2014 most organisations are just waking up to this reality and trying quickly to get themselves on the DevOps wagon: to all of them I say “Welcome to the game!”.

About the contributor