Contributor Q & A – Build Quality In

Contributor Q & A

How did you get involved in Continuous Delivery / DevOps?

Chris O’Dell: When I joined 7digital in 2010 the development team there had decided that Continuous Delivery was something they wished to achieve. They’d already made use of a CI server and automated tests but were being held back by the monolithic application. They described the ideal Continuous Delivery environment they planned to achieve, and over the next few years I helped build it with them.

Niek Bartholomeus: I was part of a team that was responsible for the development tools and practices within a large investment bank. at one point we realised that the biggest bottleneck in delivering the software into production was the lack of understanding between the development teams and the operations teams. Until then my activities has remained mostly technical, but this issue really had to be solved on the human level, so a whole different approach was needed.

Rob Lambert and Lyndsay Prewer: We got involved with Continuous Delivery because of a natural evolution of our release process. We needed to get features out to our customers quickly so we had to evolve the release process to facilitate this. It was no good being able to create software rapidly if there was no way to release it to our live environments so we started to optimise our entire build and delivery mechanism.

Phil Wills and Simon Hildrew: Etsy’s discussion of how they push to production multiple times a day was one important influence, but a more surprising influence was from many years ago, when we deployed by copying and pasting from one IDE window to another. Whilst the actual process was a mistake-riddled minefield, the the ability to deploy quickly, whenever we needed had been valuable.

Alex Wilson: I was thrown in at the deep end when I started at Unruly with similar experiences to Benji (deploying early on my first day) - since Unruly is my first position after university I believed that this was just how everyone does it, which I discovered is very much not the case.

Benji Weber: I saw how things were done when we joined Unruly, and a team was deploying multiple times a day. I only later discovered it was called Continuous Delivery and read the book by Dave Farley & Jez Humble.

Anna Shipman: GDS is the first place I’ve worked where they have a DevOps culture so joining GDS was my first exposure. About eighteen months ago I joined the infrastructure team because I wanted to learn how web operations works up close, and that’s when I really started to understand the value and purpose of DevOps.

Amy Phillips: Continuous Delivery was one of the options we considered to unblock the Songkick release pipeline. After talking to other companies who were successfully using Continuous Delivery, and Continuous Deployment, we decided it might be the right option for us.

James Betteley: Once upon a time in a software team in London, we needed someone to automate our deployments, and everyone else was out at lunch, so I drew the short straw. It turned out to be more interesting than I’d expected, and learning more about infrastructure and the runtime system was actually good fun. “Infrastructure as code” didn’t really exist in those days though, so I’ve seen the DevOps world emerge and change the way I work immeasurably.

John Clapham: Looking back the first time I got involved was as a Java developer building releases at IngentaConnect, there was a small, bright, team there, and the technical roles overlapped considerably, we just did what was necessary to keep the site healthy. I hadn’t experienced alternative approaches and it seemed natural to release when ready, automate a little more each time, and work closely with the people that ran systems. The first time I noticed DevOps as a ‘thing’, or rather the problem it aimed to solve, was tangentially, through continuous delivery. In 2009 I was working with Nokia Entertainment, where operations and development had evolved away from each other, both technically and culturally. They benefited from optimising in their own domains, but at the expense of cross team collaboration. Working as Product Owner of the continuous delivery Team it soon became clear to us that developing a DevOps culture was key to building a continuous delivery capability.

Jennifer Smith: Working for ThoughtWorks, Continous Delivery is something that has become very much a default setting for me.

The DevOps movement is something that I have followed with interest for some time. As someone that is coming very much more from the ‘dev’ rather than the ‘ops’ side I was really interested in understanding more about what goes on underneath the abstractions I work with day to day. Partly through curiosity and wanting to learn new things and partly because I think it’s important to have at least an initial understanding as a developer.

Sriram Narayanan: I started off as a developer, and then spent a few years in Infrastructure. I’ve learned the hard way that I should have invested time in automating various project environments. I learned quite a few lessons and had some great experiences with learning and implement various process and automation improvements. Today, I consult to customers on their CD journey.

Jan-Joost Bouwman: Sometime in 2011 the manager of the IT Service Management department started pushing people to read Continuous Delivery by Jez Humble and David Farley. At first it didn’t seem to have a whole lot of relevance to our ITIL based process oriented service management world. But he kept pushing and pushing. So finally I got the book. Coming from a process management background in service management with little development knowledge I didn’t get all of it, but I soon understood why my boss was so enthusiastic about this book. I could certainly see the areas where we could use the ideas to improve our performance!

At the same time our development colleagues started experimenting with Agile/Scrum. Soon all development teams were using Agile/Scrum methodologies, but the hand overs to production were still a major obstacle. Yes, we made progress with our quality objectives by standardizing non functional requirements, but it was still a lot of work with loads of paperwork.

When by the end of 2012 the organisational transformation to DevOps was announced I started researching it online and attended some seminars. What I heard there made me realise that the collaboration that comes with DevOps was for us the only way to really unlock the potential of Continuous Delivery. On our internal social network I started our own DevOps community to share what I had learned and read so far, encouraging people to share their stories. Since then the community has grown and grown, reaching well beyond the borders of our own department of 150 DevOps teams to the rest of the ING Netherlands departments and abroad. I still work as a process owner, but have transformed our ITIL processes to the needs of the DevOps Agile/Scrum way of working.

Martin Jackson: I first heard about this “DevOps thing” from a podcast series called Cloud Cafe by John Willis and Michael Cote and thought “Wow I want to be a part of this” and managed to attend my first DevOps days in Hamburg soonish after and it was an eye (and mind) opening experience all those people in the same place with the same problems, working though them together.After that I un-officially got into the world of Continuous Delivery when I worked for Clinical Research company as a Build and Release Engineer, I set up my first Jenkins server and learned through a lot of trials and tribulations that the path to Continuous Delivery was a bit like trying to cross a minefield some organisations but if you can get there it was well worth it.

Rachel Laycock: Before working at ThoughtWorks I had heard about CD but wasn’t really doing it. I joined ThoughtWorks around the time the book came out and there was a real buzz about it in the London office. I didn’t really know what it meant so I read the book whilst I was on my first project and immediately became interested in learning more about operations and specifically how to make software operational. This was one of the moments I realised I still had a lot to learn. Those moments still happen pretty often, which is one of the reasons I love my job.

Matthew Skelton: My background is in software development: desktop, mobile, web, and industrial control systems. From around 2006 I became increasingly involved in the build, deployment, and operations of software systems for some large client organisations. It became clear to me that being on Production support is a great way to become a better software developer, and for a while ran a service desk based on ITIL practices. Seeing the effect of addressing (or not addressing) operational concerns as part of software delivery made a big impression on me.

Steve Smith: Between 2008 and 2010 I worked with Dave Farley and a bunch of other smart people at LMAX, where we lovingly crafted a pipeline out of Cruise Control 2.8.2 and Ant 1.7. When I paired with Dave I teased him about the time it was taking him to write his “Continuous Delivery” book, and felt like an utter idiot once it was published.

What do you see as the biggest advantage of Continuous Delivery / DevOps?

Chris: Can I pick two? One technical and one cultural. Technically it’s the ability to deliver a flexible system which can respond to changes in direction. This was particularly relevant at 7digital where the music and digital media industry is fast moving and we needed to be able to deliver quick enough to maintain and exceed the pace of the market. Culturally, it brought the whole company closer together. Trust and teamwork greatly increased and so the quality of the entire platform was massively improved as a result. We had a phrase at 7digital – “Use The Voice”, which reminded people to talk in person instead of via email or tickets. I believe this a cultural artifact of the faster moving Continuous Delivery approach.

Niek: When the development and operations teams have the possibility to learn from each other, to appreciate each others work, they will start seeing the bigger picture of the work they are doing, which is delivering value to the end user. This insight alone will help them make decisions in their own area that are beneficial for the whole organisation, and not only for their team.

Rob and Lyndsay: The biggest advantage of Continuous Delivery is the ability to release small incremental changes rapidly and then get feedback on these changes. This short feedback loop ensures our customers get the right features quickly and with minimal wasted development time. Features being used in a live environment by real customers is the ultimate feedback as to whether you’ve built the right thing in the right way and Continuous Delivery helps us to achieve this.

Phil and Simon: Continuous Delivery allows you to build more useful software in a more collaborative, fun way by shortening the feedback cycle.

Alex: CD and automation go hand-in-hand as I see it, and in order to keep our feedback loops tight and enable early decision making we’ve had to automate our processes aggressively. As an added bonus of this, we’re able to quickly deploy servers and applications from scratch with a reduced amount of pain - a victory in its own right.

Benji: It allows us to get feedback on the value of our work in the shortest possible time. Often it is even possible to measure how much money a change is generating/saving after releasing it. Therefore it minimises the amount of time we spend building the wrong things.

Anna: From an organisation’s perspective, a DevOps culture underpins the most important parts of an agile software development methodology: release early, release often, fail fast, etc, as well as communication between all parts of the business.

Amy: Being able to release features and bug fixes to the user as soon as they are ready. Moving to Continuous Delivery means we have a release pipeline that aids the team rather than controls it.

From a personal perspective, the biggest advantage for me has been the opportunity to get hands-on experience with web operations. Understanding how your code works in production, how it is supported, and what can go wrong, makes you a much better developer.

James: “The biggest advantage of DevOps is that it reduces your time to market by 40%” Seriously, there is actually a report out there which makes this claim. It’s my favourite DevOps made-up fact. In reality, for me the biggest advantage of DevOps is better quality systems. Where a DevOps culture exists, delivery teams are able to design and deliver software which is tailor made for the production environment, which means more reliable, better performing systems.

John: That’s a tough question, because when Continuous Delivery and DevOps are done right there are numerous, sometimes unexpected, benefits. I’ll start with a caveat though, because it’s perfectly possible to get it wrong. For example, investing a disproportionate amount of time or money, or alienating people and reducing engagement with an insensitive approach to change. It is also easy for tool and process choices to have a detrimental impact on quality, reliability and security. The biggest advantage in my mind is that Continuous Delivery is one of those rare initiatives that is good for business and individuals. Done right it accelerates the organisation’s ability to learn, assuming it knows what it’s goals are and can react to feedback. Releasing changes often means more opportunities to try something, check feedback, refine and try again. Damon Edwards put it nicely: ‘You get more shots at the prize’. On the individual side, people working in a Continuous Delivery and DevOps environment are often more engaged, putting responsibly in the right places promotes learning, eases frustration, increases autonomy and encourages personal development.

Jen: I think that the biggest advantage of the DevOps movement/culture is ably demonstrated by the portmanteau itself - bringing together the two perspectives of development and operations together for greater good.

These two groups of people can learn a lot from one another - developers like myself who are exposed to running systems get a really good understanding of the challenges of keeping systems running, available and stable. I can’t speak too much from the other perspective, but I feel like there are developer-lead approaches to automation and readability that are a major benefit.

As I mention in my article, exposing developers to operations and support is one step on the way to helping them support their own systems. This not only gives them great experience and feedback but also frees up the operations folks to do bigger and better things.

Sriram: The ability to apply fixes and add features boldly without the constraints on environments, deployment times, long test times, or worries that additions or changes to the code would introduce new defects or re-introduce bugs.

Jan-Joost: For me the biggest advantage is the shared responsibility. There is immense value in letting developers share responsibility for the stability of your production environment. That combined with the immediate feedback from the business at the end (and during) each sprint have been a real boost for our quality.

Martin: The biggest advantage I see is to get the right thing into customer hands in the fastest possible manner possible by doing just enough design while constantly iterating, experimenting and refining. Also the ability to fail fast and correct course quickly allows us to challenge the way we do things without the fear of getting it wrong paralysing us.

Rachel: To me it’s about thinking about how software will run in production right from the start. Not just is this feature built well and maintainable from a code design perspective, but how does it need to run, what kind of hardware, caching, networking and operating system. I think it’s good for developers to have to learn that early because the “beautiful software” you write doesn’t really amount to much until it’s being used and is under the load of real users.

Matthew: The biggest advantage for organisations that manage to adopt and sustain a Continuous Delivery or DevOps approach for their software systems is that those systems can be built, tested, deployed, and operated much more effectively than before. The systems can be made more relevant to the organisation’s requirements by being easier to change and improve on an ongoing basis, providing better ‘return on investment’.

Steve: Releasing smaller changesets more frequently increases revenues and reduces risk. What’s not to like?

What do you see as the biggest challenge in Continuous Delivery / DevOps?

Chris: Trust. Trust in your tests, trust in your deployment scripts, trust in the production environment. Trust in the Quality Assurance person, trust in the Product Manager, trust in Sales and Marketing. These things are hard to earn and come with time, repeated proof of robustness, openness and honesty.

Niek: In many traditional enterprises there is still an old-school mindset that software development is not a strategic advantage and therefore cost-efficiency is considered more important than delivering business value. In these enterprises the teams in the IT department are typically structured by technology, effectively creating specialist silo-teams with little opportunity for mutual collaboration, in an attempt to increase efficiency. Unfortunately, these types of organisation structure simply don’t work in a domain like software development that is complex and creative. Instead, what is need is a better collaboration and alignment of the objectives between the teams, or, in other words, DevOps.

As this old-school mindset gets stronger as we go up in the hierarchy of the organigram, the easiest strategy on first sight to introduce DevOps seems to be a bottom-up approach where the people on the floor initiate the effort and then let it organically “infect” the rest of the organisation. However, as the old and new mindsets are so fundamentally different it may not be possible to make the switch by gradually evolving from one to the other. Instead we may need something more disruptive, where the pressure to change not only comes from the bottom, but also from the top, and from everywhere else.

In the end, my guess is that only external forces like increasing competition, shrinking market shares and eventually the struggle to survive can set the stage for building up the necessary pressure to change.

Rob and Lyndsay: The biggest challenge in Continuous Delivery is in creating a safe, predictable, efficient and careful way to deploy software in to live with minimal impact on our customer’s service. Releasing often means frequent disruptions to your customers if the Continuous Delivery process is not effective so controlling a frequent release process and then optimising it can often be the first challenge when moving to Continuous Delivery.

Phil and Simon: Fear. There’s always the temptation to add one more step into the process for getting software to production when something goes wrong. Sometimes that’s the right thing to do, but it’s a real cost that should only be paid as a last resort.

Alex: As with any methodology, it can be hard to get buy-in from other developers when the benefits aren’t immediately obvious and it’s difficult to understand the value of doing CD without having tried it first-hand. It’s also easy to fall into the “tooling trap” where you focus too much on tooling and not enough on actual practices - you can easily implement a decent CD workflow with nothing more than a set of shell scripts.

Benji: Like most things in software development, the biggest challenge is communication. Helping new team members understand why we do CD, and how to do it effectively, and maintaining awareness in the team about the reasons for what we do. There are constant forces pulling us away from CD. We add new tests which slow down deploy cycles. We are influenced by hype and “best practices” from the rest of the software development community which don’t play well with CD.

Anna: Making sure the concept is understood. As with Agile before it, misunderstandings of DevOps are rife, and also like Agile, people are trying to adopt it, or say they are adopting it, without being clear about the concepts behind it and what the costs as well as the benefits are. The biggest challenge is communicating what DevOps actually means and what is required from the business to support it.

Amy: Creating and maintaining the culture needed to prevent low quality releases. Everyone involved in the delivery needs to understand what you are trying to achieve and needs to allow time for code (and test) design and maintenance.

James: I think it’s the whole “cultural change” thing. Most people now “get” that DevOps is about culture (as opposed to being just about automation), but what we still struggle to understand is exactly how to change a culture effectively. Depending on your existing culture, changing to DevOps can be a radical departure, and people just don’t appreciate how hard that can be.

John: People; their history and habits. In this area tools have evolved rapidly, there are some amazing projects out there, and smart people to use and integrate them. Ultimately though it’s people that start the conversations that lead to technical changes. Despite what appear to be obvious benefits it can take an age for initiatives in Continuous Delivery and DevOps to gain traction. Often it is not a matter of a couple of people experimenting, the change spans departments, challenges existing roles, and the principles and conventions that built careers. There is a wealth of anecdotal and data evidence to support the approach. Despite this some organisations, and people, are still reluctant to take the first steps, the challenge is finding ways to encourage them…

Jen: I think that one of the biggest challenges is around the complexity of the tools, stacks and concepts in the operations space. People who have been doing it a while can start to lose the appreciation of this complexity (maybe a curse of knowledge effect http://en.wikipedia.org/wiki/Curse_of_knowledge).

Although some things just are plain complicated, we can work a lot on addressing this via tools, visualisation and just taking a different perspective on the issue.

Sriram: Lack of executive buy in backed by strong action can be the biggest challenge. You may have the best of technology, developers, intent and automation, showcase all manner of ideas for improvement. But if there isn’t executive level buy in backed by concrete action, then the status quo will not change and any initiative can fall flat on its face.

Jan-Joost: For me the biggest advantage is the shared responsibility. There is immense value in letting developers share responsibility for the stability of your production environment. That combined with the immediate feedback from the business at the end (and during) each sprint have been a real boost for our quality.

Martin: I see culture as on of the biggest enablers and obstacles to Continuous Delivery and DevOps without a bottom up culture supported from the top down many Continuous Delivery and DevOps initiatives will literally die on the vine without delivering any benefits and literally poison the well for future initiatives.

Rachel: That’s easy, your organisational structure as that informs your design, which in turn informs how it will run.

Matthew: The biggest challenge for organisations wanting to adopt Continuous Delivery and DevOps – a challenge I see time and again with many different client organisations – is that we as IT folk have so far done a very poor job at explaining to non-techies (and other techies) that building and operating modern, complex software systems is a high-skill, value-add activity, not simply the ‘execute-only’ activity that many people seem to think it is. We need to become better at characterising the patterns of teams, technology choices, and funding models that will help to produce more effective systems without ever-increasing costs due to the accretion of functionality via a naive ‘new features first’ approach to software evolution.

Steve: Changing organisations to be better aligned with product development flow. Automating a release script is child’s play compared to convincing people they should actually talk to each other.