Scaling DevOps for the Enterprise

A decade ago, an enterprise adopting DevOps was a rare event. In the last decade, some businesses dipped their toes in, others spent big money going all in. Now the question isn’t whether to make the culture shift, but how to be successful as it grows. It’s how to scale the adopted DevOps practices in the enterprise. This article is going to focus on highlighting common missteps and offer advice that can help you make decisions to handle the challenges ahead. 

The Culture Bomb

Why culture? Because DevOps is more of a cultural shift than a technology adoption. Enterprises don’t have the same flexibility and freedoms as startups, and culture spreads much slower in a large organization. Properly motivating teams is a clear consideration in an enterprise because, unless you’re getting to work with cool technology, or on completely green-field projects, there are limits to innovative freedoms, or escaping the regiment of meetings and the red tape to get anything done. If the culture isn’t cohesive, fun, interesting, and engaging, your DevOps talent will likely look for other opportunities (and leave you with knowledge gaps your legacy team can’t manage).

Focusing on the building that collaborative, innovative culture in your DevOps teams is great. It’s time to spread the culture to that  “legacy” workforce. If you want adoption and scale, it’s time to start revamping the old processes and start shifting the culture across the entire company. People will resist at first, but when they see the company’s focus turn from “block checking” and meetings to focusing on quality and automating processes, they will understand the adoption. They will be part of improving quality internally and for customers and not just maintaining status quo. Don’t let the prior guard just run its course, or you may risk an “us-versus-them” scenario. Also, following two sets of politics will not scale.

Standardize? Or Use Whatever Works

This is a challenging question but it directly impacts scale of budget and resources. Here are a few challenge examples:

• Most enterprises buy support contracts with vendors, but paying for redundant or cross-boundary solutions is wasteful. This can easily happen when each team picks its tools.
• Building integration solutions to maturity is a large investment, think monitoring/logging, or services like Salesforce.
• Going all in on a single platform per solution means teams can focus and share efforts on adoption. Also, it’s more likely to get volume discounts from vendors if buying for the enterprise rather than a section of the business. But, in doing so you are investing in vendor lock that will likely last for many years to justify the investment.

These were obvious problems, so what are the solutions? Here are a few guidelines to help decision-making when deciding on what is standard and what is dealer’s choice.

• Outsource where speed to innovation outweighs full ownership. The obvious example is building on cloud services (AWS, Azure, etc.).
• Offloading heavy management of logging, monitoring, and reporting. Examples are ELK services or Splunk.
• Building proof of concepts (POCs) or green-field projects should not be restricted, it’s the time to spend money on evaluating solutions and trying new things. After the POC, it’s time to decide on making changes or integrating POCs into existing services.
• Evaluate vendors carefully. Most have open-source services, financially justify the pros before locking in (as an enterprise). Sometimes working with the open-source tool enough will be telling on whether to even invest further into a product or make a switch. Once the purchase agreement is signed, that door is closed.

Actually Do Agile

If you want to scale, if you want to build faster, reduce burnout, increase quality, then do agile the right way. If you are trying to jam your waterfall release cycle into the agile process, expect none of the above. There is nothing more demotivating to a team than struggling to hit an “expected” velocity each sprint and then cramming in a ton of work before quarterly release, just to start the cycle again. It’s not sustainable and it will generate a ton of tech debt. Don’t do it.

Also, every team isn’t a scrum team. There is a reason some teams adopt kanban easier, because their workflow is more assembly line (such as a help desk). There are other divergences, such as scrumban that may work best. The point is, each team should be empowered by leadership to keep these principles at the forefront:

• Don’t strive for more than 70% utilization each sprint. Keep your teams busy but leave them time to sharpen the saw, or socialize ideas, and time for the required meetings.
• Failing sprints highlight learning opportunities, not failure. The sprint cycle is a living process and the team should continue to learn from one another. It shouldn’t be a two-week struggle.
• The team is formed to deliver quality in an iterable fashion. Leadership is there to serve them and remove barriers. More on this later.

Double Down on Testing

Here is the justification often used against allocating enough time for test building: it takes a lot of time and we have deadlines. It’s developer stuff that should just be created if they have time. Or it can be added later. The infamous “later” that never arrives. Every single time the decision is made to put off an important step until “later,” a tech debt deposit is made. And the interest always accrues.

That being said: every team should be building tests for their supported infrastructure and applications.

Automated testing is its own success and reward. DevOps teams who embrace test-driven development are building for scale. They are investing time into eliminating expensive regression testing. They are wanting to deploy with confidence. They are wanting to increase quality in software deployed. They are reducing the overhead to manage software. They are reducing capital overhead of positions held by manual box-checkers. Good automated testing is the difference between being the enterprise of yesteryear, and building CI/CD for the future.

Budget to Replace Platforms That Don’t Scale

No enterprise executive board wants to hear it will cost 10M to replace their original mainframes backend or five years to redesign an aging .NET platform. It’s an unpopular effort and often vetoed in favor of spending on new innovations and projects.

The fact remains, that old systems don’t replace themselves and the sooner they are built into modern frameworks, (stateless, containerized, serverless) platforms, the sooner they will enter the CI/CD pipelines and become a scalable technology. Older technologies require administrators, developers, release engineers, contractors, often entire support ecosystems to update and keep alive. This is a direct cost in labor and time and it can keep companies in the hybrid-cloud models or resorting to managing their own data centers. There is a tremendous amount of hidden cost that infects budgets to keep aging systems alive for the next 10–20 years.

The Top Must Learn to Serve

Most enterprises are hierarchical. From the president all the way to interns, there is a top-down approach to getting things done. This is a proven effective model to manage enterprises, but it too must shift in the adoption of DevOps. Executives must be able to change expectations with board members and company leaders to adopt a quality-first model. They must become servant leaders and understand that it is on them to knock down the largest barriers stifling innovation at the bottom.

Effective agile teams often have a strong scrummaster and a PM/PO who sets reasonable expectations up and down. That chain of communication and expectation settings has to go all the way to the top, or forget it. This also means changing the culture to include more modern workplace experiences such as:

• Things like bringing pets to work, free beer, recreation rooms: all large distractions and meaningful moral boosters. They will be used in moderation because adults will act like adults when they are empowered to blow off steam when they need to.
• Promoting hack-a-thons at the office. This can include opening the events to outsiders to help recruiting efforts and showing other business in the space that you’re investing in networking and tech.
• Admitting fault. It is too often management takes credit for wins, and blames staff for loses. Being genuine to your staff will retain talent and that reputation will spread. Talent turnover is a major hidden cost enterprises deal with every day. Especially in the DevOps space where engineers are in extremely high demand.
• Find the solution to enable your teams. If you are breaking down their barriers, they will hit your deadlines.
• Remind the executive board that it is okay to fail. Failure allows you to learn and do better. Everything comes down to money, but if there are always finding fault after failures, then forget scaling your culture.

Summary

It is a real challenge to scale DevOps in the enterprise. It takes leadership to make difficult choices and adapt with their teams. It means taking the time to do things right now the right way, so that it pays off later. It is about making choices on tool adoption or allowing tool sprawl. It’s about breaking down barriers between dev, operations, change management and other silos to be part of one process. It’s about learning from failure last night and building the appropriate test to monitor and eliminate that failure for next time. Scaling DevOps for the enterprise is about coming together to automate away tomorrow’s problems, today.