A team working well is one where the developers not only build the code, but also the automation for deploying the software and the ability to monitor and support it. Done right this means the team is doing DevOps and Continuous Delivery. Right from day one they are building out Continuous Integration, coding the deployment mechanisms and setting up dashboards and alerts for the deployed code. When they showcase code, it is code that has been deployed to a realistic environment, so if required that code really could be deployed.
Even before go-live the team is pushing code to various environments. Often going live is about opening up firewalls and configuring DNS, the code is done. The team’s test suite includes checks that code is live and available in different environments, often using a subset of the acceptance tests. Logging and alerting is in place, and has been evolved over time as new features are added. Requirements include acceptance criteria for both business users and for those who will monitor the system.
DevOps and Continuous Delivery done together are two practices that reinforce each other, but if done on their own can create pain for your teams and increase the cost of your software. We’re going to help you identify that pain, so you know how to apply the fix.
Heuristic: “Over the wall” support
The predominant model in software delivery continues to be for a project team to throw software “over the wall” to a support group. The pain this causes have been well documented, lowlights include:
- The support group is treated as 2nd class citizens; maintenance is often seen as less desirable career-wise.
- Must-have operational features such as logging and alerting are deliberately ignored to save project budget. They end up being added later by the support team.
- Software in “support-mode” decays over time, eventually necessitating replacement.
- Project developers rarely learn how to improve their software as they don’t have to deal with the consequences of their decisions.
- A lot of cost and complexity is introduced through organisational hand off from a project team to the support team. It often involves hefty paperwork and signing-off documents.
These issues sit against a background of an increasing pace of delivery and innovation in our industry. This is leaving many companies behind who are unable to compete in the face of organisations that can release in weeks what once took months to get out the door.
“DevOps as a role does not make it a practice.”
Heuristic: Ops 2.0: DevOps
The first key issue to overcome is a fundamental and unfortunately increasingly common mis-understanding of DevOps: Ops people are just being re-labeled as DevOps. This misses the point entirely, if one of these relabelled Ops people is on the team this does not mean you are doing DevOps. A role does not a practice or discipline make!
DevOps is a practice the whole team does. Unfortunately, many teams just have an Ops person to set up CI, do some deployment scripts, and maybe turn up just before the first release. This is no better than the Build Guy that has to do everyone else’s merges!
Heuristic: Stop waiting for the specialist
The lack of on-team knowledge means things like new integration points or configuration of new environments get deferred until the Ops person is available to help the team. This can mean there are large periods where the team could not “push” the release button and are often doing showcases in unrealistic environments. In turn this means stories can get signed off as done when they are effectively saying they are “done on dev” or “done in test”. In CD done means released into a prod-like environment so a false indication of progress is given by ignoring the fact a requirement is not really delivered until it is deployable to production.
It is good to have an Ops specialist to help the team, but it’s important to have the whole team do the actual DevOps work and share the knowledge.
Heuristic: Where is the value in that?
A further trap is that many business sponsors have been trained to defer ops requirements. They are used to leaving these until the end or not having to prioritise them at all. There can be pressure during delivery to just focus on the requirements that add business value visibly. Although of course if you can’t deploy the software you are not going to get any value at all! So you need to treat your operational requirements as a first class citizen.
“You are only going to get maximum value out of DevOps if you do CD and visa versa.”
Heuristic: No Monitoring
Many teams who are starting to adopt CD have no way to monitor or get alerted about issues after deployment. You can’t do continuous anything without feedback so you should be adding logging in from day one, placing application stats and alerts on an information radiator. Of course on day one the log file might just say “deployment: version #” and the amount of logging needed will evolve over time. The trap to avoid is to go-live and then find out there is no monitoring in production, you might be doing CD, but you are running a large risk of an outage or even not knowing there is one until users complain.
Heuristic: Big Bang Continuous Delivery
Another anti-pattern is the team who plan to have CD, but only after the first release. So they spend several months doing manual or unrealistic deployments always planning to do “CD” at the end, perhaps by adding in some “DevOps” resource during the last week or so.
This doesn’t work as there is no way to understand the impact of technical decisions on deployment and monitoring until the very end. It also means the first release might be the first time the automation gets tested, this is no better than a big bang release.
The deployability of software is not something that just happens. You need to design for it and test it. When you get there the first actual production release should be a non-event. The question of deployability and testability applies equally to Off-The-Shelf software, so don’t forget this when gathering your requirements.
⁂
All of these situations can be avoided by having teams doing Continuous Delivery and DevOps as cross-team disciplines. By having developers more exposed the issues of deployment and production, they get to learn how their decisions impact an organisation downstream. DevOps and CD have value on their own, but are at their best when done together.