This is the second installment of our three-part DevOps series and deals with the methods and tools involved in the process of moving from development and test to deployment and delivery.
If you haven’t done so already, you can read the first post here – or check out the final part here.
Develop & Test
In our core development process we use scrum and Agile in a fairly standard 2 week cycle so nothing special there. Beyond that we use the principles of continuous integration for developing and testing our software. This means that developers integrate their code into the main code repository as often as possible, reducing the risk of conflicts and failures which can take a lot of time and effort to fix.
Once developers have committed their local code, they can take a new copy of the central repository before beginning their next task. So, continuous integration ensures that the code on the developers’ machines and the central code repository are never allowed to stray far apart.
As part of integrating their code, our developers write tests and verify their code through automated builds. Automating this process has allowed us to discard the error-prone and tedious manual processes. We use TeamCity to run tests and provide developers with quick feedback on whether their commit contains any errors or conflicts. Developers must fix any issues straight away, so that only working code is committed. Our TeamCity tests are complemented by more thorough nightly builds that include functional Selenium WebDriver tests and deployment simulations thus ensuring even better coverage.
The constant feedback loop from developing to testing means that our developers are immediately updated on the quality, functionality and system-wide impact of the code they’re writing. They are also forced to break down tasks into smaller components in order to integrate as often as required, meaning that their code becomes less complex and easier to work with. For our testers, the continuous integration and automated builds mean that they always have access to an updated build to do their testing on.
On top of that we try to enable feature-flipping on most major features. This allow us to do two things: Gradually enable the feature for a subset of our customers, and in case the feature introduces problems we can easily turn it off in production.
These are the core elements necessary to implement a continuous integration model:
- A version control system
- A solid base of automated tests
- A scripted build
- A Continuous Integration system like Teamcity, CruiseControl etc.
The DevOps process at e-conomic
Deploy & Deliver
We have made a big effort to automate our deployment process as much as possible. This way, we get fewer deployment errors and we become more flexible because more developers can manage the deployment and it involves less effort for them. This also means less tedious work for the developers, which can be replaced with other, more interesting tasks.
Even with automated deployments, the deployment process can be stressful if preparations haven’t been made to account for deployment failure. That’s why we have precautions in place to permit failure, and to fail fast so we can recover gracefully and not affect the system uptime or performance. We have backup snapshots in place allowing us to enable rollback if we can see that a deployment contains issues that can’t be fixed immediately. With the appropriate rollback mechanisms in place it is less risky to deploy and although we still test our code the same way we do not hesitate so much when pushing new features.
In terms of deployment frequency, we have managed to shorten our release cycle so the code from all developer teams is released every 2 weeks. This makes it easier to debug any issues since the deployments are relatively small and makes the deployment itself easier to handle. Also, the time from a feature is developed until it’s deployed into production is reduced, meaning that we’re able to deliver the features our customers want much faster.
And we are not stopping there. We’d like to shorten this time span even further by moving towards a form of continuous deployment where features are deployed as soon as they are completed and tested, rather than as part of the standard 2-week release cycle. We will see how far we can move towards this goal without running into practical obstacles.