JLL Technologies

It is common for leadership to look across the technology teams and wonder why it still takes so long to make updates to software. I have read about the wonders of Netflix, Facebook and Amazon making hundreds of updates to production a day. When I compared that to our own 2-week cycles, I felt a sense of dread that we were so far off the curve. We have solved some of the foundational challenges – we have automated build jobs integrating new code into lower environments and have a tradition of blameless postmortems. We have chased after whatever tool was the hottest on the market and bought it for more figures than we would like to admit but keeping up with the FAANGs still seems like a dream. One of the key problems is one of scale. If you have 300 engineers across 50 product teams, adding in 5 DevOps engineers is not going to change the course. Netflix is interesting since they have only a handful of Site Reliability Engineers (DevOps specialists), and they have spent years building their internal practices so that the discipline of DevOps is practiced across all the roles and teams, not isolated into a job description. This is the key that makes the equation balance out.

"Integration Tests Are Built Into The Pipeline And Critical Smoke Tests Are Automated And Built Into Jobs That Can Be Shared Across Teams"

 In order to scale up we need to be like Netflix and build our own internal practices. The best place to start is to look for the time bottlenecks in the process– Quality Assurance. QA, like SysAdmins of old, got a reputation as grumpy gatekeepers who like things just so. In the bygone days of waterfall, they held the keys to the kingdom and were often seen as roadblocks or fools who missed the obvious bug. QA had to get everything right all the time and make sure that the project was not delayed. Yes, the scars do run deep.

DevOps happened when Operations was de-stigmatized and got a seat at the table. Transforming the QA organization is one of the quickest shortcuts to many of the benefits that DevOps has gotten the credit for. What is the fastest way to get code from development to production? Make sure that when it is code complete it is already validated for production. When I bring up this concept, I get a lot of frustrated looks from both QA and developers. “Of course, we would love to have QA validate our code at the same time as we commit it but that’s impossible.” “Of course, we would love to be able to test developer’s code as they commit it, but we have to run so many tests that there is always going to be a significant delay.” I have 2 answers to these two statements.

QA is not responsible for running tests. Second – Stop running the same test on the same code. This might take a second to land, but it is vital. QA is not responsible for running tests, QA’s True North is to build a system that verifies the code. Once this is the focus other shifts come quickly. Unit tests are the foundation and builds are set to fail if unit tests don’t pass or if there is insufficient coverage. Integration tests are built into the pipeline and critical smoke tests are automated and built into jobs that can be shared across teams. As each of these changes happen the team is alerted to problems in the application earlier in the development cycle. Most importantly, no human is running the tests or interpreting the results. Testing is no longer reactive; it is all built into the pipeline and is an integral part of the coding process. With this in place the second principle becomes more clear - running the same test on the same code is an antipattern.

Since we have automated the deployment pipeline (if not yet the process) we aren’t compiling the code to go to UAT and Production – we are promoting the build artifact to the upper environment. The training wheels are left in the lower environments where a few manual tests are still needed. After those tests are done deployments to upper environments are fully automated. Configuration files are automatically rebuilt using the proper environmental variables. Secrets such as passwords are securely modified by automation. We do need to run a critical subset of tests – Smoke Tests – to verify the new environment doesn’t break our code, but broader regression testing at this point is wasteful, it adds time to the release and toil for the team. If it is needed, then the problem is in the build and deploy pipeline or the environmental strategy and needs to be fixed there.

In the end the keys to QA being successful are the same keys that unlock the potential of DevOps. Automation needs to be built into the system as close to where changes happen as possible. Feedback loops should be short and the responsibility for maintaining the loop falls on the person it assists. QA is an owner of the big picture of how the system fits together and works with the team to continuously improve it from both a speed and accuracy standpoint. With each of these pieces in place the QA team starts to look a lot like a DevOps team. Their work has scaled both the culture and the reach of the few DevOps specialists in your company. Cultural change is never easy but finding where the principles align across roles brings the teams together in their quest for improvement.