Dear Data Engineers, Write More Tests

A while ago, I was working with a team when the centralized DevOps team that oversaw the entire technology domain rolled out a new requirement, enforcing mandatory test coverage of 75% for all future releases. At the time, I was working on a new framework for migrating some of the data products and got caught out by this update since I was not following a typical test-driven development (TDD) approach. I had to retrospectively add tests to meet this requirement.

What I found interesting was that the team of data engineers I was working with were unanimously against this new way of working. In fact, I distinctly recall a senior data engineer declaring that we do not and should not be writing tests for data engineering workflows. I was relatively new to the team and, given it was quite early in my career, I did not form a strong opinion on the topic and decided to trust my seniors. At the same time, I did not stay on the team long enough to realize the pros and cons of high test coverage.

As time went by, I moved on to a new project with a new team that did not have any rules around testing in place. In this new team, I was given the opportunity to work on more complex pieces of work. As the complexity of the system increased, I started to notice a pattern: despite being able to meet the initial deliverables, maintenance and extensions were becoming a challenge. What I soon realized was that if I modified something in one part of the system, I would break something elsewhere, and it likely wouldn't get picked up until much later; sometimes only in production. This isn't surprising, as it's humanly impossible for one person to keep track of everything in a system, even if they designed and built it all themselves.

Over time, the inability to confidently make changes to parts of the system meant I had to sacrifice speed and quality of development, leading to a poor development experience. It was frustrating!

I was sure this was not an isolated problem and was likely solved elsewhere by someone. I started to look into a related domain; software engineering where engineers manage and collaborate on bigger and more complex systems. I wanted to understand how it was done at scale and see if I could bring some of those principles into data engineering. To my surprise, I was again introduced to this idea of TDD, which seemed to be an industry standard for building highly reliable and maintainable systems. I didn't fully understand how it would benefit me, but I decided to start introducing those principles gradually into my own work.

For those who might be new to TDD, it is the practice of writing tests for your feature first, followed by translating the product or feature requirements into code that meets the test conditions.

At first, TDD was not a natural style of development for me, and I didn't notice the benefits of this new way of working until the project got bigger. As the size of the projects grew, so did the test suite that covered most parts of the codebase. The immediate benefit I noticed with this approach was that I could confidently make changes to different parts of the system, knowing that my test cases would catch any surprises and/or edge cases that I might have introduced as a result of the new change.

The second benefit I noticed was that the tests became living documentation for a particular feature. In many teams, it's not uncommon to see engineers not having dedicated time to document everything they've worked on. The end result is that you have a handful of folks who know how it's all stitched together, but new team members may not understand it or might take a long time to get up to speed. On the other hand, if you have good test coverage and assuming the tests were written in a manner that includes comments, doc-strings, and semantically meaningful function definitions, then they can double as documentation. This is not to say that we should stop documentation in general and only write tests going forward, but it's a nice side effect of TDD.

The third and final benefit from TDD is somewhat more relevant in the new agent-enabled coding era. With the increasing use of AI, we're going to see a significant uptick in development speed, which by default means writing tests will become non-negotiable if the goal is to build safe, reliable, and maintainable systems. As for crafting tests, you could leverage AI to write some basic testing, but more nuanced testing of edge cases that are specific to the business should fall on the shoulders of the user. My current opinion and recommendation (subject to change in the future) is that tests are sacred and as such, all tests should be completely written and maintained by users. In other words, don't let AI write tests if you want to sleep peacefully at night.

Once you start writing tests, you also want to start automating those tests as part of your deployment pipeline so that you have guardrails in place to prevent any corrupted work from reaching production. We won't go into the details of how that might look in this post, but in a nutshell, you want to ensure that automated testing is part of your pull request and code review process.

Another perspective on this topic is directed at the delivery and product teams. From their standpoint, it may seem that testing as something that is a responsibility of the developer. However, if you are looking to empower the team to be more productive, it is important to include test coverage requirements as part of your acceptance criteria. In the end, completing a story or feature with appropriate test coverage means the team can safely release and/or make updates to in the future with minimal friction and/or downtime.

Data engineering is an emerging and fast-evolving discipline. As such, it's not really anyone's fault for not having standards when it comes to development. However, we are witnessing a convergence of data engineering and software engineering practices, allowing data engineers to build systems that are more reliable, scalable, and maintainable. When it comes to testing, it's true that testing is not commonly practiced in data engineering, but if you're looking for an approach that might mean less time fighting fires and more time actually building high-value products, my recommendation is to write more tests...