Does anyone care about your non-functional tests? (part 1)
Let’s start this story with a few preconditions.
You have a development team working on a public-facing application.
You have carefully selected the technology stack that would best fit the client’s needs and team skills.
Your team took care of writing all kinds of functional tests to ensure bugs and regressions are down to a minimum.
You have automated CI/CD pipelines that ensure tests are being run regularly, before merging new changes, every night for longer running tests, etc.
This way you also don’t have to worry about inconsistencies when deploying to production since the process is automated and there is no room for human-errors anymore.
You have also taken care of the previously well-known ping-pong issue of the “it works on my machine” excuse by leveraging the power of virtualization (Docker containers or similar). This way you ensure whatever setup is on production, developers have (as close as possible to) the same software, versions, etc.
You have also thought about the underlying infrastructure. Vertical-only scaling is old-school, so you have prepared your application to be able to run on multiple instances and potentially scale in/out when needed. This way you ensure that you have a high-availability setup and if one instance go down temporarily the others can cover for it.
You have also integrated some kind of zero-time deployment strategy like blue/green deployments and can potentially deploy at any point without causing your client any drama. If for some reason you do deploy something broken that slips through the tests, you can easily revert to a previous stable version of the app.
If you have ticked most of the boxes so far, you have already done a pretty good job, congrats!
Is there something else you can take care of to make your product more reliable and the user experience better for the end-user? Of course! And it is not a single thing, there must be a dozen more strategies and best practices that can be applied.
Today I would like to focus on something in particular – what about the non-functional kind of tests? Do they bring any meaningful value?
For example, performance testing, is it any good?
Quite a lot of teams don’t even reach the point of caring about it, and they probably do just fine, so it must be something optional that you probably don’t need to worry about, right?
Remember how I started this post and the “public-facing application” part?
Well, if your user base stays relatively small and there aren’t any huge spikes in demand, then yes, you are probably right to skip on these tests. But do you want to work on a project that won’t succeed and just accept that in the beginning?
“Our project is a really bad idea and I am pretty sure no one will use it, we don’t need to do non-functional tests at all!” – no doubt that is pretty motivating and surely everyone would fight hard in order to get a spot working on that project…
No need to remind you that in the end we are still human and finding a purpose in what we do is probably the single most important thing for us.
For argument’s sake, let’s imagine that this project actually makes it. People start browsing/downloading it in masses and the spikes start to occur on a daily basis.
What then?
Do you know how much traffic any single server you have can handle?
Would you be comfortable going to sleep when you know nothing about how much your app can handle and the client can wake you up in the middle of the night because his service is down?
But we have automatic scaling, right, why should I care? This is a whole other story that deserves a post on its own, but let’s just say that autoscaling is not magic, when there are huge spikes for short periods of time the provisioning of the newly created servers can take longer than the spike itself.
Metrics, metrics, metrics!
I can’t stress enough that, but without data in today’s world you are only guessing, and usually guessing it wrong.
Ok, that’s enough, you would say, I don’t fully trust you, but give me something more concrete of how this can be done without spending most of my work time in preparation for something that might not happen?
In the second part of this article I would try to show you how we did it, what do we measure and how is this helping us.
Btw, the beginning of the story is legit, we do all these things here and more.
