Tip 5: Give Your Assertions A Looser Touch

Another principle I consciously use in reducing the brittleness of my tests, is to have a light touch, in the sense of minimally coupling with the underlying code.

Some examples will probably make this easiest to follow.

So here I have a test that allows a freelancer I hired to add a law case to my website. You can see what it looks like over here.

And I wanna show you this assertion here.

So at the final stage of the test, I check that the page title - that’s the HTML title - matches the law case_name. What am I using as the case_name here?

R. v Stephenson in the test, that won't be the same for the actual website.

So let me show you the website one more time.

Here you can see that this case is Pennington v Waine.

What is my title tag? It is "Pennington v Waine [2003] Conv 192".

This last bit is a legal citation, so therefore my title differs from the name of the case.

However, my test still passes. I'll prove that to you here.

That's because I am using a regex match on the title, instead of an exact match.

An exact match would be overly tightly coupled, in that any changes I make to the title for SEO or branding purposes are gonna break my test.

I just want to ensure that the case name is there, somewhere. That is enough.

A corollary to this is when you're testing emails sent by your system.

Whatever you do, don't regex against an entire sentence worth of words eg. Against something like ”click this link to unsubscribe”.

Instead, do a simple match against a single word, a single link: ”Unsubscribe”.

The point I'm getting at here is that the exact text, the email, or on the page is sort of immaterial. It's not worth testing against.

The big picture is that you want to test every email gets sent, can be rendered, and contains a link to unsubscribe. That is what we test against. That is what we couple against at that level of granularity.

Lastly, I try not to ever assert against full URLs.

By that, I mean "http://www.localhost:3000/...", What I instead do is just assert against the relative path.

The protocol and the host and the port, all that sort of stuff is immaterial to most tests, and they just create extra reasons for the test to break, whenever these things change for reasons that are totally orthogonal to your tests.

Why introduce chances for your test to break?

Tip 6: Abstract Away Your Config To Reduce Brittleness

Software tends to have some key variables that change over its lifetime.

Some of the ones in Oxbridge Notes can be seen in MyConfig file here.

Let me scroll down a bit.

Here we have, for example, a couple of fixed email addresses. For example, mails come from server@oxbridg.....co.uk, my admin email is currently info@oxbri.....co.uk.

So this is the cut that PayPal takes, I track this for my profitability calculation.

Similarly, I also track the sales_tax_percent_cut.

Scroll down a bit more, we see the stores I am active in, that changes over time.

Another interesting one are the discount amounts.

So whenever someone upgrades a product they get a discount of 66%. That has changed over the lifetime of my business.

Now imagine a situation where I had hardcoded these values within my tests. If my tests were hardcoded with a 66% discount for products, then as soon as I change that discount amount, they would break.

However, thanks to this MyConfig object, I'm able to fix a certain value at the start of each test, and then test against that frozen value, such that my tests always test against a fixed value, let's say it's 50%, as opposed to the changing one I'm using here in MyConfig.

Or I can also change this buyer environment, as I sometimes do.

I'll give you an example of that over here.

Let's say that in the test environment, I always want to have as ”0.03”, whereas otherwise, I want to have it at ”0.05”... I don't know.

This piece of code would fix the value of the paypal_percent_cut for all the tests to ”0.03”.

The other thing I can do with this MyConfig object is to test against concepts.

For example, I have this test that I send out a holiday email when I'm on vacation, and this checks for the last_email sent (to: MyConfig[‘admin_email’]).

You saw this was info@xbride....co.uk, but this might change over time.

However, I assert against this concept of admin_email, rather than any particular instance of this.

This allows this test to have a larger shelf life.

Tip 7: Ensure A Clean Slate Between Tests

Much of the art of reducing brittleness in tests, comes down to ensuring that there is a clean slate, whenever every test is started.

It's very easy for state to leak between individual tests, and cause confusing results that are often dependent on the order in which the tests were run.

The first step to avoiding this is to take into account all the places where state gets accumulated.

In this particular app, there's active_storage, which is a Rails built-in for storing files. There is the test_downloads. What I mean by that is the downloads folder. Sometimes I ask the JS web browser to download individual files and they end up somewhere. I want to clear them between tests, to ensure that the download functionality works between tests.

Then there are emails that get sent by my Rails app, they should be cleared between each test.

I don't access an email from two tests ago in my current test, that would cause false positives and so on.

The Rails.cache: This is caches of objects and strings, especially HTML strings at the Rails level.

I test with my cache on, which is a little bit abnormal, but I find it helps me reduce my bug count, since my test code is more similar to my production code.

Then I can also accumulate state in redis, my key-value store.

And finally, I can accumulate state in my fulltext-search, which is Elastic Search.

All of these are places where a state can accumulate, therefore I need to go out of my way to clear them between tests.

I also obviously accumulate state in my Postgres database, but Rails automatically takes care of clearing that for me, so I don't have to think about it.

That is a lot of potential places where state can accumulate. And I would advise whenever you're adding a dependency of some sort of database, some sort of data store, to immediately think about automatically clearing that between test runs.

I prefer to clear before each test, instead of after each test.

The reason why is because sometimes you might press Ctrl-C to stop a test halfway through. If you stop the test halfway through in a sort of forceful way and after each block, i.e. some sort of cleanup that happens after the test, might not get run. Therefore, you get test leakage and you end up confused. Therefore, this is my preferred approach.

In larger codebases, it might be inefficient to clear every type of state between every test, but this isn't an issue for me so I just do it all at once.

That's all I've got to say for today.

Thanks for watching and see you next week.