Java

assertTrue vs assertThat vs assertJ

Jakub Jelonek

Feb 14, 2018 • 4 min read

A while ago we’ve participated in a discussion about the pros and cons of using assertJ and assertThat over assertTrue¹.

We’ve heard some arguments against using assertThat and assertJ that we really did not anticipate (e.g. a dev expressed that he prefers to see exactly what methods are being tested, as opposed to relying on auto generated code). We’ve given examples and arguments of why assertTrue is inferior to the two other methods, but we couldn’t quite convince assertTrue supporters.

Recently, after being affected by a test written using assertTrue, I've decided to

First, a little back story

I've just merged changes made on develop into my own branch. I've noticed that a couple of things weren't compiling, so I went ahead and fixed them. The issues with compilation were fine, because I changed the return type of a method from String (that was a hex-encoded byte[]) to byte[].

After making some code fixes, I ran the tests (which also needed slight tweaking), and got a failure.

The Tests

Assert True

Observe the following test method, written using assertTrue:

@Test
public void get_public_key_hash_returns_hex_encoded_sha256_assertTrue() {
  // get the public key hash
  CryptoSystemApiDelegate api = new CryptoSystemApiDelegate(ivCryptoApiLifeCycle, ivApiKeyVerifier);
  Response res = api.getPublicKeyHash(API_KEY, null);
  Object entity = res.getEntity();

  // test the hash
  Assert.assertNotNull(entity);
  Assert.assertTrue(entity instanceof PublicKeyHash);
  PublicKeyHash pubKeyHash = (PublicKeyHash) entity;
  Assert.assertTrue("PublicKeyHash.algorithm is not SHA-256", pubKeyHash.getAlgorithm().toString().equals("SHA-256"));
  Assert.assertTrue("PublicKeyHash.value is null.", pubKeyHash.getValue() != null);
  Assert.assertTrue("Public Key Hash Value is not a SHA-256 value", pubKeyHash.getValue().length == 64); // Hex Encoded SHA-256 is 64 chars
}

and the output it produced:

java.lang.AssertionError: Public Key Hash Value is not a SHA-256 value

at
xyz.smartcoders.CryptoApiLifeCycleIT.get_public_key_hash_returns_hex_encoded_sha256_assertTrue(CryptoApiLifeCycleIT.java:94)

Ok... what the hell does it mean that it's not a SHA-256 value?!? What does that mean? The way to find out? Debug. What does it cost? Time.

In an attempt to learn more (and to have a better insight to see if there's qualitative improvement in using different approaches), I've rewritten the test using assertThat and assertJ.

Assert That

Now observe output of a modified test method, written using assertThat:

@Test
public void get_public_key_hash_returns_hex_encoded_sha256_assertThat() {
  // get the public key hash
  CryptoSystemApiDelegate api = new CryptoSystemApiDelegate(ivCryptoApiLifeCycle, ivApiKeyVerifier);
  Response res = api.getPublicKeyHash(API_KEY, null);
  Object entity = res.getEntity();

  assertThat(entity, is(instanceOf(PublicKeyHash.class)));

  PublicKeyHash pubKeyHash = (PublicKeyHash) entity;

  assertThat(pubKeyHash.getAlgorithm(), equalTo("SHA-256"));
  assertThat(pubKeyHash.getValue(), notNullValue());
  assertThat(pubKeyHash.getValue().length, equalTo(64));
}

and the output it produced:

java.lang.AssertionError: 
Expected: <64>
     but: was <32>

at xyz.smartcoders.CryptoApiLifeCycleIT.get_public_key_hash_returns_hex_encoded_sha256_assertThat(CryptoApiLifeCycleIT.java:115)

Bingo! That's more like it! The returned value is 32 bytes long, not 64 as was expected! Because it's not hex-encoded anymore, it's just a byte array, so the value is 32 bytes long, indeed!

That's already enough to quickly figure out what's wrong, and how to fix it.

AssertJ

To get a better picture of what I could expect from assertJ, I've decided to write the same test using just that.

@Test
public void get_public_key_hash_returns_hex_encoded_sha256_assertj() {
  // get the public key hash
  CryptoSystemApiDelegate api = new CryptoSystemApiDelegate(ivCryptoApiLifeCycle, ivApiKeyVerifier);
  Response res = api.getPublicKeyHash(API_KEY, null);

  Object entity = res.getEntity();

  assertThat(entity, is(instanceOf(PublicKeyHash.class)));

  PublicKeyHash publicKeyHash = (PublicKeyHash) entity;

  PublicKeyHashExtendedAssert.assertThat(publicKeyHash)
    .hasAlgorithm("SHA-256")
    .hasValueWithLength(64);
}

Let's pause here for a moment, before looking at the output. Looking at the code, you immediately see that you're expecting the public key hash to be a 64-byte long SHA-256 value! With so little code! And just take a look at the output it produces:

java.lang.AssertionError:
Expected size:<64> but was:<32> in:
  <[-63, -17, 64, -6, 41, -75, -40, 75, 74, 89, 35, -12, 81, 91, 97, 47, 47, -21, -77, 32, -41, -30, -106, 26, 50, -65, 69, -94, -1, -55, 96, -65]>

at xyz.smartcoders.CryptoApiLifeCycleIT.get_public_key_hash_returns_hex_encoded_sha256_assertj(CryptoApiLifeCycleIT.java:135)

The output is just as useful as that of assertThat, but it also provides the byte[] that was being tested. While in this case it's not very useful, as the value is random, it might be very handy in other tests!

Conclusions

I think that this demonstrates, beyond any doubt, in my mind, that assertTrue is by far the worst, giving you the least context. assertTrue also heavily depends on the developer correctly and clearly expressing what is being tested. If the description is not clear, the result of the test will not give you any indication about what's wrong at best. At worst, it'll confuse the heck out of you.

assertThat is far better; it's good enough to quickly figure out what's wrong. The description is provided by the framework, so the developer doesn't need to spend time writing potentially confusing messages.

assertJ is by far the best. It's not only concise in expressing expectations, but also in providing the most context to facilitate quick and painless fix.

Footnote

If you say that I should have thought about changing the expected length of the public key hash from 64 to 32 upon changing encoding—you're right! I'm sure you would've caught it and it wouldn't be an issue. But I'm not that good; I'll take any help I can get to fix things quickly.

I need to underline that my gripe is with the abuse of assertTrue for all assertions in test code. I'm not saying that assertTrue is not the way to go when testing boolean values. What prompted me to write this was the proliferation of tests written using assertTrue exclusively.