Improving unit tests with equivalence instead of equality for URLs

Like any good developer I try to ensure that my code is not just covered with unit tests, but that they are meaningful. However I often find that my tests would break because they made too many assumptions about the implementation of the code it was testing. In this post I'm going to be using the example of testing URLs, but this approach can be taken more generally. When working with web services I often find myself writing code that takes an model object and produces a URL request. So naturally I'll write unit tests for this conversion and typically I would have written something like below, were for a given input I check the output URL is equal to what I expect.


For simple cases this works fine, but as things get more complex I would find my tests would become very fragile because the order of the query items could change. This might be because I had used a Dictionary or a Set were the order of the elements is not guaranteed. Indeed even in the simple case above my test makes implicit assumptions about the implementation. The is no reason for firstname to be the first item in the query string, the URL would still be perfectly valid if the order of the items were rearranged or indeed if any of the characters in the URL were percent encoded. My unit test is highly coupled to the specific implementation of my function. Now in this trivial example that may not matter too much, however as things get more complex this coupling leads to unit tests that are brittle. At the root of this brittleness is the equality assertion. When an object (such as a URL) can have multiple alternative but equivalent forms then using equality leads to implementation coupling.

Escaping the tyranny of exact equality

To break this coupling, I've used the concept of Equivalence that will test if two objects are alternative forms of each other. I've created a protocol Equivalent that is modelled after the Equatable protocol in the standard library. Using the triple-bar operator (≡) as the equivalent-to operator. With this we can have types conform to the Equivalent protocol were they have alternative forms.

Equivalence for URLs

Now that we have the Equivalent protocol we apply it to URL. In the below code I use the URLComponents class to do much of the heavy lifting. This implementation will account for the following alternative forms:-

• Query item order independence.
• Percent encoding alternative forms.
• Default port numbers.
• Relative paths.

Writing equivalent unit tests

For consistency with the rest of the unit tests I wrote a function XCTAssertEquivalent similar to XCTAssertEqual, as it's name implies it uses equivalence rather than equality (see the attached playground for the code). The original unit test changes to the following.

Not much has change with the code of the test, however now the test is far more flexible on testing the output of the function and therefore less coupled to the implementation.

Playground file available here.

Tracking down bugs using git bisect with Xcode unit tests

Overview

When working on software projects, fixing bugs is an inevitable fact of life. Sometimes you're lucky and the nature of the issue is obvious, however other times tracking down the source of the issue can be a time consuming and laborious task. If your project is using git for source control, then git bisect can be used to track down the commit that introduced the bug. Hopefully by narrowing the focus of investigation to the changes within a single commit, finding the root cause is significantly easier. This is especially true if commits within your project are small and focused.

Git bisect works on the basis of giving a known bad commit (were the bug is present, such as the current head) and a earlier good commit (were the bug is not present). The command works by switching into a special bisect mode were git will checkout a commit and the user will label that commit as good or bad. Git will then checkout a new commit and the process continues until git has determined the first bad commit. For those interested git uses a binary search of the commits in between the initial good and bad commit. So not every commit in-between the initial starting points will be checked. We enter bisect mode by running the following from the terminal: -

$ git bisect start [bad commit] [good commit]

Once in bisect mode, git will checkout a commit for us that we need to label as either: -

• good – The bug is not present.
• bad – The bug is present.
• skip – It could not be determined if the bug was present, such as the project failed to compile.

$ git bisect good | bad | skip

After labelling the commit git bisect will checkout a new commit and the process continues until the first bad commit is found.

Automating the process

This works, however this requires user interaction to determine if the bug is present within a given commit. Instead let’s consider writing a unit test that will determine that for us. That way we could set the whole process in motion and git bisect could determine the first commit in the range that causes our unit test to fail. Git bisect has a command to run a shell command to automatically label a commit.

$ git bisect run [cmd]

The exit code of the shell command is used to determine the label for the commit.

• Exit Code 0 - good
• Exit Codes 1 through 127, excluding 125 - bad
• Exit Code 125 - skip

Apple has documented the process of running unit tests from the command in Automating The Test Process using the xcodebuild command. For those interested Apple's Building from the Command Line with Xcode FAQ is also a good source of information.

The last piece of the puzzle is how to run a unit test that was not part of the project in the past and therefore when git performs a checkout the test will not be present. The simplest method I found was to create a stash of the changes to add the unit test into the project. That way we can apply the stash after each checkout, but before running the tests.

To make running the unit test easier we'll write a shell script to build and run the test target.

You'll notice that the script separates the steps of building the test target from running the unit test. In an ideal world each commit in your project would compile and run. However the reality is that some commits may break the project and fail to build. In these situations we exit the script with status code 125 which git bisect will interpret as the skip command. This will get us to carry on our search, at the cost of potentially making the end result less focused. Additionally the script uses the -only-testing argument to xcodebuild to only run a specific unit test rather than all of them.

Example

If like me you find a worked example easier to digest. Let's use a small, if somewhat contrived example. You'll need to suspend your sense of disbelief here as the source of the bug will be some what obvious. However the aim of the exercise is working with git bisect not fixing this particular issue. 

This demo project can be found on GitHub here. The project is an iOS app, that has a simple slider control for numbers between 0 and 10 and a label that will report the current value as well as the sum and average of all integers between 0 and the current value.

To makes things a little more realistic within the repository I have created a main develop branch and 3 feature branches that are created from develop before being reintegrated.

At some point in the development a bug was introduced such that if the slider is at position zero the app will terminate. This did not occur initially so let's use the technique above to determine the commit that introduced the bug.























We start by writing a unit test that can be run to determine if the bug is present.



Next we stash this change so that it can be applied for each checkout.

$ git stash save "unit test"

We can then start the git bisect session with the given bad and good commits. Then issue the command for git bisect to run the script automatically.

$ git bisect start 08a7cbf 664b6a1
$ git bisect run ./bisect-unittests/BisectDemo/unittest.sh

The process will take off and after awhile the following printed out: -

fb4b22d is the first bad commit
commit fb4b22d
Author: Raymond McCrae
Date:   Tue Mar 20 20:06:39 2018 +0000

    Refactor Utilities.average


bisect run success

Looking at the commit the following method

was changed to

During the refactor a bug was introduced that an empty array will cause a division by zero causing the crash. It is worth pointing out that despite giving the initial range of commits from the develop branch, the commit git bisect identified is on feature/3 branch. Clearly git bisect will follow all paths in between the initial commits not just a simple straight path along the current branch.

Finally we can leave bisect mode by running: -

$ git bisect reset

Since we have a unit test we can include it with any fix that we make to the code to ensure that this type of bug is not made again.

Conclusion

Git bisect is a powerful tool, especially when combined with automation. It is however more suited to finding regressions were we had a previous good state and now things are broken. I have used unit tests in this post, however it would also possible to you UI automation tests instead.