improving tests: code coverage usecase

A recent addition to the release pipeline is the “code coverage” report. Although you can find several posts about the intricacies of code coverage for PowerShell (my favourite is this post by June Blender), you may be asking yourself what is code coverage in simple terms.

defining code coverage

Coupled with tests, code coverage is a metric that measures how much of the code is tested.

This is helpful to realize the test written is really covering for all the “features” of the code. A function with a test is a good thing. A function with a test that covers the 10% of the lines definitely helps, but tells you that a lot of the code involved may not be failproof.

While reaching 100% is not entirely possible due to Powershell limitations, it should be the goal for everyone. The more code coverage we reach:

  • the more stable the module.
  • the more “situations” are being checked and handled correctly
  • the more tests will be complete, documenting what to expect from the function
  • the more documentation about how a function needs to behave, the more users can fiddle with the code and add features being sure they don’t break anything in the process

We run a suite of tests at each commit via appveyor. The service we chose for code coverage is codecov. We chose them mostly due to the fact that their support is great and we needed their exclusive feature of “merge report” because we run different test suites for each build (even some C# code). Also, its interface is as good as a code coverage report can be, you can see the full source code and line-by-line coverage.

dbatools coverage

For public consumption, our development branch shows the most current coverage report for the whole project.

Now that the project is open about coverage metrics, you can contribute to the project writing more complete tests for all functions which are not 100% covered.

Let’s see a practical example.

A few days ago, on 2017-11-13, the development branch showed Get-DbaDbRecoveryModel.ps1 with a 90% coverage. Codecov’s urls are prettier but for the sake of this blogpost, which must point to specific commits, they’ll be lengthy and ugly.

original coverage

Clicking on the function itself, you can see that only one line is specifically not covered

original coverage – source details

Line 87 basically is about the possibility to specify a specific recovery model as a filter to retrieve only matching databases.

If you run Get-Help Get-DbaDbRecoveryModel you’ll see this parameter

If you inspect the relevant test you’ll see there is nothing relative to that specific parameter.

Unfortunately, this means that if someone makes a modification to the function in the future, there won’t be any assurance that parameter continues to work as expected. We want to extend that test to include that too, so that the behaviour will always be the same.

giving more coverage

First, we need to create a database to test against the new case. Remembering that dbatoolsci_ is the default prefix for any resource created, let’s go with dbatoolsci_getrecoverymodel.

Cannibalizing a recurrent pattern seen in most tests, let’s add a new Context with a BeforeAll and an AfterAll stanzas to keep everything clean

Summing up, BeforeAll runs before ANY test within the upper-level stanza (in our case, the brand new Context), while AfterAll runs after, no matter the results (keeping things clean is a priority, so cleanup needs to run no matter what)

You’ll see the BeforeAll checks and removes any preexisting dbatoolsci_getrecoverymodel database, it then proceeds to create a new one with BULK_LOGGED. The AfterAll instead just removes the database we created. Then, we create the real test case:

Now, let’s run the test locally, before submitting our changes. Create a C:\Temp\constants.ps1 or a .\tests\local.constants.ps1 that points to a live instance (in my case, the simplest of them all, localhost, results in this)

Then, let’s launch the test

manual test run

merging to the dbatools repository

Now that we’ve confirmed that everything checks out, let’s create the pull request (PR).

First, we open a proper feature branch spawned from development

add the relevant commits, then push the new branch

wait a few seconds, go on github and create the PR

open a PR

change the base branch to development

change base branch

Once you filled out the details, you’ll have written something like this PR

Now, we just wait for the build to complete (impatient ones, watch appveyor’s dbatools “homepage” to monitor the status in real-time)

inspecting the results

Once the build has done, the list of recent opened PRs on codecov is at dbatools.io/coverage

If you inspect the one I created with this blogpost, which is #2646 , when you browse to the functions folder to see what happened with the coverage of Get-DbaRecoveryModel.ps1 … voilà

coverage increase

You can click further to see the details: you can see that the line which wasn’t covered is now green.

coverage increase line by line

Enjoyed this post? Come join the Pester-lovers on Slack’s #dbatools-pester to improve current tests!

Ciao!
Simone 🇮🇹

Leave a Reply

Your email address will not be published. Required fields are marked *