A recent addition to the release pipeline is the “code coverage” report. Although you can find several posts about the intricacies of code coverage for PowerShell (my favourite is this post by June Blender), you may be asking yourself what is code coverage in simple terms.
defining code coverage
Coupled with tests, code coverage is a metric that measures how much of the code is tested.
This is helpful to realize the test written is really covering for all the “features” of the code. A function with a test is a good thing. A function with a test that covers the 10% of the lines definitely helps, but tells you that a lot of the code involved may not be failproof.
While reaching 100% is not entirely possible due to Powershell limitations, it should be the goal for everyone. The more code coverage we reach:
- the more stable the module.
- the more “situations” are being checked and handled correctly
- the more tests will be complete, documenting what to expect from the function
- the more documentation about how a function needs to behave, the more users can fiddle with the code and add features being sure they don’t break anything in the process
We run a suite of tests at each commit via appveyor. The service we chose for code coverage is codecov. We chose them mostly due to the fact that their support is great and we needed their exclusive feature of “merge report” because we run different test suites for each build (even some C# code). Also, its interface is as good as a code coverage report can be, you can see the full source code and line-by-line coverage.
dbatools coverage
For public consumption, our development branch shows the most current coverage report for the whole project.
Now that the project is open about coverage metrics, you can contribute to the project writing more complete tests for all functions which are not 100% covered.
Let’s see a practical example.
A few days ago, on 2017-11-13, the development branch showed Get-DbaDbRecoveryModel.ps1 with a 90% coverage. Codecov’s urls are prettier but for the sake of this blogpost, which must point to specific commits, they’ll be lengthy and ugly.
Clicking on the function itself, you can see that only one line is specifically not covered
Line 87 basically is about the possibility to specify a specific recovery model as a filter to retrieve only matching databases.
If you run Get-Help Get-DbaDbRecoveryModel you’ll see this parameter
-RecoveryModel <String[]> Filters the output based on Recovery Model. Valid options are Simple, Full and BulkLogged Details about the recovery models can be found here: https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/recovery-models-sql-server
If you inspect the relevant test you’ll see there is nothing relative to that specific parameter.
Unfortunately, this means that if someone makes a modification to the function in the future, there won’t be any assurance that parameter continues to work as expected. We want to extend that test to include that too, so that the behaviour will always be the same.
giving more coverage
First, we need to create a database to test against the new case. Remembering that dbatoolsci_ is the default prefix for any resource created, let’s go with dbatoolsci_getrecoverymodel.
Cannibalizing a recurrent pattern seen in most tests, let’s add a new Context with a BeforeAll and an AfterAll stanzas to keep everything clean
Context "RecoveryModel parameter works" {
BeforeAll {
$server = Connect-DbaInstance -SqlInstance $script:instance2
$dbname = "dbatoolsci_getrecoverymodel"
Get-DbaDatabase -SqlInstance $server -Database $dbname | Remove-DbaDatabase -Confirm:$false
$server.Query("CREATE DATABASE $dbname; ALTER DATABASE $dbname SET RECOVERY BULK_LOGGED WITH NO_WAIT;")
}
AfterAll {
Get-DbaDatabase -SqlInstance $script:instance2 -Database $dbname | Remove-DbaDatabase -Confirm:$false
}
}
Summing up, BeforeAll runs before ANY test within the upper-level stanza (in our case, the brand new Context), while AfterAll runs after, no matter the results (keeping things clean is a priority, so cleanup needs to run no matter what)
You’ll see the BeforeAll checks and removes any preexisting dbatoolsci_getrecoverymodel database, it then proceeds to create a new one with BULK_LOGGED. The AfterAll instead just removes the database we created. Then, we create the real test case:
It "gets the newly created database with the correct recovery model" {
$results = Get-DbaDbRecoveryModel -SqlInstance $script:instance2 -Database $dbname
$results.RecoveryModel -eq 'BulkLogged' | Should Be $true
}
It "honors the RecoveryModel parameter filter" {
$results = Get-DbaDbRecoveryModel -SqlInstance $script:instance2 -RecoveryModel BulkLogged
$results.Name -contains $dbname | Should Be $true
}
Now, let’s run the test locally, before submitting our changes. Create a C:\Temp\constants.ps1 or a .\tests\local.constants.ps1 that points to a live instance (in my case, the simplest of them all, localhost, results in this)
PS C:\dbatools-dev\tests> get-content .\constants.local.ps1 $script:instance1 = "localhost" $script:instance2 = "localhost" $script:instance1_detailed = "localhost,1433" $script:appveyorlabrepo = "C:\github\appveyor-lab" $instances = @($script:instance1, $script:instance2) $ssisserver = "localhost"
Then, let’s launch the test
.\manual.pester.ps1 -path .\Get-DbaDbRecoveryModel.Tests.ps1 -TestIntegration
manual test runmerging to the dbatools repository
Now that we’ve confirmed that everything checks out, let’s create the pull request (PR).
First, we open a proper feature branch spawned from development
git checkout development git checkout -b tests/Get-DbaRecoveryModel
add the relevant commits, then push the new branch
git push origin tests/Get-DbaRecoveryModel
wait a few seconds, go on github and create the PR
open a PRchange the base branch to development
change base branchOnce you filled out the details, you’ll have written something like this PR
Now, we just wait for the build to complete (impatient ones, watch appveyor’s dbatools “homepage” to monitor the status in real-time)
inspecting the results
Once the build has done, the list of recent opened PRs on codecov is at dbatools.io/coverage
If you inspect the one I created with this blogpost, which is #2646 , when you browse to the functions folder to see what happened with the coverage of Get-DbaRecoveryModel.ps1 … voilà
coverage increaseYou can click further to see the details: you can see that the line which wasn’t covered is now green.
coverage increase line by lineEnjoyed this post? Come join the Pester-lovers on Slack’s #dbatools-pester to improve current tests!
Ciao!
Simone 🇮🇹