The release of FASLR v0.0.7 introduces a test suite to ensure the accuracy of calculations. A test suite is a sub-package or collection of files that runs the project’s code against a set of expected inputs, and compares the generated outputs to a set of expected outputs. For example, one of the tests may run the chain ladder method against a sample dataset to make sure the resulting ultimate losses a calculated correctly. This test is then run whenever we make code changes to the project. In this manner, we can check that FASLR continues to work correctly as the project matures.
If you are new to FASLR, it stands for Free Actuarial System for Loss Reserving. This update has more of a project management-related focus, so if you are more interested in discovering FASLR’s capabilities, you can browse previous posts about the project on this blog. You are also free to browse the source code on the CAS GitHub or view the documentation on the project’s website.
Motivations
What motivated this update, focusing on the development workflow rather than adding new features, was that certain packages that the project depends on, such as pandas and sqlalchemy, have become out-of-date. Although I had the option of simply upgrading all of the packages without running the code against a test suite, I figured the time has come to add one since the project was growing in complexity and has picked up some interest from people other than myself, so there was the need to make sure that further updates to the project would not break its existing functionality.
My IDE telling me my dependencies are out of date.
FASLR’s sister project, chainladder-python, has had a working test suite for several years now, so adding one would demonstrate that the project aims to deliver on quality, and further cement the CAS GitHub as a guide for other actuaries on how to properly use Python, write libraries, and deploy software in their own careers.
Preventing Errors
When an actuary assesses the accuracy of a project, not just one involving code, but anything involving computations – they tend to ask themself the following questions:
- How do I know my formulas are correct?
- If I make changes, how can I make sure they don’t break existing computations?
- If more people join the project and start contributing, how can I integrate their changes without messing things up?
If you have ever done a data reconciliation, or added a spreadsheet to check premium and loss totals against your data source, then you have already performed some testing on your work. If any changes to your project were to break your reconciliation sheet or lead to odd totals, then the change has broken your project. The motivations behind a software testing suite are no different – except instead of adding extra formulas to a spreadsheet, we add extra lines of code to a project.
Building a Testing Suite
There are several ways to build a test suite – for example, you may choose to write your own custom functions to test critical parts of your code or you can use a framework such as unittest or pytest. A framework, usually implemented in the form of a package, can make the testing process easier by providing a standardized way to separate test code from feature code, running the entire test suite or specified parts of it, or reducing redundancy by enabling test functions to run multiple times using different input data sets.
I chose to use pytest, partly because chainladder also uses it, and also because members of the Python community recommend it. Writing a test suite for FASLR also has the complication of having to test GUI components, which unlike testing a package, involves testing visual features such as button clicks, key presses, and menu selections. In order to do this, I used a package called pytest-qt, which invokes a bot to simulate user actions. For example, this bot can be used to enter a custom development factor into a field and then pytest can use that value to see if it correctly flows through the intended calculations.
Automating tests with GitHub Actions
FASLR combines pytest with GitHub Actions to automatically run its test suite upon certain events, such as when changes are pushed to the repo, or when someone opens up a pull request, allowing me to make sure their proposed changes won’t break the program.
Code Coverage
One question that arises when building a test suite is – how do I know I’ve written enough tests? Although I don’t believe there is a bulletproof way to prevent all bugs, there are certain metrics we can use to gain comfort and convince others that we have put in a good faith effort to do so. One such metric is code coverage, which is the percentage of lines of your project that the test suite has run. For example, if a project has 95% code coverage, 95% of the source code was executed by the test suite. This does not mean your code is 95% bug free, just that 95% of it has been executed. This also does not mean that your tests were well-thought out or that you correctly anticipated that parts of the code that needed the most testing. It can however, help to identify glaring holes in your package that you may have omitted from testing.
FASLR uses codecov, a third-party product that integrates with GitHub and hosts your project’s coverage reports. In the image below, we can see that our test suite has covered 92% of our code:
A coverage report can provide detail as to which files are completely covered, and which files need further testing. The example below shows some of the output from pytest:
To view the codecov summary on FASLR, click here.
Other Updates
FASLR v0.0.7 has added some further updates, in addition to the test suite:
- An improved heatmap function, written by P. Sharma
- Automated documentation deployment via GitHub Actions
- Schema documentation page on the FASLR website
- A development chatroom, on Gitter