Skip to main content
Back to Reports

Testing our Toolkit

Tests we run

This toolkit is built using an open-source tool called Jekyll. We write content in markdown format, which is then parsed by Jekyll which produces a static HTML website that we host on Github pages.

Content and other changes are proposed by team members using the pull requests on Github. There are a bunch of automated tests the proposed change must satisfy to be accepted (as well as manual testing). The automated tests are fast and means the pull request is in good shape by the time a team member begins their peer review. The tests catch many of our mistakes, as well as being an easy way to learn about content standards.

Flow diagram of our tests

Our Tests

We use a continuous integration service called CircleCI to run our tests. CircleCI is an open-source tool and is available to us for free because our projects are also Open Source (as per NZGOAL).


YAML is a standard format for configuration files (and also the name of Brenda’s cat). Jekyll uses YAML, so we use a linter to inspect our files. yamllint is a python tool available on pip. It catches any strange/incorrect indenting with better and more helpful errors messages than the tool that reads those YAML.


CI builds the static content on every pull request to ensure that once your change it merged into mainline (the dev branch) it can be successfully built for deployment


markdownlint is a linter that checks you have followed our style guide for markdown files. The linter catches when you’re using bold instead of headers. It fails your pull request if you don’t put descriptions on images if it’s not clear which are the header cells on a table, and if your header nesting doesn’t make sense in a tree structure.


Percy is a service that takes screenshots of every page and compares them to the previous screenshot. If there are differences (i.e. you’ve modified a page on the toolkit) you’ll need to go to Percy and then view and approve the changes. If you’ve broken the display on a page, you’ll see it in the screenshots (and of course if you’ve broken a page, don’t approve it!)


The accessibility check uses a tool called pa11y-ci to check we adhere to WCAG21 standard. The New Zealand Government - Web Accessibility Standard 1.1 is based on WCAG21. This linter is configured by a file named .pa11yci.json.


Automated tests catch common mistakes quickly, and more thoroughly than our manual testing. A linter can give quick feedback on proposed changes before the author asks another team member for their peer review. With automated content testing, we can get better quality markup on our toolkit pages, and with a quicker publishing turnaround.

On a pull request the checks look like this:

github screen shot, showing passing, failing, and pending checks

Related posts