Writing system tests

The QA Tools project employs the expect tool to perform system testing against the Phar file that is distributed to developers. Contributors can describe their expectations of the interactive dialogue with the tool and assert what the program results are. Together, these form a specification.

System specifications consist of two parts, a dialogue expectation and a test file. These must be placed in the directory ./tests/system/specs.

.
├── tests
│   ├── system
│   │   ├── specs
│   │   │   ├── reticulates-splines.php # Assertions
│   │   │   └── reticulates-splines.tcl # Dialogue expectation

The specification is executed in a temporary directory. Initially, this directory contains symbolic links to the distributable and its public key. This is also the directory where QA Tools will generate files.

/tmp
├── qa-tools_system5790bd1c4b5c68.27898406
│   ├── qa-tools
│   └── qa-tools.pubkey

The dialogue expectation

The dialogue expectations are written in TCL. No comprehensive knowledge of TCL is required to write these dialogue expectations.

First, you indicate what QA Tools subcommand must be executed with which arguments. It is important you specify that no ANSI colouring be included in the QA Tools' output.

test ./qa-tools configure --no-ansi

For each interactive question, state the expected question text, and the answer expect should send in response. The expectation times out after a hard-coded 2 seconds.

answer "What is the project's name?" with "Wobbly Widdershins"

The intermediate presence of a string can also be asserted.

should_see "Reticulating splines..."

answer and should_see are procedures defined in the expectation harness so that expressive dialogue expectations can be written.

Finally, you can assert that the program will end and will exit with a zero exit code. If the program does not end within the configured time-out the expect script exits with exit code 1. When the program exits with a non-zero exit code, the expect script exits with the same exit code.

exits_with 0

Pitfalls

One pitfall is the use of brackets in strings; these execute commands. The below snippet, expecting a specific multiple-choice answer, attempts to execute 0:

answer "[0] Symfony 3" with "0"

The solution is to escape the brackets:

answer "\[0\] Symfony 3" with "0"

The test file

The test file contains plain PHP. The expect script is available as the callable $expect. This enables you to arrange things before executing QA Tools, like adding conflicts to the project's Composer configuration. Note that the test file ought to reside in the namespace Ibuildings\QaTools\SystemTest; this makes sure PHPUnit's assertion functions are in scope.

<?php

namespace Ibuildings\QaTools\SystemTest;

Composer::initialise();

/** @var callable $expect */
$expect();

assertFileExists('qa-tools.json');
Composer::assertPackageIsInstalled('phpmd/phpmd');

Multiple dialogue expectations

When a specification requires multiple interactions with the QA Tools program, you can use multiple dialogue expectations. By passing a chapter to the $expect helper callable, different dialogues are executed.

// 050_my-first-test.php

namespace Ibuildings\QaTools\SystemTest;

// Runs dialogue 050_my-first-test_setup.tcl
$expect('setup');
// Runs dialogue 050_my-first-test_test.tcl
$expect('test');

Example specification

spawn ./qa-tools configure --no-ansi

should_see "Configuring the Ibuildings QA Tools"

answer "What is the project's name?" with "Boolean Bust"

answer "Where would you like to store the generated files?" with "./"

should_see "What type of project would you like to configure?"
answer "\[0\] PHP" with "0"

should_see "What type of project would you like to configure?"
answer "\[1\] Symfony 3" with "1"

answer "Would you like to integrate Travis in your project?" with "Y"

exits_with 0

<?php

namespace Ibuildings\QaTools\SystemTest;

Composer::initialise();

/** @var callable $expect */
$expect();

assertFileExists('qa-tools.json');
Composer::assertPackageIsInstalled('phpmd/phpmd');

Installing Composer packages

Most tools will want to install Composer packages. However, installing these packages during testing, both locally as on Travis, makes tests slow and brittle. To countermand this, all tests involving Composer (pretty much all system tests) work with locally emulated packages. An example of this is phpmd/phpmd, emulated in tests/composer/packages/phpmd/phpmd/composer.json. These emulated packages are registered with Composer during the bootstrap of the test suite in Ibuildings\QaTools\ComposerTest\Composer::mockRepositories().