Tests
The filosofy to test is:
- Test multiple distributions
- Of each distribution, test the current and previous version
- Test multiple version of Ansible, current, previous and next previous and future.
In Travis CI these combinations are called a matrix. You can consider this overview per role:
| Distribution | Ansible 2.6 | Ansible 2.7 | Ansible devel |
|---|---|---|---|
| Alpine latest | yes | yes | yes * |
| Alpine edge | yes * | yes * | yes * |
| Archlinux (base) | yes | yes | yes * |
| Centos 6 | yes | yes | yes * |
| Centos latest | yes | yes | yes * |
| Debian stable | yes | yes | yes * |
| Debian latest | yes | yes | yes * |
| Debian unstable | yes * | yes * | yes * |
| Fedora latest | yes | yes | yes * |
| Fedora rawhide | yes * | yes | yes * |
| OpenSuse Leap | yes | yes | yes * |
| OpenSuse Tumbleweed | yes | yes | yes * |
| Ubuntu Artful (17) | yes | yes | yes * |
| Ubuntu latest | yes | yes | yes * |
| Ubuntu devel | yes * | yes * | yes * |
This matrix runs 15 (distributions) times 3 (Ansible versions) equals 45 build.
Distributions or Ansible versions marked with an astriks are allowed to fail. This combination is built to prepare for future distributions or releases of Ansible.
If a distribution or ansible version is not supported, the strategy is to also test that, and ensure if fails.
There are multiple tests configured, here is how they relate.
Full tests
A container does not offer all features that a full virtual machine offers, for example:
- systemd does not work in many major distributions container.
- rebooting and containers are not a thing.
- Some files in /etc are mounted from the host to the guest.
That’s why these roles offer two other molecule scenarios:
- vagrant - to locally start full virtual machines.
- ec2 - to remotely tests commercial full virtual machines.
An overview of the test strategies and their features.
| Scenario name | Tests |
default |
Multiple distributions |
vagrant |
Full virtual machines |
ec2 |
Commercial distributions* |
| travis | Multiple Ansible versions |
*) The testing of commercial distributions (RHEL & SLES) is not on by default and has to be uncommented.
Unit tests
To test an Ansible role, Travis CI runs molecule on a commit. This verifies that the role does it’s job, but does not ensure that it works in combination with other roles.
An example for the unit test for the Ansible role java.
Time based unit tests
Because distriutions, molecule, ansible and goss change over time, a monthly test is done to all roles using this schedule:
Integration
To test a combination of Ansible roles, Travis CI runs terraform and a complex playbook.
There is currently one integration test. A report is saved every run.