Tests

The filosofy to test is:

In Travis CI these combinations are called a matrix. You can consider this overview per role:

Distribution Ansible 2.6 Ansible 2.7 Ansible devel
Alpine latest yes yes yes *
Alpine edge yes * yes * yes *
Archlinux (base) yes yes yes *
Centos 6 yes yes yes *
Centos latest yes yes yes *
Debian stable yes yes yes *
Debian latest yes yes yes *
Debian unstable yes * yes * yes *
Fedora latest yes yes yes *
Fedora rawhide yes * yes yes *
OpenSuse Leap yes yes yes *
OpenSuse Tumbleweed yes yes yes *
Ubuntu Artful (17) yes yes yes *
Ubuntu latest yes yes yes *
Ubuntu devel yes * yes * yes *

This matrix runs 15 (distributions) times 3 (Ansible versions) equals 45 build.

Distributions or Ansible versions marked with an astriks are allowed to fail. This combination is built to prepare for future distributions or releases of Ansible.

If a distribution or ansible version is not supported, the strategy is to also test that, and ensure if fails.

There are multiple tests configured, here is how they relate.

Full tests

A container does not offer all features that a full virtual machine offers, for example:

That’s why these roles offer two other molecule scenarios:

An overview of the test strategies and their features.

Scenario name Tests
default Multiple distributions
vagrant Full virtual machines
ec2 Commercial distributions*
travis Multiple Ansible versions

*) The testing of commercial distributions (RHEL & SLES) is not on by default and has to be uncommented.

Unit tests

To test an Ansible role, Travis CI runs molecule on a commit. This verifies that the role does it’s job, but does not ensure that it works in combination with other roles.

An example for the unit test for the Ansible role java.

Time based unit tests

Because distriutions, molecule, ansible and goss change over time, a monthly test is done to all roles using this schedule:

Day of month Ansible Role
1 anaconda
1 ansible
1 ansible_lint
1 apt_autostart
1 ara
1 artifactory
1 at
1 atom
1 awx
2 backup
2 bios_update
2 bootstrap
2 buildtools
3 ca
3 cargo
3 cntlm
3 common
3 cron
3 clamav
3 cloud9
4 dhcpd
4 digitalocean_agent
4 dns
4 docker
4 docker_ce
4 dovecot
5 epel
5 etherpad
6 fail2ban
6 firewall
7 git
7 glusterfs
7 go
7 gotop
8 haproxy
8 haveged
8 httpd
9 investigate
9 irslackd
10 java
10 jenkins
11 kernel
12 logrotate
12 logwatch
12 locale
12 lynis
13 memcached
13 mediawiki
13 mitogen
13 molecule
13 mssql
13 mysql
14 natrouter
14 nginx
14 npm
14 ntp
15 openssh
15 openvas
15 owncloud
16 packer
16 php
16 php_fpm
16 phpmyadmin
16 postfix
16 powertop
16 python_pip
18 reboot
18 redis
18 release
18 restore
18 revealmd
18 roundcubemail
18 rsyslog
18 ruby
18 rundeck
19 scl
19 selinux
19 service
19 snort
19 sosreport
19 spamassassin
19 squid
19 storage
19 sudo-pair
19 sysstat
20 terraform
20 tftpd
20 tomcat
20 travis
21 update
21 users
22 vagrant
24 xinetd
25 y
26 zabbix
26 zabbix_agent
26 zabbix_proxy
26 zabbix_repository
26 zabbix_server
26 zabbix_web

Integration

To test a combination of Ansible roles, Travis CI runs terraform and a complex playbook.

There is currently one integration test. A report is saved every run.