An overview of articles

A big thanks to … many

My parents, sister and family have been on a holiday to Lanzarote, Spain during christmast and new-year. 11 people in total, 6 adults, 5 children.

During this holiday our oldest got sick. It started as a fever but did not get better, even afer some 2 weeks.

Our oldest is an 8 year old boy, pretty strong and autonomous, no history of sicknesses.

We went to the doctor, he referred us to the hospital in Lanzarote, in Arrecife. He did not get better after receiving antibiotics for some 4 days.

A CT scan indicated his sinuses were infected an the infection broke through to his brains. A medical helicopter was called in to transport him to the hospital on another Canary island; Gran Canaria.

Surgery was performed and he was sent to the “high intensive care”. Quite impressive, but please realize what a horrible time this was for me and my wife.

It’s now 10 days after his surgery, he’s eating, speaking, moving and laughing again. We’ll probably spend this year recovering all of his functions, like walking, properly speaking, etc. I’m hopeful he’ll recover quickly and fully, but what a shock.

TL;DR: Son got sick, he’s doing well, time to code.

All the hospital days and nights gave me quite some time to write code. Because of crappy internet connections, It was a challenge to test code.

Here are the services I’ve used.

I’m very thankful for all the medical staff in Spain, I’m impressed by the quality of the hospital services and staff.

Family, friends and colleagues have expressed their concerns, it does not help my son, but it did help me to know that people are thinking of us.

How to write and maintain many Ansible roles

It’s great to have many code nuggest around to help you setup an environment rapidly. Ansible roles are perfect to describe what you want to do on systems.

As soon as you start to write more roles, you start to develop a style and way of working. Here are the tings I’ve learned managing many roles.

Use a skeleton for stating a new role

When you start to write a new role, you can start with pre-populated code:

ansible-galaxy init --role-skeleton=ansible-role-skeleton role_name

To explain what happens:

Use anible-lint for quick feedback

Andrew has written a tool including many rules that help you write readable and consistent code.

There are times where I don’t agree to the rules, but the feedback is quickly processed.A

There are also times where I initially think rules are useless, but after a while I’m convinced about the intent and change my code.

You can also describe your preferences and use ansible-lint to verify you code. Great for teams that need to agree on a style.

Use molecule on Travis to test

In my opinion the most important part of writing code is testing. I spend a lot of time on writing and executing tests. It helps yourself to prove that certain scenarios work as intended.

Travis can help test your software. A typical commit takes some 30 to 45 minutes to test, but after that I know:

  1. It works on the platforms I want to support.
  2. When it works, the software is released to Galaxy
  3. Pull requests are automatically tested.

It makes me less afraid of committing.

Use versions or tags to release software

When I write some new functionality, I typically need a few iterations to make it work. Using GitHub releases helps me to capture (and release) a working version of a role.

You can play as much as you want in between releases, but when a release is done, the role should work.

Go forth and develop!

You can setup a machine yourself for developing Ansible roles. I’ve prepared a repository that may help.

The playbook in that repository looks something like this:

---
- name: setup an ansible development environment
  hosts: all
  become: yes
  gather_facts: no

  roles:
    - robertdebock.bootstrap
    - robertdebock.update
    - robertdebock.fail2ban
    - robertdebock.openssh
    - robertdebock.digitalocean_agent
    - robertdebock.common
    - robertdebock.users
    - robertdebock.postfix
    - robertdebock.docker
    - robertdebock.investigate
    - robertdebock.ansible
    - robertdebock.ansible_lint
    - robertdebock.buildtools
    - robertdebock.molecule
    - robertdebock.ara
    - robertdebock.ruby
    - robertdebock.travis

  tasks:
    - name: copy private key
      copy:
        src: id_rsa
        dest: /home/robertdb/.ssh/id_rsa
        mode: "0400"
        owner: robertdb
        group: robertdb

    - name: copy git configuration
      copy:
        src: gitconfig
        dest: /home/robertdb/.gitconfig

    - name: create repository_destination
      file:
        path: ""
        state: directory
        owner: robertdb
        group: robertdb

    - name: clone all roles
      git:
        repo: "/.git"
        dest: "/"
        accept_hostkey: yes
        key_file: /home/robertdb/.ssh/id_rsa
      with_items: ""
      become_user: robertdb

When is a role a role

Sometimes it’s not easy to see when Ansible code should be captured in an Ansible role, or when tasks can be used.

Here are some guidelines that help me decide when to choose for writing an Ansible role:

Don’t repeat yourself

When you start to see that your repeating blocks of code, it’s probably time to move those tasks into an Ansible role.

Repeating yourself may:

Keep it simple

Over time Ansible roles tend to get more complex. Jeff Geerling tries to keep Ansible roles under 100 lines. That can be a challenge, but I agree to Jeff.

Whenever I open up somebody else’ Ansible role and the code keeps on scrolling, I tend to get demotivated:

Cleanup your playbook

Another reason to put code in Ansible roles, is to keep your playbook easy to read. A long list of tasks is harder to read than a list of roles.

Take a look at this example:

- name: build the backend server
  hosts: backend
  become: yes
  gather_facts: no

  roles:
    - robertdebock.bootstrap
    - robertdebock.update
    - robertdebock.common
    - robertdebock.python_pip
    - robertdebock.php
    - robertdebock.mysql
    - robertdebock.phpmyadmin

This code is simple to read, anybody could have an understanding what it does.

When input is required

Some roles can have variables to change the installation, imagine this set of variables:

httpd_port: 80

The role can assert variables, for example:

- name: test input
  assert:
    that:
      - httpd_port <= 65535
      - httpd_port >= 1

Check yourself

To verify that you’ve made the right decision:

Could you publish this role?

That means you did not put data in the role, except sane defaults.

Would anybody else be helped with your role?

That means you thought about the interface (defaults/main.yml).

Is there a simple way to test your role?

That means the role is focussed and can do just a few things.

Was it easy to think of the title?

That means you knew what you were building.

Conclusion

Hope this helps you decide when a role is a role.

Testing CVE 2018-19788 with Ansible

So a very simple exploit on polkit has been found. There is not solution so far.

To test if your system is vulnerable, you can run this Ansible role.

A simple playbook that includes a few roles:

---
- name: test cve 2018 19788
  hosts: all
  gather_facts: no
  become: yes

  roles:
    - robertdebock.bootstrap
    - robertdebock.update
    - robertdebock.cve_2018_19788

And a piece of altered-for-readability code from the role:

- name: create a user
  user:
    name: cve_2018_19788
    uid: 2147483659

- name: execute a systemctl command as root
  service:
    name: chronyd
    state: started

In my tests these were the results: (snipped, only kept the interesting part)

TASK [ansible-role-cve_2018_19788 : test if user can manage service] ***********
    ok: [cve-2018-19788-debian] => {
        "changed": false, 
        "msg": "All assertions passed"
    }
    fatal: [cve-2018-19788-ubuntu-16]: FAILED! => {
        "assertion": "not execute_user.changed", 
        "changed": false, 
        "evaluated_to": false, 
        "msg": "users can manage services"
    }
    ...ignoring
    fatal: [cve-2018-19788-ubuntu-18]: FAILED! => {
        "assertion": "not execute_user.changed", 
        "changed": false, 
        "evaluated_to": false, 
        "msg": "users can manage services"
    }
    ...ignoring
    fatal: [cve-2018-19788-ubuntu-17]: FAILED! => {
        "assertion": "not execute_user.changed", 
        "changed": false, 
        "evaluated_to": false, 
        "msg": "users can manage services"
    }
    ...ignoring
    fatal: [cve-2018-19788-fedora]: FAILED! => {
        "assertion": "not execute_user.changed", 
        "changed": false, 
        "evaluated_to": false, 
        "msg": "users can manage services"
    }
    ...ignoring
    fatal: [cve-2018-19788-centos-7]: FAILED! => {
        "assertion": "not execute_user.changed", 
        "changed": false, 
        "evaluated_to": false, 
        "msg": "users can manage services"
    }
    ...ignoring
    ok: [cve-2018-19788-centos-6] => {
        "changed": false, 
        "msg": "All assertions passed"
    }

So for now these distributions seem vulnerable, even after an update:

Ansible on Fedora 30.

Fedora 30 (currenly under development as rawhide) does not have python2-dnf anymore.

The Ansible module dnf tries to install python2-dnf if it running on a python2 environment. It took me quite some time to figure out why this error appeared:

fatal: [bootstrap-fedora-rawhide]: FAILED! => {"attempts": 10, "changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Error: Unable to find a match\n", "stderr_lines": ["Error: Unable to find a match"], "stdout": "Last metadata expiration check: 0:01:33 ago on Thu Nov 29 20:16:32 2018.\nNo match for argument: python2-dnf\n", "stdout_lines": ["Last metadata expiration check: 0:01:33 ago on Thu Nov 29 20:16:32 2018.", "No match for argument: python2-dnf"]}

(I was not trying to install python2-dnf, so confusion…)

Hm; so I’ve tried these options to work around the problem:

so far the only reasonable option is to set ansible_python_interpreter as documented by Ansible. Hm, so my molecule.yml now contains this code to tell Ansible to use python3:

provisioner:
  name: ansible
  inventory:
    group_vars:
      all:
        ansible_python_interpreter: /usr/bin/python3

This means all roles that use distributions that:

will need to be modified… Quite a change.

2 December 2018 update: I’ve created pull request 49202 to fix issue 49362.

TL;DR On Fedora 30 (and higher) you have to set ansible_python_interpreter to /usr/bin/python3.

Ansible Molecule testing on EC2

Molecule is great to test Ansible roles, but testing locally with has it’s limitations:

I use my bus-ride time to develop Ansible Roles and the internet connection is limited, which means a lot of waiting. Using AWS EC2 would solve a lot of problems for me.

Here is how to add an EC2 scenario to an existing role.

Save AWS credentials

Edit ~/.aws/credentials using information downloaded from [AWS Console].

[default]
aws_access_key_id=ABC123
aws_secret_access_key=ABC123

Install extra software

On the node where you initiate the tests, a few extra pip modules are required.

pip install boto boto3

Add a scenario

If you already have a role and want to add a single scenario:

cd ansible-role-your-role
molecule init scenario --driver-name ec2 --role-name ansible-role-your-role --scenario-name ec2

Start testing

And simply start testing in a certain region.

export EC2_REGION=eu-central-1
molecule test --scenario-name ec2

The molecule.yml should look something like this:

---
dependency:
  name: galaxy
driver:
  name: ec2
lint:
  name: yamllint
platforms:
  - name: rhel-7
    image: ami-c86c3f23
    instance_type: t2.micro
    vpc_subnet_id: subnet-0e688067
  - name: sles-15
    image: ami-0a1886cf45f944eb1
    instance_type: t2.micro
    vpc_subnet_id: subnet-0e688067
  - name: amazon-linux-2
    image: ami-02ea8f348fa28c108
    instance_type: t2.micro
    vpc_subnet_id: subnet-0e688067
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: ec2

Weirdness

It feels as if the ec2 driver has had a little less attention as for example the vagrant or docker driver. Here are some strange things:

Molecule and ARA

To test playbooks, molecule is really great. And since Ansible Fest 2018 (Austin, Texas) clearly communicated that Molecule will be a part of Ansible, I guess it’s safe to say that retr0h’s tool will be here to stay.

When testing, it’s even nicer to have great reports. That’s where ARA comes in. ARA collects job output as a callback_plugin, saves it and is able to display it.

Here is how to set it up.

Install molecule

pip install molecule

Install ara

pip install ara

Start ara

ara-manage runserver

Configure molecule to use ara

Edit molecule.yml, under provisioner:

provisioner:
  name: ansible
  config_options:
    defaults:
      callback_plugins: /usr/lib/python2.7/site-packages/ara/plugins/callbacks

Now point your browser to http://localhost:9191/ and run a molecule test:

molecule test

[204] Lines should be no longer than 120 chars

It seems Galaxy is going to use galaxy-lint-rules to star roles. One of the controls tests the length of the lines. Here are a few way so pass those rules.

Spread over lines

In YAML you can use multi line to spread long lines.

Without new lines

The > character replaces newlines by spaces.

- name: demostrate something
  debug: 
    msg: >
      This will just
      be a single long
      line.

With new lines

The | character keeps newlines.

- name: demostrate something
  debug:
    msg: |
      The following lines
      will be spread over
      multiple lines.

Move long lines to vars

Sometimes variables can get very long. You can save a longer variable in a shorter one.

For example, too long would be this task in main.yml:

- name: unarchive zabbix schema
  command: gunzip /usr/share/doc/zabbix-server-{{ zabbix_server_type }}-{{ zabbix_version_major }}.{{ zabbix_version_minor }}/create.sql.gz

Copy-paste that command to vars/main.yml:

gunzip_command: "gunzip /usr/share/doc/zabbix-server-{{ zabbix_server_type }}-{{ zabbix_version_major }}.{{ zabbix_version_minor }}/create.sql.gz"

And change main.yml to simply:

- name: unarchive zabbix schema
  command: "{{ gunzip_command }}"

Conclusion

Yes it’s annoying to have a limitation like this, but it does make the code more readable and it’s not difficult to change your roles to get 5 stars.

Ansible roles for clusters

Ansible can be used to configure clusters. It’s actually quite easy!

Typically a cluster has some master/primary/active node, where stuff needs to be done and other stuff needs to be done on the rest of the nodes.

Ansible can use run_once: yes on a task, which “automatically” selects a primary node. Take this example:

inventory:

host1
host2
host3

tasks/main.yml:

- name: do something on all nodes
  package:
    name: screen
    state: present

- name: select the master/primary/active node
  set_fact:
    master: ""
  run_once: yes

- name: do something to the master only
  command: id
  when:
    - inventory_hostname == master

- name: do something on the rest of the nodes
  command: id
  when:
    - inventory_hostname != master

It’s a simple and understandable solution. You can even tell Ansible that you would like to pin a master:

- name: select the master/primary/active node
  set_fact:
    master: ""
  run_once: yes
  when:
    - master is not defined

In the example above, if you set “master” somewhere, a user can choose to set a master instead of “random” selection.

Hope it helps you!

Ansible Galaxy Lint

Galaxy currently is a dumping place for Ansible roles, anybody can submit any quality role there and it’s kept indefinitly.

For example Nginx is listed 1122 times. Happily Jeff Geerling’s role shows up on top, probably because it has the most downloads.

The Galaxy team has decided that checking for quality is one way to improve search results. It looks liek roles will have a few criterea:

The rules are stored in galaxy-lint-roles. So far Andrew, House and Robert have contributed, feel free to propose new rules or improvements!

You can prepare your roles:

cd directory/to/save/the/rules
git clone https://github.com/ansible/galaxy-lint-rules.git
cd directory/to/your/role
ansible-lint -r directory/to/save/the/rules/galaxy-lint-rules/rules .

I’ve removed quite a few errors by using these rules:

You can peek how your roles are scored on development.

Ansible 2.7

As announced, Ansible 2.7 is out. The changed look good, I’m testing my bootstrap role agains it.

In 2.7 (actually since 2.3) all package modules don’t need with_items: or loop: anymore. This make for simpler code.

- name: customize machine
  hosts: all

  vars:
    packages:
      - bash
      - screen
      - lsof

  tasks:
    - name: install packages
      package:
        name: "{{ packages }}"
        state: present

Wow, that’s simpler so better.

A reboot module has been introduced. Rebooting in Ansible is not easy, so this could make life much simpler.

AnsibleFest2018 (Austin, Texas)

So, it’s been such a good week! Dennis, Marco, Jonathan and I (and 1300 other Ansible fans) visited Ansible Fest 2018.

Good to meet you

After working with quite some people online, I was really happy to finally meet some Ansible heroes:

Highlights

Although nearly each talk was valuable, these are some (randomly ordered) takeaways that influence me:

Difficult to overwrite a single value:

apache:
  start_servers: 2
  max_clients: 2

Easy to overwrite:

apache_start_servers: 2
apache_max_clients: 2

All in all, I’m really happy with the direction Ansible is going and feel that most decicions I’ve done in the past year are correct.

I expect that molecule testing will be integrated into Galaxy and reports will be created using ARA will be integrated.