Testing ansible playbooks

This document describes a testing environment in ansible that is not ment to check your code, but to test the functionality of the platform your code implements...

Most organizations have linting and testing for ansible code in place, but in complex enviroments, testing of functionality is mostly done after delivery in the testing environment. Testing in development before delivery, can speedup delivery and help your department to deliver 'First Time Right' platforms. It also enables developers to do regression tests on changed platforms. Just run it again..

An additional goal was, it must run from the command line and in 'Tower or AWX', without modifications.
This gives great power, not having to wait for the run to start in tower, just run it from the commandline.....
Or shedule all tests on a regular basis, just to be sure..

With that goal in mind, this was created.

The code explained here is created with collections in mind. To test idividual roles the base code must be somewhat adapted.
If you understand what I am doing here, you can

What does it do?

Out of the box? almost nothing...

Just the framework is in the repository, no tests that run on any platform. It just does one thing 'out of the box', checkout your configured repository to a local directory and run a syntax check, the rest is up to you.

If you use custom credentials in Ansible Automation Platform, it can check the presence of them before the run starts.

If you use a vault role, it will also check it out and use the vars defined in them.

What more do we need?
- a webserver for the reporting (csv)file (can also be an ordinary server) - maybe grafana reading the csv for dashboard

As the test playbok runs, it will write its status 3 times into the remote report.csv: - RUNNING - FAILED - SUCCESS along with a number of other fields that identify the running code..

Yust create a simple dashboard, and show your code maturity The simple way
A more colorfull (management) version

What can it do?

Your creativity is the only limit this framework has....

Examples are:

Oracle dataguard testing

Runnning a complete baseline test on an Oracle dataguard cluster, with switchovers, failovers, data retension tests and pluggable database verifications in between the scenarios.
All in one run

Loadbalacer tests

A load balancer does nothing without an applicaion behind it, so we created a test platform with:

2 apache servers
2 loadbalancer nodes (ipvs in this case)

The first test is a verification if it all works..

After that the play orchestrates failures on all parts of the platform and verifies that the page on the site is still serviced.

Only when all tests pass, the platform may be delivered.

This gives assurance that your code is correct.

How do we implement this?

Read the next pages to find out more on how to implement the test plan.

Setup the template in GIT

In this repository, you will find the template code, just copy and store it in a new template repository for testing.

This template repository looks as follows:

.
├── ansible.cfg
├── collections
│   └── requirements.yml
├── group_vars
│   └── all.yml
├── main.yml
├── prep_03_check_credentials.yml
├── README.md
├── roles
│   └── requirements.yml
├── test_01_run_collection_roles.yml
├── test_number_role_rolename.yml
└── vars.yml

Create a new repository in your project named 'test_plan_template' or something like it. Be sure to make this a template, not modifying the code. This is where you fork the other testing repositories from.

that's all...

Now you are almost ready to let your creativity take control..

The files in the template repository

ansible.cfg
collections/requirements.yml
group_vars/all.yml
main.yml
prep_03_check_credentials.yml
README.md
roles/requirements.yml
test_01_run_collection_roles.yml
test_101_role_name_test_name.yml
vars.yml

Fork the template

Lets start simple...
Say you want to test apache server in you development environment..

Fork (or make a copy) the test_plan_template into a new repository named
apache_testing.
Clone the new repository to your work environment, using git clone.

You now have a complete framework on your machine to test....
well, nothing yet :-)

The apache_testing repo needs to be configured for the environment

configuration

The framework uses a group_vars/all.yml file for configuration like:

collection_to_test: <namespace.collection>
collection_role_list:
  - <namespace.collection.role_name>
env_version: dev
report_host: <report_webserver>
report_dir: <webserver_root>
report_file: report.csv
use_vault_role: 
  - role_vault

extra_roles: []

required_credentials: []
bb_name: <role_name>

You might recognize some fields, but let me describe them for you anyway:

• collection_to_test
  The collection from wich the role will be run to be tested. This can NEVER be empty or the play will fail.
• env_version
  The branch/tag to checkout for the role_to_test, this gives the opportunity to test different versions of roles (later).
• report_host
  A virtual machine that is configured for ansible, to write a record of testing to. This is done only when a test is performed from 'Tower' or 'Ansible Automation Platform', so when a test is scheduled, you can trace back the job_id for this run and its result. This gives your tests tracebility for management reporting (you could upload the file data into splunk/grafana) to create dashboards.
• report_dir
  The directory on the reporting server where the file will be written.
• report_file
  The file that holds all records of the tests that have been run. Due to the current nature of the reporting, the code overwrites records on key values, if you don't want this just adapt the template accordingly and updated the exsisting forks.
• use_vault_role
  If you are using vault, just state the vault repository here, it will be cloned, run and cleaned after the run. Be sure to add the vault credential to the play.
• extra_roles
  Extra roles can be used to do configuration before testing.. For example, create a pluggable database on an empty Oracle database platform you want to test.
  The role will be cloned into the roles directory, but you will have to call them from your plays yourself. At the end of the run, these roles a cleaned up.
• required_credentials
  If you use custom credentials in your environment, you can insert the lookup variables here, so they will be checked before the run starts, this prevents failed or half-way runs.

If you are happy with the configuration, save it and push it into git, so others can use it. The base config is now complete.
You could run it now just so you know it runs without error. Before you do, review the README.md in the apache_test repository, there you will find how to run it.

writing tests

Now the real work starts...writing test plans is not just writing ansible code.
You want your tests to mean something, so don't go writing tests like:

'is that line in file x.y' if you wrote that in your ansible code, ansible will tell you that the line is there, by giving you a 'changed' or 'OK' status in your deployment play.

Testing here means knowing the product / middleware / application and what it does, so your test must be a functional one.

For example, is the deployment you opened a firewall port(80) for apache. Now you could test if the port is in the firewall config file (ERROR!) , you should test from a remote server if the port is open and if a service is listening on it, that is a functional test.

In a complex enviroment there is a lot to test, so make a layout of the environment to test and plot test scearios on this. Write the scenarios down in the README.md of the test repository, this way you and your team mates know what is tested.

Breakup your environment into functional blocks, so you don't endup testing the complete enviroment in one big play, it will never be manageble. Make a testset for each block. As your platform probably has a layered structure, such as infrastructure / OS / middleware, try to create test repositories for each layer you want to test on.
create an index of all tests, try not to repeat yourself, do not retest something that has already been tested in another layer.

For each scenario, create a list of chonological steps to accomplish the test result. After the steps are clear, you add checks in between the steps to prove the result is what it needs to be. Once you have this, you can start writing the ansible tasks to test the scenario.

Best practice

Create a yml file for each scenario, so that all tasks for a certain scenario are in one file, with a descriptive filename in relation to the README.md

Take your time planning and making this, it will save you eventually. Have your test play's reviewed ( I know, we all do that, right )
Have the results reviewed.
Report to management about the tests ( Make them happy )

File naming convention

The test framework needs no configuration for tests in scenario files, they just need to have a name that is in line with the naming convention....
What is this naming convention?

<run_stage>_<run_number>_<desciptive_name>.yml

The code will adapt to new files added to the directory / repository, it searches for files starting with prep_<num>_*.yml and test_<num>_*.yml and will include them into the main play.

• \<run_stage> the framework knows 2 stages:

• prep

This is the preparation stage, here we checkout any role that are defined in all.yml.
We can do a syntax check on the role_to_test code. We check if all the defined credentials are there You might add extra prep tasks just by adding a yaml file with a new, unique number.. Be aware, only add local tasks here, not remote, this is reserved for the test run.
So a new preparation task file might be named something like this:

prep_04_do_nothing.yml

• test

This is where the magic happens, you add as many test scenarios as you need and they will be executed in the given order of the numbers of the scenarios/files.

A typical test file is named:

test_10_check_firewall.yml

• This depicts the scenario number as you have identified them. It also depicts the order in which the framework will execute the tests.
  The numbers do not need to be sequentail, you may have some gaps to be filled in by other test files you will be adding later.
  Often scenarios are grouped and numbered as:
• 10-19 non-invasive tests
• 20-29 invasive tests (like data changes..)
• 30-39 destructive ( like killing processes/ stopping services )
• ...

• 90-99 recovery and cleanup (undo any data changes, remove files)

• \<descriptive_name>

The name of the scenario, so everyone knows what is tested in this part of the play.
For maintenance when there are a lot of yml files, it it convenient to have descriptive names.

The test code

Here is an example for a scenario, we will test a number of firewall ports on a system.
As you can see in the codeblock below, a scenario is not a full playbook, it is just a collection of tasks, in a certain order, on certain hosts.
While this is a verry simple example, it makes clear what we mean by functional testing.

---
#- name: "[AT-1O] Test if firewall ports are realy open"

- name: Check firewall ports
  ansible.builtin.wait_for:
    host: "{{ inventory_hostname }}"
    port: "{{ item }}"
    timeout: 10
  with_items:
    - 22
    - 80
    - 443
  delegate_to: "{{ another_host_in_in_env_to_check_from }}"
  when: ansible_hostname in test_hosts

- name: Setting fact to split later...
  ansible.builtin.set_fact:
    steps: |
      [AT-1O] Test that the firewall ports are really opened and something is listening
      Steps completed:
      1. Check firewall configuration for apache from remote host

- name: Acceptance test completed message
  ansible.builtin.debug:
    msg: "{{ steps.split('\n') }}"

When you run a test and the result is not what you want it to be, ensure that the test fails, so have a correct faillure task in your test playbook.
This way you can implement test-driven-developement, by first writing a test that fails and adding code to the collection that will make the test be successfull.

As you can see, there are three tasks in this yml file, the first does the actual testing, the other 2 are just for documentation purposes and give a nice output in logs. The first line documents the scenario in our case [AT-10] stands for 'Acceptance Test 10', which is one of the scenarios we defined in the test
repos README.md.
Keeping the scenario number in the filename makes it traceable, the descriptive name says it all.

The numbers also depict the order in which the tests will be conducted, so we start at 01, 02,......

Important Housekeeping

Be sure to end each test scenario with a working situation!

In a complex scenario, You might simulate failures by killing processes or stopping sevices on certain hosts.

At the end of each scenario all things stopped must be running again, so the next scenario has the correct starting point for testing.

Be sure to keep track of these!

You won't be the first to search for hours for a fault that was in the scenarios....

Run all tests

Now we have created our testscenarios and plays, we would like to run them.

Running the individual files is not efficient and, because they are not full playbooks, not possible without another playbook to include them.
This is where the power of the framework comes in, no need to edit configurations when a playbook is added, just add it to the repository and it will run.

As motioned before, there are 2 ways of running this framework: * on the commandline * in Ansible Automation Platform

Before you want to run the tests in Ansible Automation Platform, you might want to verify if it is working.

So we will start by running it manually:

• First You need to be sure you have all collections are installed that you use in the plays.
• Secondly, you need a clone of the inventory you want to use for testing ( NOT production!).
• Third, the machines to test with need to be reachable for your ansible machine, where you run this on.
• You need to have key authentication to the machines (maybe run as ansible user)

make you current directory, that of the framework.
Now run the command:

ansible-playbook main.yml -i ../inventory/inventory_file -e test_hosts=\<test_node_name\> --ask-vault-pass

This will start the framework and run the test.

If this runs without problems, you can add the project to the automation controller and create a job template for the tests. Add the credentials needed and...

Enjoy... sit back and just wait for it...

Output of the example play

When you run this exact play against a webserver that is only configured to listen on port 80, it will fail the run.
So I have a problem in my scenario and should not test port 443.

As you will see, the framework outputs much more than only the firewall task you have written, it outputs all
tasks

Vault password: 

PLAY [Prepare for testing] **********************************************************************************************

TASK [Set full repo URL from role_to_test] ******************************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
ok: [localhost] => {"ansible_facts": {"repo_name": "git@gitserver.localdomain:ivory_tower/os_rhel.git"}, "changed": false}

TASK [Create temp_group] ************************************************************************************************
changed: [web-apache1.localdomain] => {"add_host": {"groups": ["test"], "host_name": "web-apache1.localdomain", "host_vars": {"group": "test"}}, "changed": true}

TASK [Gather distribution facts] ****************************************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Set os_fact] ******************************************************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Gather date_time facts] *******************************************************************************************
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Create directory for report file if not exist] ********************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Write initial test fail to central repo file] *********************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Find prepare tasks] ***********************************************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
ok: [localhost] => {"changed": false, "examined": 13, "files": [{"atime": 1678436877.3243928, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200434, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "prep_01_get_roles_for_testing.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 613, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}, {"atime": 1678436877.3273928, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200436, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "prep_02_syntax_code.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 613, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}, {"atime": 1678436877.3303928, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200435, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "prep_03_check_credentials.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 358, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}], "matched": 3, "msg": "All paths examined", "skipped_paths": {}}

TASK [Include preparation tasks] ****************************************************************************************
skipping: [web-apache1.localdomain] => {"changed": false, "skip_reason": "Conditional result was False"}
included: /home/ansible/source/test/prep_01_get_roles_for_testing.yml for localhost => (item={'path': 'prep_01_get_roles_for_testing.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 613, 'inode': 1200434, 'dev': 64768, 'nlink': 1, 'atime': 1678436877.3243928, 'mtime': 1678436484.713323, 'ctime': 1678436522.720951, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False})
included: /home/ansible/source/test/prep_02_syntax_code.yml for localhost => (item={'path': 'prep_02_syntax_code.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 613, 'inode': 1200436, 'dev': 64768, 'nlink': 1, 'atime': 1678436877.3273928, 'mtime': 1678436484.713323, 'ctime': 1678436522.720951, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False})
included: /home/ansible/source/test/prep_03_check_credentials.yml for localhost => (item={'path': 'prep_03_check_credentials.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 358, 'inode': 1200435, 'dev': 64768, 'nlink': 1, 'atime': 1678436877.3303928, 'mtime': 1678436484.713323, 'ctime': 1678436522.720951, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False})

TASK [Set the list fact first] ******************************************************************************************
ok: [localhost] => {"ansible_facts": {"roles_to_get": ["role_vault", "os_rhel"]}, "changed": false}

TASK [Checkout role from GIT source] ************************************************************************************
ok: [localhost] => (item=role_vault) => {"_extra_role": "role_vault", "after": "e65a777debd3147f099dd7b4f3957d37e4b504d9", "ansible_loop_var": "_extra_role", "before": "e65a777debd3147f099dd7b4f3957d37e4b504d9", "changed": false, "remote_url_changed": false}
ok: [localhost] => (item=os_rhel) => {"_extra_role": "os_rhel", "after": "3019903afae3336feb0cd56f71fb8c08a882f689", "ansible_loop_var": "_extra_role", "before": "3019903afae3336feb0cd56f71fb8c08a882f689", "changed": false, "remote_url_changed": false}

TASK [Checkout role os_rhel  from git] **********************************************************************************
ok: [localhost] => {"after": "3019903afae3336feb0cd56f71fb8c08a882f689", "before": "3019903afae3336feb0cd56f71fb8c08a882f689", "changed": false, "remote_url_changed": false}

TASK [Run syntax-check on the role] *************************************************************************************
ok: [localhost] => {"changed": false, "cmd": "ansible-playbook --syntax-check syntax.yml", "delta": "0:00:00.463179", "end": "2023-03-10 09:28:41.777357", "failed_when_result": false, "msg": "", "rc": 0, "start": "2023-03-10 09:28:41.314178", "stderr": "[WARNING]: provided hosts list is empty, only localhost is available. Note that\nthe implicit localhost does not match 'all'", "stderr_lines": ["[WARNING]: provided hosts list is empty, only localhost is available. Note that", "the implicit localhost does not match 'all'"], "stdout": "Using /home/ansible/source/test/ansible.cfg as config file\n\nplaybook: syntax.yml", "stdout_lines": ["Using /home/ansible/source/test/ansible.cfg as config file", "", "playbook: syntax.yml"]}

TASK [Fail if any of the credentials are missing] ***********************************************************************

PLAY [Run the tests] ****************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************
ok: [web-apache1.localdomain]

TASK [Include vault role] ***********************************************************************************************

TASK [role_vault : Include vars] ****************************************************************************************
ok: [web-apache1.localdomain] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

TASK [Find vars file] ***************************************************************************************************
ok: [web-apache1.localdomain -> localhost] => {"changed": false, "examined": 13, "files": [{"atime": 1678436884.486314, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200441, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "vars.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 31, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}], "matched": 1, "msg": "All paths examined", "skipped_paths": {}}

TASK [Include vars for testing] *****************************************************************************************
ok: [web-apache1.localdomain] => (item={'path': 'vars.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 31, 'inode': 1200441, 'dev': 64768, 'nlink': 1, 'atime': 1678436884.486314, 'mtime': 1678436484.713323, 'ctime': 1678436522.720951, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False}) => {"_vars": {"atime": 1678436884.486314, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200441, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "vars.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 31, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}, "ansible_facts": {}, "ansible_included_var_files": ["/home/ansible/source/test/vars.yml"], "ansible_loop_var": "_vars", "changed": false}

TASK [Find test playbooks] **********************************************************************************************
ok: [web-apache1.localdomain -> localhost] => {"changed": false, "examined": 13, "files": [{"atime": 1678436884.749311, "ctime": 1678436522.720951, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 1200439, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436484.713323, "nlink": 1, "path": "test_01_include_role.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 307, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}, {"atime": 1678436884.752311, "ctime": 1678436767.3155923, "dev": 64768, "gid": 1000, "gr_name": "ansible", "inode": 52450174, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mode": "0664", "mtime": 1678436767.3075924, "nlink": 1, "path": "test_10_check_firewall.yml", "pw_name": "ansible", "rgrp": true, "roth": true, "rusr": true, "size": 637, "uid": 1000, "wgrp": true, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}], "matched": 2, "msg": "All paths examined", "skipped_paths": {}}

TASK [Include testcases] ************************************************************************************************
included: /home/ansible/source/test/test_01_include_role.yml for web-apache1.localdomain => (item={'path': 'test_01_include_role.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 307, 'inode': 1200439, 'dev': 64768, 'nlink': 1, 'atime': 1678436884.749311, 'mtime': 1678436484.713323, 'ctime': 1678436522.720951, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False})
included: /home/ansible/source/test/test_10_check_firewall.yml for web-apache1.localdomain => (item={'path': 'test_10_check_firewall.yml', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 637, 'inode': 52450174, 'dev': 64768, 'nlink': 1, 'atime': 1678436884.752311, 'mtime': 1678436767.3075924, 'ctime': 1678436767.3155923, 'gr_name': 'ansible', 'pw_name': 'ansible', 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False})

TASK [Display test name] ************************************************************************************************
ok: [web-apache1.localdomain] => {
    "msg": "Running TEST 01 - run role os_rhel from start to finish"
}

TASK [run the role to test] *********************************************************************************************

TASK [Run handlers after finishing role] ********************************************************************************

TASK [Check firewall ports] *********************************************************************************************
ok: [web-apache1.localdomain -> localhost] => (item=22) => {"ansible_loop_var": "item", "changed": false, "elapsed": 0, "item": 22, "match_groupdict": {}, "match_groups": [], "path": null, "port": 22, "search_regex": null, "state": "started"}
ok: [web-apache1.localdomain -> localhost] => (item=80) => {"ansible_loop_var": "item", "changed": false, "elapsed": 0, "item": 80, "match_groupdict": {}, "match_groups": [], "path": null, "port": 80, "search_regex": null, "state": "started"}
failed: [web-apache1.localdomain -> localhost] (item=443) => {"ansible_loop_var": "item", "changed": false, "elapsed": 10, "item": 443, "msg": "Timeout when waiting for web-apache1.localdomain:443"}

NO MORE HOSTS LEFT ******************************************************************************************************

PLAY RECAP **************************************************************************************************************
localhost                  : ok=9    changed=0    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0   
web-apache1.localdomain    : ok=9    changed=1    unreachable=0    failed=1    skipped=8    rescued=0    ignored=0   

If we remove the test for the HTTPS port, it will run nicely and you have verified the framework was implemented
correctly.
Last part of the output when run again with correct ports to test

PLAY RECAP **************************************************************************************************************
localhost                  : ok=12   changed=0    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   
web-apache1.localdomain    : ok=12   changed=0    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   

You see that there are no failed tasks in the play, so all is well.

Have fun using this.

The full code review

In this section we explain the full code of the testing template, we take the main.yml and disect it section by section and explain why its there and what it does.

On which hosts are we testing?

As any other playbook it needs to know on which hosts the play will run. We passed the hosts to test on by giving the parameter 'test_hosts' with a host or group to test on. This will be used as hosts for the play, but we add the localhost to the play as well. We do that so we can report the result of the play to the reporting host from localhost.

---
- name: Prepare for testing
  hosts: "{{ test_hosts }},localhost"
  any_errors_fatal: true
  gather_facts: false

Initialize Variables

Before we do any tests, we need to initialize some variables zo we can find things, like a gi repository to checkout.. We create a specal group 'test' with all the host of this play in it. We set the changed when_false on almost every task of the main.yml itself, so the play overview will have changes grom the role or test plays only. We also get the distribution_major_version from the targeted hosts, this is used in the report section. A fact is set with the inventory os_buildingblock_name and the major_ditrubution_version (eg. 'rhel8') fir use in the report record that will be written. This is done because you might want to test the same code against rhel8 and rhel9.

    pre_tasks:

    - name: Set full repo URL from collection_to_test
      ansible.builtin.set_fact:
        repo_name: "{{ repo_url }}{{ collection_to_test }}.git"
      when: inventory_hostname == 'localhost'

    - name: Create temp_group
      ansible.builtin.add_host:
        name: "{{ inventory_hostname }}"
        group: test
      run_once: true
      changed_when: false

    - name: Gather distribution facts
      ansible.builtin.setup:
        gather_subset: 'distribution_major_version'
      when:
        - tower_job_id is defined
        - inventory_hostname != 'localhost'

    - name: Set os_fact
      ansible.builtin.set_fact:
        bb_name: "{{ hostvars[inventory_hostname]['os']['name'] }}{{ hostvars[inventory_hostname]['ansible_distribution_major_version'] }}"
      when:
        - tower_job_id is defined
        - inventory_hostname != 'localhost'

Report we have started the play

This section writes the record stating that the test-run has started (RUNNING), this means here that it has started if the play breaks anywhere, the always section of the test block ensures that the final message reads (FAILED or SUCCESS). The report record will only be written if there is a job_id (eg. runs through tower or AWX), from the command line you can see it runnig obviously.

  - name: Gather date_time facts
      ansible.builtin.setup:
        gather_subset: 'date_time'
      when:
        - tower_job_id is defined 
        - inventory_hostname != 'localhost'

    - name: Create directory for report file if not exist
      ansible.builtin.file:
        path: "{{ report_dir }}"
        state: directory
        mode: '0755'
        recurse: true
      delegate_to: "{{ report_host }}"
      failed_when: false
      when:
        - tower_job_id is defined 
        - inventory_hostname == 'localhost'

    - name: Write header to central repo file
      ansible.builtin.lineinfile:
        path: "{{report_dir }}{{ report_file }}"
        regexp: "^Collection"
        line: "Collection,Env,Role,Result,Job_id,Timestamp,State"
        state: present
        create: true
      delegate_to: "{{ report_host }}"

    - name: Set timestamp
      ansible.builtin.set_fact: 
        timestamp: "{{ ansible_date_time.date }} {{ ansible_date_time.time }}"
      when: inventory_hostname != 'localhost'


    - name: Write initial test fail to central repo file
      ansible.builtin.lineinfile:
        path: "{{report_dir }}{{ report_file }}"
        regexp: "^{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}"
        line: "{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}, RUNNING, {{ tower_job_id }}, {{ timestamp }}, 1"
        state: present
        create: true
      changed_when: false
      delegate_to: "{{ report_host }}"
      when:
        - tower_job_id is defined
        - inventory_hostname != 'localhost'

Find and run the 'preparation' tasks

The first task is a find in the current directory on localhost for any files that match 'prep_*.yml'. By default, this should find the following file:
* prep_03_check_credentials.yml This checks if the defined custom credentials are valid, if any. Any ansible task you want to run in this stage, is automaticly added if the naming is correct. The order in wich they are added, is the numbering order of the filename.

    - name: Find prepare tasks
      ansible.builtin.find:
        paths: "."
        patterns: 'prep*.yml'
      register: _preps
      when: inventory_hostname == 'localhost'

  tasks:
    - name: Include preparation tasks
      ansible.builtin.include_tasks: "{{ _prep.path }}"
      loop_control:
        loop_var: _prep
      with_items: "{{ _preps.files | sort(attribute='path') }}"
      when:
        - _preps.files is defined
        - inventory_hostname == 'localhost'

Prepare for running the tests

In this section we include the vault role, so we have all passwords we need to run the tests. Additionally, we include any vars_files we can find so the vars needed for tests are availlable. Last we do a search on the current directory to find any test plays we have to run.

- name: Run the tests
  any_errors_fatal: true
  hosts: "{{ [test_hosts] | flatten | join(',') }},localhost"

  pre_tasks:

    - name: Include vault role
      ansible.builtin.include_role:
        name: "{{ use_vault_role | join('') }}"
      when: use_vault_role is defined

    - name: Find vars file
      ansible.builtin.find:
        paths: "."
        patterns: 'vars*.yml'
      register: _varsfiles
      delegate_to: localhost

    - name: Include vars for testing
      ansible.builtin.include_vars:
        file: "{{ _vars.path }}"
      loop_control:
        loop_var: _vars
      with_items: "{{ _varsfiles.files }}"
      when:
        - _varsfiles.files is defined
        - _varsfiles.matched > 0

    - name: Find test playbooks
      ansible.builtin.find:
        paths: "."
        patterns: 'test*.yml'
      register: _tests
      delegate_to: localhost

Finally run the tests

So now we include all test files we found in the previous section and include them into the play to run them in sequence. We run this as a block, so we can always do a cleanup at the end of the play (eg. remove checked out roles).

  tasks:

    - name: Run as block 
      block:
        - name: Include testcases
          ansible.builtin.include_tasks: "{{ _test_case.path }}"
          loop_control:
            loop_var: _test_case
          with_items: "{{ _tests.files | sort(attribute='path') }}"
          when:
            - _tests.files is defined
            - inventory_hostname != 'localhost'

Cleanup any mess

As the tests have run, we clean all roles we checked out, even if the tests failed. And here we overwrite the running state with failed, so any failure is always registered.

      always:
        - name: Remove vault role
          ansible.builtin.file:
            path: "roles/{{ use_vault_role | join('') }}"
            state: absent
          changed_when: false
          when:
            - use_vault_role is defined
            - inventory_hostname == 'localhost'

        - name: Remove extra roles
          ansible.builtin.file:
            path: "roles/{{ _extr_role }}"
            state: absent
          changed_when: false
          loop_control: 
            loop_var: _extr_role
          with_items: "{{ extra_roles }}"
          when: inventory_hostname == 'localhost'

        - name: "Write test result to central repo file"
          ansible.builtin.lineinfile:
            path: "{{ report_dir }}{{ report_file }}"
            regexp: "^{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}"
            line: "{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}, FAILED, {{ tower_job_id }}, {{ timestamp }}, 2"
            state: present
          changed_when: false
          delegate_to: "{{ report_host }}"
          when:
            - tower_job_id is defined
            - inventory_hostname != 'localhost'

Write final record, if we got here

When run through tower or AWX, update the report by replacing the line thet said failed with a line that says 'SUCCESS', so your report shows the correct result. When you read the file you will see the correct result, but beware...tools like splunk read every update and could show you the wrong results...You will have to tweak the report for this behavure.

    - name: "Write test result to central repo file"
      ansible.builtin.lineinfile:
        path: "{{ report_dir }}{{ report_file }}"
        regexp: "^{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}"
        line: "{{ collection_to_test }}, {{ env_version }}, {{ hostvars['localhost']['bb_name'] }}, SUCCESS, {{ tower_job_id }}, {{ timestamp }}, 0"
        state: present
      changed_when: false
      delegate_to: "{{ report_host }}"
      when:
        - tower_job_id is defined
        - inventory_hostname != 'localhost'

This is all there is...

Happy testing (and maybe happy managers)

Automate the hell out of IT...

Back to Homepage