Example projects
Here we collect some projects that could serve as examples to build your own projects from.
Some will be rather simple, others rather complex.
I wil try to add some form of scoring to each project in the range from 1 to 5 (1 being simple).
Check_oauth2_token
Complexity score: 2
This project will check the lifetime of a OAuth2 API token from the AAP gateway and will send an alert to a configured alertmanager instance for further handling.
When an Oauth2 API token is created in rhaap it has a default lifetime of 365 days, this cannot be overwritten or updated. So a token created
in rhaap will expire and become obsolete.
When using these tokens actively (and you will), you will forget they are there and they will expire!
This may cause unwanted outage, because playbooks won't run, collections can't download and more possible errors, depending on where you are
using these tokens.
In this document, I will only show the most important (for me), that is the token that I create during the configuration as code base pipeline run. This creates a token that is actively used in the configuration of automation platform (Ansible Galaxy/Automation Hub API Token) used by the organizations. This token is also used in the distributed ansible.cfg that is downloadable from the webserver. So there can be many dependencies to this token.
So what I did to check and alert this:
- I created an alertmanager instance in my network
- Alertmanager can send alerts to a telegram bot account
- Created a playbook to check the token and send the alert
- Added this task as schedule to the configuration as code of rhaap
After these steps, I get an alert if the token expires within 30 days on my phone (or any other alertmanager target) wich tells me to run the
base configuration as code to create a new version of the token and upload a new version of ansible.cfg to the webserver.
I could even create an EDA rulebook to run the configuration as code for me (self healing).
The playbook/project
This check consists of just this one simple playbook with a few variables defined into the playbook itself.
It will need a few values that must be passed to the playbook, using a credential or as extra variables (as these are sensitive variables passing them as a credential is preferred/mandatory).
Variables:
alertmanager_url: "http://alertmanager.[your-domain]:9093"
The url where your instance of the alertmanager can be reached to post the alert to.
token_description: "ansible_cfg_token"
The desciption of the token to find in rhaap (this must be unique!).
expire_alert_days: 30
The minimal number of days the token has to be valid for, is it smaller or equal, an alert will be sent.
The play will further need a credential to be able to login to the API and list the token. This must at least be an
account with sufficient rights to read the token specified (preferably an admin account).
The credential needs to have the following fields:
- aap_hostname The fqdn of the gateway host
- aap_username The username for the admin account
- aap_password The password for the admin account
The playbook:
---
- name: Check and renew ansible.cfg API token
hosts: "{{ instance | default('localhost') }}"
connection: local
gather_subset:
- min
vars:
# Change this to your Alertmanager instance URL
alertmanager_url: "http://alertmanager.[your-domain]:9093"
token_description: "ansible_cfg_token"
expire_alert_days: 30
tasks:
# Generate a hub token to use for the galaxy credentials in controller
# This token is not echoed, so for use in ansible.cfg, you must use a new account
# and create a token for that account after configuration.
- name: Gateway | Read the current token
ansible.builtin.uri:
url: "https://{{ aap_hostname }}/api/gateway/v1/tokens/?description={{ token_description }}"
user: "{{ aap_username }}"
password: "{{ aap_password }}"
force_basic_auth: true
method: GET
body_format: json
validate_certs: false
register: _ansible_cfg_token
no_log: true
- name: Set time facts for calculating remaining lifetime
ansible.builtin.set_fact:
expires_epoch: "{{ (_ansible_cfg_token.json.results[0].expires | to_datetime('%Y-%m-%dT%H:%M:%S.%fZ')).strftime('%s') | int }}"
current_epoch: "{{ ansible_date_time.epoch | int }}"
- name: Calculate token lifetime remaining in days
ansible.builtin.set_fact:
token_life: "{{ ((expires_epoch - current_epoch) / 86400) | int }}"
- name: Post firing alert to Alertmanager API v2
ansible.builtin.uri:
url: "{{ alertmanager_url }}/api/v2/alerts"
method: POST
status_code: 200
body_format: json
headers:
Content-Type: "application/json"
body:
- labels:
alertname: "Token-Ansible-Expiry-Notice"
severity: "critical"
instance: "{{ aap_hostname }}"
job: "rhaap_check_token"
annotations:
summary: "Ansible.cfg token expires in {{ token_life }} days."
description: "Please run the 'base config' configuration as code."
generatorURL: "https://{{ aap_hostname }}"
startsAt: "{{ now(utc=true).isoformat() }}Z"
register: alert_response
when: token_life <= expire_alert_days
- name: Print API response status
ansible.builtin.debug:
msg: "Alert sent successfully! Status code: {{ alert_response.status }}"
when: alert_response.status is defined
In configuration as code
Project: Add this into the file: group_vars/dev/controller_projects.yaml
controller_projects_dev:
- name: prj_check_rhaap_ansible_token
description: Check the lifetime for token
organization: [your org]
scm_type: git
scm_url: [your path to the project]/rhaap-check-api-token.git
scm_credential: gitlab
scm_branch: master
scm_clean: false
scm_delete_on_update: false
scm_update_on_launch: true
scm_update_cache_timeout: 0
allow_override: false
timeout: 0
Job_template: Add this to the file: group_vars/dev/controller_templates.yaml
controller_templates_dev:
- name: check_rhaap_ansible_token
description:
organization: MGT
project: prj_check_rhaap_ansible_token
inventory: Default_inventory
playbook: main.yml
job_type: run
fact_caching_enabled: false
credentials:
- aap_credential
concurrent_jobs_enabled: false
ask_scm_branch_on_launch: false
ask_tags_on_launch: false
ask_verbosity_on_launch: false
ask_variables_on_launch: false
execution_environment: ee_default 1-5
survey_enabled: false
survey_spec: {}
Schedule: Add this to the file: group_vars/dev/controller_schedules.yaml
controller_schedules_dev:
- name: Check ansible token
description: Check the ansible token lifetime
unified_job_template: check_rhaap_ansible_token
rrule: "DTSTART;TZID=Europe/Amsterdam:20260812T090000 RRULE:FREQ=DAILY;INTERVAL=1;BYDAY=MO,TU,WE,TH,FR,SA,SU"
When the configuration as code for this organization is run, the project, job_template and schedule are added to the controller.
If your alertmanager is setup correctly, you will get an alert when the token is about to expire.
Check website for dead-links
Complexity score: 3
Nothing is more annoying than clicking on a link in a site and get a 404 message, it will eventually cause
visitors to avoid that site. If this is your site, you don't want to miss these visitors.
This is why you check your site.
This project reads the sitemap.xml from any given site and checks all pages found in this sitemap for dead links, domain errors inside the sitemap, will stop the check instantly.
As the check will run over the normal internet, it will add some load to your webserver.
This playbook will execute the following actions:
- Validate the passed variables.
- It will try to download the sitemap.xml
- Read the sitemap and extract the domains used in the sitemap (there should only be one!).
- If the sitemap.xml is for a different domain, exit with error.
- Extract the links to all pages from sitemap.
- load all pages in the list and check all links.
- Check the result code and relevant messages.
- Show the messages.
- Maybe send an error to alertmanager
The playbook/project
This check consists of just this one simple playbook with a few variables passed through extra_variables on the commandline or through AAP.
It will need a few values that must be passed to the playbook as extra variables.
Variables:
site_to_check: www.ansible-guru.nl The website to run the test against.
protocol: https The protocol to use, should never be anything else than 'https'
alertmanager_url The url where the alertmanager can be found to send alerts to.
The playbook main.yml
---
- name: Check all links in Sitemap for broken Links
hosts: localhost
connection: local
gather_facts: false
vars:
sitemap_url: "{{ protocol }}://{{ site_to_check }}/sitemap.xml"
# Standard ignore-paterns (sane defaults for WordPress)
ignore_defaults: "xmlrpc.php|//fonts.googleapis.com/|//fonts.gstatic.com/|//secure.gravatar.com/|//akismet.com/|//wordpress.org/|//s.w.org/|//gmpg.org/"
alertmanager_url: "http://alertmanager.url:port"
tasks:
- name: Validate site_to_check variable
ansible.builtin.fail:
msg: "ERROR: No site_to_check passed. Use -e 'site_to_check=...' on execution."
when: site_to_check is not defined or site_to_check | length == 0
- name: Download sitemap XML content
ansible.builtin.uri:
url: "{{ sitemap_url }}"
return_content: true
timeout: 10
register: sitemap_response
- name: Configure base domain
ansible.builtin.set_fact:
sitemap_domain: "{{ site_to_check }}"
- name: Extract all unique domains from the <loc> tags in sitemap
ansible.builtin.set_fact:
found_domains: "{{ sitemap_response.content | regex_findall('(?<=<loc>)https?://([^/]+)') | unique }}"
- name: Check for domain-typo's (mismatch in sitemap URL)
ansible.builtin.fail:
msg: |
ERROR: A different domain is found in sitemap! There is probably a typo.
Expected domain: {{ sitemap_domain }}
Found domains: {{ found_domains | difference([sitemap_domain]) | join(', ') }}
when: (found_domains | difference([sitemap_domain])) | length > 0
- name: Combine all ignore-paterns
ansible.builtin.set_fact:
final_ignore: "{{ ignore_defaults }}"
- name: Extract URL's from sitemap <loc> tags
ansible.builtin.set_fact:
extracted_urls: "{{ sitemap_response.content | regex_findall('(?<=<loc>)https?://[^<]+') }}"
- name: Create a temporary file for the URL-list
ansible.builtin.tempfile:
state: file
suffix: temp_urls
register: temp_file
- name: Write the extracted URL's to the temporary file
ansible.builtin.copy:
content: "{{ extracted_urls | join('\n') }}"
dest: "{{ temp_file.path }}"
mode: '0600'
- name: Execute wget spider-scan on the URL's # noqa: command-instead-of-module
ansible.builtin.command: >
wget --input-file={{ temp_file.path }}
--reject-regex "{{ final_ignore }}"
--spider
--recursive
--no-directories
--no-verbose
--span-hosts
--level=1
--timeout=5
--tries=3
-D {{ site_to_check }}
register: wget_result
failed_when: wget_result.rc not in [0, 4, 5, 7, 8]
changed_when: false
- name: Delete the temporary file
ansible.builtin.file:
path: "{{ temp_file.path }}"
state: absent
- name: Analyse the results and show errors
when: wget_result.rc != 0
block:
- name: Add specific wget error message for each rc
ansible.builtin.set_fact:
_check_result: >-
{{(
{
4: 'Network failure (wget rc 4): check network/firewall/DNS.',
5: 'SSL verification failure (wget rc 5): check certificates and TLS-settings.',
7: 'Protocol error (wget rc 7): Unsupported protocol.',
8: 'Server error response (wget rc 8): Error code (4xx/5xx).'
}[wget_result.rc]
)
}}
when: wget_result.rc in [4, 5, 7, 8]
- name: Filter the broken links and DNS-errors
ansible.builtin.set_fact:
broken_links: "{{ wget_result.stderr | regex_findall('(.+)\n.*broken link!') | map('regex_replace', ':$', '') | list }}"
dns_errors: "{{ wget_result.stderr | regex_findall('wget: (unable to resolve.+)') | list }}"
- name: Create the error message
ansible.builtin.set_fact:
error_message: |
Found broken links on {{ sitemap_url }}:
{% if broken_links %}
Broken URL's:
{{ broken_links | to_nice_yaml }}
{% endif %}
{% if dns_errors %}
DNS Errors:
{{ dns_errors | to_nice_yaml }}
{% endif %}
Wget error: {{ _check_result }}
when: wget_result.rc in [4, 5, 7, 8]
- name: Post firing alert to Alertmanager API v2
ansible.builtin.uri:
url: "{{ alertmanager_url }}/api/v2/alerts"
method: POST
status_code: 200
body_format: json
headers:
Content-Type: "application/json"
body:
- labels:
alertname: "Website-Link-Errors"
severity: "critical"
instance: "{{ site_to_check }}"
job: "rhaap_check_website"
annotations:
summary: "Problems found on {{ site_to_check }}."
description: "{{ error_message }}"
generatorURL: "https://{{ ansible_hostname }}"
when: (wget_result.rc in [4, 5, 7, 8]) and alertmanager_url is defined
- name: Toon gevonden fouten in de terminal
ansible.builtin.debug:
msg: |
Problems fount on: {{ site_to_check }}
{{ error_message }}
when: wget_result.rc in [4, 5, 7, 8]
- name: Show if there is no error
ansible.builtin.debug:
msg: "No Errors or dead links found in: {{ site_to_check }}"
when: wget_result.rc == 0
In configuration as code
Project: Add this into the file: group_vars/dev/controller_projects.yaml
controller_projects_dev:
- name: prj_check_rhaap_website
description: Check the website for broken links
organization: [your org]
scm_type: git
scm_url: [your path to the project]/rhaap-check-website.git
scm_credential: gitlab
scm_branch: master
scm_clean: false
scm_delete_on_update: false
scm_update_on_launch: true
scm_update_cache_timeout: 0
allow_override: false
timeout: 0
Job_template: Add this to the file: group_vars/dev/controller_templates.yaml
controller_templates_dev:
- name: check_rhaap_website
description:
organization: [your_org]
project: prj_check_rhaap_website
inventory: Default_inventory
playbook: main.yml
job_type: run
fact_caching_enabled: false
credentials:
- aap_credential
concurrent_jobs_enabled: false
ask_scm_branch_on_launch: false
ask_tags_on_launch: false
ask_verbosity_on_launch: false
ask_variables_on_launch: false
execution_environment: ee_default 1-5
survey_enabled: false
survey_spec: {}
Schedule: Add this to the file: group_vars/dev/controller_schedules.yaml
controller_schedules_dev:
- name: Check website from site.xml
description: Check the website
unified_job_template: check_rhaap_website
rrule: "DTSTART;TZID=Europe/Amsterdam:20260812T090000 RRULE:FREQ=DAILY;INTERVAL=1;BYDAY=MO,TU,WE,TH,FR,SA,SU"
extra_data:
site_to_check: 'www.example.org'
protocol: https
When the configuration as code for this organization is run, the project, job_template and schedule are added to the controller.
Create your own mkdocs site
Complexity score: 3
Create your own site using gitlab -> pipeline -> webserver automation. Updates to the gitlab repository will automagicly be
published to the site.
What will we need for this to work:
- A gitlab repository
- A gitlab runner
- mkdocs + advanced-seo plugin
- A pipeline script
- A webserver using nginx or apache
Installing gitlab and a gitlab-runner are described elsewhere on this site ( Lab on Proxmox ), or use the gitlab online documentation.
This project replaces the whole site on update in your repository.
As your website grows, you don't want to write html code to get your text published and formatted
(nobody does that anymore), we want to write the text and not worry about the way it goes to the site.
We might want to do some simple formatting (in my case), but not much more.
Be aware that you will be needing multiple (virtual)machines for this, a least 3:
- gitlab repository server
- gitlab-runner
- nginx webserver
This is how I did this:
Webserver
I have a webserver configured with nginx serving the site from this directory: /var/www/site/html If you are using proxmox, just create a new nginx lxc container from a template. This should be easy enough.
Create a virtual server configuration, using the example below and be sure to change at least the server name to the virtual machine name that you created.
This is the configuration file for that virtualserver:
server {
listen 80;
server_name site.nginx_server;
access_log /var/log/nginx/site_access.log;
error_log /var/log/nginx/site_error.log;
root /var/www/site/html;
index index.html;
server_tokens off;
charset utf-8;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
This is an internal server only, so no ssl configured, the exposure to the outside world is done through a proxy server
that handles the certitficate and ssl for me.
The most interesting in this configuration file is the location of the files that form your website.
Installing and configuring a nginx server should not be a problem, using the documentation.
Gitlab and runner
For gitlab and the gitlab runner, you will need 2 virtual machines on your proxmox installation. These machines should meet the following
requirements:
gitlab:
- 8Gb memory
- 4 cores
- 30Gb disk minimum
gitlab-runner
- 1Gb memory (depends on tasks)
- 2 cores
- 20Gb disk
The installation of gitlab and a gitlab runner are described elsewhere.
The gitlab repository
The source for the website is hosted inside a gitlab repository, this is what the structure is like:
.
├── .gitlab_ci.yml
├── dev.ini
├── assets
│ └── my_image.png
├── docs
│ ├── index.md
│ ├── sub_dir
│ │ └── index.md
│ └── second
│ └── index.md
└── mkdocs.yml
Mkdocs expects the whole site to be under the docs directory, here you start your site using index.md.
The site navigation is built with the file mkdocs.yml:
site_name: 'Place your site name here'
site_url: 'website_url'
site_author: Your name
copyright: ©your name or organization
nav:
- Index: index.md
- Sub_dir:
- index: sub_dir/index.md
- Second:
- index: second/index.md
theme:
name: readthedocs
plugins:
- advanced-seo:
url_base: "your website url" # Highly Recommended for canonical/OG links
# Feature Toggles
use_canonical_url: false
use_open_graph: true
use_twitter_cards: false
add_schema_org_json_ld: true
# Default Values (Fallbacks)
og_type: "website"
og_image: "assets/my_image.png"
og_locale: "en_US"
# Twitter Defaults
# twitter_card_type: "summary_large_image"
# twitter_site: "@yourproject"
# twitter_creator: "@yourhandle"
# Advanced Integration
support_document_dates: true # Auto-detect dates from other plugins
This basic configuration will render your example site, tweak this as you whish to personaliza your site.
using the mkdocs documentation.
The pipeline
The file .gitlab_ci.yml in the repository has the following content, it will create the site from the .md files
and copy the resulting website to the webserver, overwriting the previous version.
This will not delete files deleted in your repository, the replace the whole site, you must delete all files in the
target directory on that server.
This pipeline uses a variable for the webuser_password. This variable is defined on the repository, so it is not exposed.
# List of pipeline stages
image: my_image_registry:5000/mkdocs-img:1.0
stages:
- deploy_site
deploy_site:
tags:
- shared
stage: deploy_site
rules:
- if: '$CI_COMMIT_BRANCH == "master"
&& $CI_PIPELINE_SOURCE == "push"'
script:
- echo "From pipeline - Start deploy on '$CI_COMMIT_BRANCH'"
- mkdocs build
- sshpass -p ${webuser_password} scp -o StrictHostKeyChecking=no -r site/* user@webserver:/var/www/site/html/
The image referred to in this pipeline, can be build using the following configuration:
Dockerfile:
FROM quay.io/rockylinux/rockylinux:9.6-minimal
USER root
ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
RUN microdnf -y install python3.12 python3.12-devel python3-pip openssh-clients fuse3-devel fuse-overlayfs && \
microdnf clean all && \
pip3 install ansible-core mkdocs mkdocs-advanced-seo
RUN /usr/bin/chmod 777 -R /opt/
The scipt to build and upload the image to a registry:
version=1.3
docker build -t mkdocs-img .
docker login https://my_image_registry
docker tag mkdocs-img my_image_registry/mkdocs-img:${version}
docker push my_image_registry/mkdocs-img:${version}
docker rmi mkdocs-img
Your might need to use -u and -p options to the docker login command (I removed those from this config script).
Using the pipeline as above, the website will be updated on every update into the 'master' branch.