Docs on Panic! At The Terminal

Fail2ban SSH Protection on RHEL9: Installation and Configuration Guide

Mon, 28 Jul 2025 00:00:00 +0000

Fail2ban SSH Protection on RHEL9: Installation and Configuration Guide #

Fail2ban is an essential security tool that helps protect your Linux server from brute force attacks and automated bot spam targeting SSH and other services. By monitoring log files and automatically banning IP addresses that show signs of malicious activity, fail2ban acts as an intrusion prevention system that significantly enhances your server’s security posture.

Installation #

# Install fail2ban and firewalld integration
dnf install fail2ban fail2ban-firewalld

# Enable and start the service
systemctl enable fail2ban --now

Basic Configuration #

1. Create Local Configuration Files #

Never edit the default configuration files directly. Instead, create local copies that won’t be overwritten during updates:

In-Place Upgrade from RHEL 9 to RHEL 10 Using Leapp

Thu, 24 Jul 2025 12:00:00 +0200

In-Place Upgrade from RHEL 9 to RHEL 10 Using Leapp #

This guide walks through the process of upgrading from RHEL 9 to RHEL 10 in place, without requiring a complete system reinstallation.

Current System Status #

Here’s what my lab’s test VM looks like:

[user@test ~]$ hostnamectl
 Static hostname: test.home.arpa
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: dec9d730df4f4e629ba20d02aed02f03
         Boot ID: 67e690ef7fbe45edb112a82dcdf72a97
  Virtualization: kvm
Operating System: Red Hat Enterprise Linux 9.6 (Plow)
     CPE OS Name: cpe:/o:redhat:enterprise_linux:9::baseos
          Kernel: Linux 5.14.0-570.28.1.el9_6.x86_64
    Architecture: x86-64
 Hardware Vendor: Red Hat
  Hardware Model: KVM
Firmware Version: 1.16.3-4.el9

We can see we’re currently running RHEL 9.6 on a KVM virtual machine.

SELinux Troubleshooting

Wed, 23 Jul 2025 10:00:00 +0200

SELinux Troubleshooting #

Here’s a little guide on how to find if SELinux is blocking something and how to add an exception to the policy.

SELinux Modes #

SELinux operates in three modes:

# Check SELinux status and mode
sestatus

Understanding the Three Modes #

Enforcing: SELinux policy is enforced
Permissive: SELinux policy violations are logged but not blocked
Disabled: SELinux is completely disabled

Check if SELinux is the problem #

# Temporarily set to permissive mode and test
setenforce 0
# Test your application
# If it works now, SELinux was blocking it
setenforce 1

Troubleshooting Workflow #

When an application fails and you are certain that SELinux is blocking it, e.g. you turned SELinux off or into permissive mode and the application worked, you can do the following:

Resolving Git Submodule Tracking Issues

Sat, 19 Jul 2025 21:30:00 +0300

Resolving Git Submodule Tracking Issues #

The Problem #

I recently encountered a frustrating Git issue with my Hugo blog setup. I had installed the Hugo Book theme as a Git submodule for my static site, but for some reason there were a bunch of changed files showing up in the theme’s directory even though I had never run git submodule update or made any intentional changes to it. At the time, I didn’t really understand how submodules work, so I tried adding it to .gitignore thinking that would solve the problem.

How to Restore a Broken KVM VM from Backup

Fri, 18 Jul 2025 20:00:00 +0300

How to Restore a Broken KVM VM from Backup #

Sometimes things go wrong with virtual machines — maybe a filesystem corruption or a bad update. When that happens, restoring from a backup is your best friend.

Here’s how I restored my broken KVM VM disk image using weekly backups stored on a NAS share.

The situation #

I have a VM called runner.home.arpa running on KVM, and its disk got corrupted. The VM disk images live at /var/lib/libvirt/images/, and my backups are stored on a NAS mounted at /mnt/backups/runner.home.arpa/.

Docker Instance Upgrades with Ansible

Wed, 16 Jul 2025 18:23:20 +0200

Docker Instance Upgrades with Ansible #

This guide shows how to automate Docker container upgrades using Ansible playbooks instead of manual updates. By automating this process, you ensure consistency, reduce errors, and save time when updating multiple Docker instances across your infrastructure.

Infrastructure Overview #

My homelab includes these Docker-based services:

Nginx Proxy Manager (proxy.home.arpa) - Reverse proxy with SSL
Healthchecks (health.home.arpa) - Monitoring service
Home Assistant (hass.home.arpa) - Home automation platform
Bitwarden (bit.home.arpa) - Password manager

Ansible Playbook Structure #

Host Configuration #

The playbook targets specific hosts with Docker services:

Infrastructure as Code with Ansible Automation Platform

Sun, 13 Jul 2025 15:00:00 +0300

Infrastructure as Code for Ansible Automation Platform Setup #

Introduction #

Setting up Ansible Automation Platform (AAP) manually through the web interface is tedious and highly prone to errors.

I’ve written an Ansible playbook that completely automates the setup of my AAP environment, from credentials and projects to job templates and workflow orchestration. So in case I ever need to rebuild the environment from scratch, all I would need to do is just add the project where that playbook is stored, and add a single template by hand and run it. I’d do it that way because I don’t like running ansible playbooks from the CLI, I always, always use AAP!

SSH Hardening and Automation User Setup with Ansible

Sat, 12 Jul 2025 12:00:00 +0300

SSH Hardening and Automation User Setup with Ansible #

Here’s a little post about how I do SSH hardening for my RHEL9 homelab and how I ensure that the Ansible automation user is properly set. The playbook stems from an incident I had in Red Hat Insights where it was reported that I had an SSH configuration that allowed legacy ciphers. It was also adviced to create a crypto policy that disables weak algorithms.

Automating KVM Backups with Ansible

Sat, 12 Jul 2025 10:00:00 +0300

Automating KVM Homelab Backups with Ansible #

When you’re running a dozen virtual machines in your homelab, manual backups quickly become a nightmare.

In this post, I’ll walk you through my Ansible-based backup strategy for my KVM homelab. It automatically backs up all VMs by shutting them down gracefully, copying their disk images and configurations to a NAS, and bringing them back online.

Backup Strategy #

My backup strategy uses Ansible to orchestrate the entire process:

Automated Network Monitoring: Adding Servers to LibreNMS with Ansible

Mon, 07 Jul 2025 14:00:00 +0300

Automated Network Monitoring: Adding Servers to LibreNMS with Ansible #

Adding servers to LibreNMS by hand is tedious, and should be done by automation. In this post, I’ll show you how I’ve automated the entire process of configuring SNMP and adding servers to LibreNMS using Ansible.

The Workflow #

Basically what the playbook does is:

Install and configure SNMP
Set up necessary firewall rules
Add the server to LibreNMS
Add it to the correct device group.

The Playbook #

Step 1: Installing SNMP Components #

- name: Ensure snmp is installed
  ansible.builtin.dnf:
    name:
      - net-snmp
      - net-snmp-utils
    state: present

The net-snmp package is needed for the SNMP daemon.

Optimizing KVM Virtual Machines with Tuned Profiles

Mon, 07 Jul 2025 12:00:00 +0300

Optimizing KVM Virtual Machines with Tuned Profiles #

The tuned service on Red Hat-based systems provides pre-configured performance profiles that can significantly improve your VM performance with minimal effort.

In this post, I’ll show you how to optimize your KVM VMs using tuned profiles and automate the entire process with Ansible.

The Playbook #

Since I manage dozens of VMs in my homelab, doing this manually would be tedious. Instead, I use this Ansible playbook to apply tuned optimization to all my VMs:

Automating RHEL Server Updates with Ansible

Sat, 05 Jul 2025 16:00:00 +0300

Automating RHEL Server Updates with Ansible #

Introduction #

I hate updating my servers manually so I’ve set up this playbook to run updates. This was probably the first playbook I ever wrote for my home lab, and it’s been running automatically for years now on a weekly schedule every Friday night through AAP (Ansible Automation Platform).

This guide shows you how to automate RHEL (and other yum/dnf based distros like Fedora, CentOS etc.) server updates using Ansible, including proper reboot handling.

Setting up BIND9 for Internal DNS on RHEL9

Thu, 03 Jul 2025 20:00:00 +0300

Setting up BIND9 for Internal DNS on RHEL9 #

This guide covers setting up BIND9/named for internal reverse/forward DNS resolution on a RHEL9 server. Unlike public authoritative DNS servers, internal DNS servers provide recursive resolution for your internal network and handle local domain queries.

All IP addresses, network ranges, and hostnames in this guide are examples. Replace them with your actual values.

For the entirety of the guide we’ll be running every single command as root.

Setting up BIND9 for Public DNS on RHEL9

Thu, 03 Jul 2025 18:00:00 +0300

Setting up BIND9 for Public DNS on RHEL9 #

This guide covers setting up BIND9/named as an authoritative public DNS server on RHEL9. Unlike internal DNS servers that provide recursive resolution, public authoritative DNS servers are responsible for answering queries about domains you own and control.

This setup includes security hardening measures such as proper logging and zone transfer restrictions to protect your DNS infrastructure from abuse.

All IP addresses, domain names, and server configurations in this guide are examples. Replace them with your actual values.

Installing Red Hat Identity Management (IPA) without DNS on RHEL9

Thu, 03 Jul 2025 16:00:00 +0300

Installing Red Hat Identity Management (IPA) without DNS on RHEL9 #

Red Hat Identity Management (IdM) provides centralized authentication, authorization, and account information by storing data about users, groups, hosts, and other objects necessary to manage the security aspects of a network of computers. This guide covers installing IdM/IPA server without the integrated DNS service on RHEL9.

This guide demonstrates setting up a high-availability IPA deployment with two servers:

Automated KVM VM Provisioning with Ansible and OSBuild on RHEL9

Thu, 03 Jul 2025 14:00:00 +0300

Automated KVM VM Provisioning with Ansible and OSBuild on RHEL9 #

Introduction #

When I started looking into automating my homelab VM provisioning, I was surprised by the lack of examples combining Ansible with OSBuild for KVM environments. Not many tutorials focus on KVM, so I wanted something that used Red Hat’s tooling - as I run a RHEL homelab.

I used to provision my homelab virtual machines by hand and eventually I got tired of doing it since I like to tinker around a lot and constantly add new VMs. So, I decided to automate the process using the combination of Ansible and OSBuild.

Setting Up Pre-commit Hooks

Thu, 03 Jul 2025 12:00:00 +0300

Setting Up Pre-commit hooks #

What are they? #

Pre-commit hooks are automated scripts that run before each commit, helping you catch issues early and maintain consistent code quality.

Steps #

Installing and configuring pre-commit
Setting up hooks for markdown files

Installation #

First, install pre-commit using pip:

pip install pre-commit

Configuration #

Create a .pre-commit-config.yaml file in your repository root:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files
      - id: check-case-conflict
      - id: check-merge-conflict

What Each Hook Does #

Basic Hooks #

trailing-whitespace: Removes trailing whitespace from lines
end-of-file-fixer: Ensures files end with a newline
check-yaml: Validates YAML syntax
check-added-large-files: Prevents accidentally committing large files
check-case-conflict: Catches case conflicts that would cause issues on case-insensitive filesystems
check-merge-conflict: Detects merge conflict markers

For a complete list of all available hooks, check out the pre-commit-hooks repository.

How to setup Ansible Vault

Thu, 03 Jul 2025 10:00:00 +0300

How to setup Ansible Vault #

Here’s a little guide on how I setup Ansible Vault for my Ansible playbook repository. It’s surprisingly simple and now all of my secrets are encrypted.

Setting Up Ansible Vault #

1. Create the Directory Structure #

First, create the standard Ansible directory structure for group variables:

mkdir -p group_vars/all

2. Create Your Vault File #

Create a vault file to store your encrypted credentials:

Use ansible-lint with Vault Files

Mon, 30 Jun 2025 18:46:13 +0300

Use ansible-lint with Vault Files #

Why I wrote this post #

I decided to write this post because I struggled to find clear, practical examples of how to make ansible-lint work with Ansible Vault files in CI/CD environments. While searching for solutions, I found a GitHub discussion where someone was asking the exact same question I had.

The official ansible-lint documentation mentions that decrypting Ansible Vault in CI is possible, but frustratingly, it doesn’t provide any actual examples of how to implement it. After some trial and error, I figured out a working solution that I want to share.

Enable EPEL Repository on RHEL9

Tue, 03 Jun 2025 17:53:34 +0300

Enable EPEL Repository on RHEL 9 #

The EPEL repository provides packages that are not included in the standard RHEL repositories, such as htop and vim for example.

Prerequisites #

Before installing EPEL, you need to enable the CodeReady Builder repository, which provides dependencies for many EPEL packages.

Installation Steps #

Enable codeready builder repository #

subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install epel-release #

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Verification #

After installation, you can verify that EPEL is enabled by listing available repositories:

How to Make a New Post on Hugo

Tue, 25 Mar 2025 18:32:04 +0200

Creating New Posts in Hugo #

Hugo is a fast and flexible static site generator written in Go. Creating new posts is straightforward once you understand the basic workflow. This guide is mostly for myself since I don’t always remember how the process goes.

Step-by-Step Process #

1. Navigate to your Hugo site directory #

/var/lib/snapd/void/blog

2. Create a new post using Hugo CLI #

hugo new content content/docs/hugo-post.md

3. Edit the post content #

vim content/docs/hugo-post.md

Edit the front matter (title, date, draft status) and add your content in Markdown.

Extending LVM Partitions

Mon, 04 Nov 2024 18:58:46 +0200

Extending LVM Partitions #

This guide walks you through extending LVM partitions when you’ve added new disk space to your system.

LVM Structure #

LVM operates with three main components:

Physical Volumes (PV): The actual disk partitions
Volume Groups (VG): Collections of physical volumes
Logical Volumes (LV): The volumes you mount and use

Common Scenario #

You’ve added disk space to a virtual machine and need to extend the root filesystem. This is a common requirement in virtualized environments where storage needs grow over time.

Linux Cheat Sheet - System Administration Commands

Mon, 04 Nov 2024 17:58:47 +0200

Linux System Administration Cheat Sheet #

This is a curated list of Linux commands for myself that I’ve kept written down over the span of five years into my career in IT. I use this often when I can’t remember spesific commands when in a rush.

User Management & SSH #

#add ansible user to server
useradd ansible
passwd ansible
usermod -aG wheel ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
chown ansible:ansible /home/ansible/.ssh
"ssh-rsa..." | sudo tee /home/ansible/.ssh/authorized_keys
chmod 600 /home/ansible/.ssh/authorized_keys
chown ansible:ansible /home/ansible/.ssh/authorized_keys

#show octal permissions of file
stat authorized_keys

Ansible #

#run ansible playbook in vscode as ansible user
eval "$(ssh-agent)"
ssh-add ansible_id_rsa
ansible-playbook playbook.yml --user ansible --private-key .ssh/ansible_id_rsa --inventory inventory.ini

Git #

#git cheat sheet
git init
git status
git add .
git commit -m "Commit message"
git remote add origin <url>
git push -u origin <branch>
git rm -r --cached public/

#show what's modified in detail
git diff
git diff themes/hugo-book
git diff --submodule=diff themes/hugo-book

#submodule operations
git submodule status
git rm --cached themes/hugo-book

File Operations #

#show rsync progress
rsync -avh --progress /var/lib/libvirt/images/mc.home.arpa.qcow2 /mnt/backups/

#use inverse grep to exclude things
df -Th | grep -v "tmpfs|squashfs"

#find string in files
grep -i 'keeper' -R /etc/apt

#find string in compressed files
zgrep -i "connected" *.log.gz

#pipe command output as argument
rpm -qa | grep htop | xargs rpm -e

#show available disk space nicely formatted
du -hs /* | sort -hr | head

#create a file filled with zeroes (1024M)
dd if=/dev/zero of=/tmp/file.txt count=1024 bs=1024

#grep multiple terms
rpm -qa | grep -Ei 'fuse-libs|libcurl|python36'

#display filetree of the root folder and another folder (with a depth of 1 and hidden files)
tree . themes/hugo-book/ -L 1 -a

#download multiple files
wget -i urls.txt -P files/ --progress=bar

SELinux #

#check and fix what SELinux is blocking
tail /var/log/audit/audit.log
grep "1675516978.657:437" /var/log/audit/audit.log | audit2why
grep "nginx" /var/log/audit/audit.log | audit2allow -M nginx
ls nginx.pp
semodule -i nginx.pp

#find SELinux errors (requires policycoreutils-python-utils)
audit2why < /var/log/audit/audit.log

Network Configuration #

#nmcli add backup vlan to NIC team
nmcli connection add type vlan con-name backup dev team0 id 2186 ip4 10.215.159.196/29  ipv4.never-default yes +ipv4.routes "81.175.254.0/24 10.215.159.193"  ipv6.method ignore
#nmcli add another NIC to regular VM (no bond)
mcli con add con-name "grpc" ifname ens162 type ethernet ip4 172.20.13.132/26 ipv4.method manual ipv6.method ignore ipv4.never-default yes +ipv4.routes "172.20.13.128/26 172.20.13.129"
#nmcli list devices
nmcli device status

#add proxy to session
export http_proxy=http://proxy.home.arpa:8080
export https_proxy=http://wproxy.home.arpa:8080
export PATH="$HOME/.local/bin:$PATH"

#test web proxy connection
curl -I https://google.com/ -x proxy.home.arpa:8080
# located in /etc/yum.conf
# proxy=http://wproxy.dnaip.fi:8080
curl -L -O https://github.com/healthchecks/healthchecks/archive/refs/heads/master.zip -x proxy.home.arpa:8080

System Monitoring #

#check what's using CPU
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu| head

#show processes with PID for /proc
ps -xj

#display process tree
ps -e --forest

#list of commands you use most often
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

#view journalctl for spesific program
journalctl -eu named -f -l

#show what ports host is listening to
netstat -tulpn | grep LISTEN

Package Management #

#rhel subs-manager commands
subscription-manager status
subscription-manager list --consumed
subscription-manager refresh
subscription-manager attach --auto
subscription-manager status
dnf repolist -v

#rhel satellite rejoin
subscription-manager status
subscription-manager release
subscription-manager repos --list-enabled
subscription-manager unregister
subscription-manager register --org='asd' --activationkey='activation-key' --force --release=<7Server,8,9>
subscription-manager attach --auto
subscription-manager repos --enable=rhel-6-server-optional-rpms

#check which packages were installed from yum history
yum history
yum history info 10

#show which package provides the command 'netstat'
yum whatprovides netstat

Virtualization #

#get IP addresses of KVM VMs
virsh list | awk '{print $2}' | xargs

#Connect to KVM with virt-manager
virt-manager --connect qemu+ssh://root@kvm.home.arpa/system

#create a checkpoint (snapshot) of a VM
virsh snapshot-create-as test.home.arpa checkpoint-name --description "Checkpoint before update"

#list all checkpoints for a VM
virsh snapshot-list test.home.arpa

#revert VM to a checkpoint
virsh snapshot-revert test.home.arpa checkpoint-name

#delete a checkpoint
virsh snapshot-delete test.home.arpa checkpoint-name

#create checkpoint with memory state (live snapshot)
virsh snapshot-create-as test.home.arpa checkpoint-name --description "Live checkpoint" --memspec file=/var/lib/libvirt/qemu/save/test.home.arpa.save

Containers #

#podman cheat sheet
podman build ./Dockerfile -t yt-dlp:latest
podman images
podman rmi 8121a9f5303b
podman run --name youtube -dt yt-dlp:latest
podman ps

#kubectl
kubectl get nodes
kubectl apply -f nginx.yaml
kubectl get deploy -A
kubectl delete deploy nginx -n default
kubectl get services
kubectl logs nginx-deployment-78c9ff5d49-cmbdn

#docker install
dnf -y install dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

#docker cheat sheet
docker ps -a
docker exec -ti awx_task /bin/bash
docker stats
docker-compose up -d
docker-compose down -d
docker-compose logs nginx
docker-compose config
docker images
docker system prune --all

#update docker compose containers
docker compose pull
docker compose up -d --force-recreate

Troubleshooting #

#check which DNS server offers results
dig 97.94.87.in-addr.arpa NS +short

#list all DNS records using dig (requires bind-utils)
dig +nocmd example.com any +multiline +noall +answer

#TXT record lookup via dig
dig -t txt lenovo.com

#test UDP port 53
nc -vz -u 1.1.1.1 53
#test TCP port 22
nc -zv 10.10.10.100 22

#tcpdump examples
tcpdump -i ens160 dst port 5544 and host 10.10.10.150 -vvv
tcpdump tcp -X -i ens192 dst port 514 and host 10.10.10.10 -w /tmp/mycap.pcap -vvv
tcpdump tcp -X -i ens192 dst 10.10.10.10 -w /tmp/mycap.pcap -vvv

#ping flooding with max packet size
ping -f -l 65536 -s 1500 10.10.10.9

#ngrep show port's traffic
ngrep -d any port 25

#simulate syslog messages (UDP) via netcat
echo 'test' | nc -u 10.10.10.50 5555

#scan all TCP and UDP ports
nmap -sU -sT -p0-65535 10.0.0.1

Fun & Novelty #

#novelty
ssh chat.shazow.net
telnet mapscii.me
fortune | cowsay
cat greeting.txt | boxes -d diamonds -a c

Mon, 01 Jan 0001 00:00:00 +0000

Automating PowerShell Code Formatting with Pre-Commit Hooks #

The Problem #

If you’re working on a PowerShell project with multiple developers, you’ve probably experienced the pain of inconsistent code formatting. One developer uses tabs, another uses spaces. Braces are sometimes on the same line, sometimes on a new line. It’s a mess, and code reviews become debates about formatting instead of logic.

VS Code’s PowerShell extension has a great “Format Document” feature, but let’s be honest—developers forget to use it.