Linux

In-Place Upgrade from RHEL 9 to RHEL 10 Using Leapp

July 24, 2025
Infrastructure, Tutorials
Rhel, Upgrade, Leapp, System-Administration, Linux

In-Place Upgrade from RHEL 9 to RHEL 10 Using Leapp #

This guide walks through the process of upgrading from RHEL 9 to RHEL 10 in place, without requiring a complete system reinstallation.

Current System Status #

Here’s what my lab’s test VM looks like:

[user@test ~]$ hostnamectl
 Static hostname: test.home.arpa
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: dec9d730df4f4e629ba20d02aed02f03
         Boot ID: 67e690ef7fbe45edb112a82dcdf72a97
  Virtualization: kvm
Operating System: Red Hat Enterprise Linux 9.6 (Plow)
     CPE OS Name: cpe:/o:redhat:enterprise_linux:9::baseos
          Kernel: Linux 5.14.0-570.28.1.el9_6.x86_64
    Architecture: x86-64
 Hardware Vendor: Red Hat
  Hardware Model: KVM
Firmware Version: 1.16.3-4.el9

We can see we’re currently running RHEL 9.6 on a KVM virtual machine.

SELinux Troubleshooting

July 23, 2025
Security, Tutorials, System Administration
Selinux, Security, Linux, Rhel, Centos, Troubleshooting, System-Administration

SELinux Troubleshooting #

Here’s a little guide on how to find if SELinux is blocking something and how to add an exception to the policy.

SELinux Modes #

SELinux operates in three modes:

# Check SELinux status and mode
sestatus

Understanding the Three Modes #

  1. Enforcing: SELinux policy is enforced
  2. Permissive: SELinux policy violations are logged but not blocked
  3. Disabled: SELinux is completely disabled

Check if SELinux is the problem # 

# Temporarily set to permissive mode and test
setenforce 0
# Test your application
# If it works now, SELinux was blocking it
setenforce 1

Troubleshooting Workflow #

When an application fails and you are certain that SELinux is blocking it, e.g. you turned SELinux off or into permissive mode and the application worked, you can do the following:

How to Restore a Broken KVM VM from Backup

July 18, 2025
DevOps, Tutorials
Kvm, Backup, Virtualization, Linux, Libvirt

How to Restore a Broken KVM VM from Backup #

Sometimes things go wrong with virtual machines — maybe a filesystem corruption or a bad update. When that happens, restoring from a backup is your best friend.

Here’s how I restored my broken KVM VM disk image using weekly backups stored on a NAS share.

The situation #

I have a VM called runner.home.arpa running on KVM, and its disk got corrupted. The VM disk images live at /var/lib/libvirt/images/, and my backups are stored on a NAS mounted at /mnt/backups/runner.home.arpa/.

Enable EPEL Repository on RHEL9

June 3, 2025
Infrastructure, Tutorials
Rhel9, Epel, Repository, Packages, Linux

Enable EPEL Repository on RHEL 9 #

The EPEL repository provides packages that are not included in the standard RHEL repositories, such as htop and vim for example.

Prerequisites #

Before installing EPEL, you need to enable the CodeReady Builder repository, which provides dependencies for many EPEL packages.

Installation Steps #

Enable codeready builder repository # 

subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms

Install epel-release # 

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Verification #

After installation, you can verify that EPEL is enabled by listing available repositories:

Extending LVM Partitions

November 4, 2024
Infrastructure, Tutorials
Linux, Lvm, Partitions, Storage, System-Administration

Extending LVM Partitions #

This guide walks you through extending LVM partitions when you’ve added new disk space to your system.

LVM Structure #

LVM operates with three main components:

  • Physical Volumes (PV): The actual disk partitions
  • Volume Groups (VG): Collections of physical volumes
  • Logical Volumes (LV): The volumes you mount and use

Common Scenario #

You’ve added disk space to a virtual machine and need to extend the root filesystem. This is a common requirement in virtualized environments where storage needs grow over time.

Linux Cheat Sheet - System Administration Commands

November 4, 2024
Reference, Tutorials
Linux, System-Administration, Devops, Commands, Bash, Terminal

Linux System Administration Cheat Sheet #

This is a curated list of Linux commands for myself that I’ve kept written down over the span of five years into my career in IT. I use this often when I can’t remember spesific commands when in a rush.

Quick Reference Categories #

  • User Management & SSH: Setting up users and SSH access
  • Ansible: Automation and configuration management
  • Git: Version control operations
  • File Operations: rsync, grep, and file management
  • SELinux: Security context management
  • Network Configuration: nmcli and network setup
  • System Monitoring: Process and resource monitoring
  • Package Management: RHEL/CentOS package operations
  • Virtualization: KVM/virsh commands
  • Containers: Docker and Podman operations
  • Troubleshooting: Network, DNS, and system diagnostics

User Management & SSH # 

#add ansible user to server
useradd ansible
passwd ansible
usermod -aG wheel ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
chown ansible:ansible /home/ansible/.ssh
"ssh-rsa..." | sudo tee /home/ansible/.ssh/authorized_keys
chmod 600 /home/ansible/.ssh/authorized_keys
chown ansible:ansible /home/ansible/.ssh/authorized_keys

#show octal permissions of file
stat authorized_keys

Ansible # 

#run ansible playbook in vscode as ansible user
eval "$(ssh-agent)"
ssh-add ansible_id_rsa
ansible-playbook playbook.yml --user ansible --private-key .ssh/ansible_id_rsa --inventory inventory.ini

Git # 

#git cheat sheet
git init
git status
git add .
git commit -m "Commit message"
git remote add origin <url>
git push -u origin <branch>
git rm -r --cached public/

#show what's modified in detail
git diff
git diff themes/hugo-book
git diff --submodule=diff themes/hugo-book

#submodule operations
git submodule status
git rm --cached themes/hugo-book

#store entered HTTPS credentials
git config --global credential.helper store

File Operations # 

#show rsync progress
rsync -avh --progress /var/lib/libvirt/images/mc.home.arpa.qcow2 /mnt/backups/

#use inverse grep to exclude things
df -Th | grep -v "tmpfs|squashfs"

#find string in files
grep -i 'keeper' -R /etc/apt

#find string in compressed files
zgrep -i "connected" *.log.gz

#pipe command output as argument
rpm -qa | grep htop | xargs rpm -e

#show available disk space nicely formatted
du -hs /* | sort -hr | head

#create a file filled with zeroes (1024M)
dd if=/dev/zero of=/tmp/file.txt count=1024 bs=1024

#grep multiple terms
rpm -qa | grep -Ei 'fuse-libs|libcurl|python36'

#display filetree of the root folder and another folder (with a depth of 1 and hidden files)
tree . themes/hugo-book/ -L 1 -a

#download multiple files
wget -i urls.txt -P files/ --progress=bar

#upload a file via SSH
scp rhel-10.2-x86_64-boot.iso antti@kvm:/home/antti/.

SELinux # 

#check and fix what SELinux is blocking
tail /var/log/audit/audit.log
grep "1675516978.657:437" /var/log/audit/audit.log | audit2why
grep "nginx" /var/log/audit/audit.log | audit2allow -M nginx
ls nginx.pp
semodule -i nginx.pp

#find SELinux errors (requires policycoreutils-python-utils)
audit2why < /var/log/audit/audit.log

Network Configuration # 

#nmcli add backup vlan to NIC team
nmcli connection add type vlan con-name backup dev team0 id 2186 ip4 10.215.159.196/29  ipv4.never-default yes +ipv4.routes "81.175.254.0/24 10.215.159.193"  ipv6.method ignore
#nmcli add another NIC to regular VM (no bond)
mcli con add con-name "grpc" ifname ens162 type ethernet ip4 172.20.13.132/26 ipv4.method manual ipv6.method ignore ipv4.never-default yes +ipv4.routes "172.20.13.128/26 172.20.13.129"
#nmcli list devices
nmcli device status

#add proxy to session
export http_proxy=http://proxy.home.arpa:8080
export https_proxy=http://wproxy.home.arpa:8080
export PATH="$HOME/.local/bin:$PATH"

#test web proxy connection
curl -I https://google.com/ -x proxy.home.arpa:8080
# located in /etc/yum.conf
# proxy=http://wproxy.dnaip.fi:8080
curl -L -O https://github.com/healthchecks/healthchecks/archive/refs/heads/master.zip -x proxy.home.arpa:8080

System Monitoring # 

#check what's using CPU
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu| head

#show processes with PID for /proc
ps -xj

#display process tree
ps -e --forest

#list of commands you use most often
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

#view journalctl for spesific program
journalctl -eu named -f -l

#osbuild composer-cli commands
cat rhel10-test.home.arpa.toml
composer-cli blueprints push rhel10-test.home.arpa.toml
composer-cli compose start rhel10 qcow2
journalctl -u osbuild-composer -f
ll
composer-cli blueprints list
composer-cli blueprints delete rhel10
composer-cli compose list
composer-cli compose delete 2538042c-72af-4c4e-a024-280dde7dd039
composer-cli compose delete 576d9142-c72e-467d-9f74-dcf12953a045

#show what ports host is listening to
netstat -tulpn | grep LISTEN

Package Management # 

#rhel subs-manager commands
subscription-manager status
subscription-manager list --consumed
subscription-manager refresh
subscription-manager attach --auto
subscription-manager status
dnf repolist -v

#rhel satellite rejoin
subscription-manager status
subscription-manager release
subscription-manager repos --list-enabled
subscription-manager unregister
subscription-manager register --org='asd' --activationkey='activation-key' --force --release=<7Server,8,9>
subscription-manager attach --auto
subscription-manager repos --enable=rhel-6-server-optional-rpms

#kernel live patching
dnf -y install kpatch kpatch-dnf
dnf kpatch auto

#remove unused kernels safely
yum remove --oldinstallonly

#check which packages were installed from yum history
yum history
yum history info 10

#show which package provides the command 'netstat'
yum whatprovides netstat

Virtualization # 

#get IP addresses of KVM VMs
virsh list | awk '{print $2}' | xargs

#Connect to KVM with virt-manager
virt-manager --connect qemu+ssh://root@kvm.home.arpa/system

#create a checkpoint (snapshot) of a VM
virsh snapshot-create-as test.home.arpa checkpoint-name --description "Checkpoint before update"

#list all checkpoints for a VM
virsh snapshot-list test.home.arpa

#revert VM to a checkpoint
virsh snapshot-revert test.home.arpa checkpoint-name

#delete a checkpoint
virsh snapshot-delete test.home.arpa checkpoint-name

#create checkpoint with memory state (live snapshot)
virsh snapshot-create-as test.home.arpa checkpoint-name --description "Live checkpoint" --memspec file=/var/lib/libvirt/qemu/save/test.home.arpa.save

Containers # 

#podman cheat sheet
podman build ./Dockerfile -t yt-dlp:latest
podman images
podman rmi 8121a9f5303b
podman run --name youtube -dt yt-dlp:latest
podman ps

#kubectl
kubectl get nodes
kubectl apply -f nginx.yaml
kubectl get deploy -A
kubectl delete deploy nginx -n default
kubectl get services
kubectl logs nginx-deployment-78c9ff5d49-cmbdn

#docker install
dnf -y install dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

#docker cheat sheet
docker ps -a
docker exec -ti awx_task /bin/bash
docker stats
docker-compose up -d
docker-compose down -d
docker-compose logs nginx
docker-compose config
docker images
docker system prune --all

#update docker compose containers
docker compose pull
docker compose up -d --force-recreate

Troubleshooting # 

#check which DNS server offers results
dig 97.94.87.in-addr.arpa NS +short

#list all DNS records using dig (requires bind-utils)
dig +nocmd example.com any +multiline +noall +answer

#TXT record lookup via dig
dig -t txt lenovo.com

#test UDP port 53
nc -vz -u 1.1.1.1 53
#test TCP port 22
nc -zv 10.10.10.100 22

#tcpdump examples
tcpdump -i ens160 dst port 5544 and host 10.10.10.150 -vvv
tcpdump tcp -X -i ens192 dst port 514 and host 10.10.10.10 -w /tmp/mycap.pcap -vvv
tcpdump tcp -X -i ens192 dst 10.10.10.10 -w /tmp/mycap.pcap -vvv

#ping flooding with max packet size
ping -f -l 65536 -s 1500 10.10.10.9

#ngrep show port's traffic
ngrep -d any port 25

#simulate syslog messages (UDP) via netcat
echo 'test' | nc -u 10.10.10.50 5555

#scan all TCP and UDP ports
nmap -sU -sT -p0-65535 10.0.0.1

Fun & Novelty # 

#novelty
ssh chat.shazow.net
telnet mapscii.me
fortune | cowsay
cat greeting.txt | boxes -d diamonds -a c