Administering Linux

Linux Administration

Comprehensive Linux system administration for managing servers, deploying applications, and troubleshooting production issues in modern cloud-native environments.

Purpose

This skill teaches fundamental and intermediate Linux administration for DevOps engineers, SREs, backend developers, and platform engineers. Focus on systemd-based distributions (Ubuntu, RHEL, Debian, Fedora) covering service management, process monitoring, filesystem operations, user administration, performance tuning, log analysis, and network configuration.

Modern infrastructure requires solid Linux fundamentals even with containerization. Container hosts run Linux, Kubernetes nodes need optimization, and troubleshooting production issues requires understanding systemd, processes, and logs.

Not Covered:

• Advanced networking (BGP, OSPF) - see network-architecture skill
• Deep security hardening (compliance, pentesting) - see security-hardening skill
• Configuration management at scale (Ansible, Puppet) - see configuration-management skill
• Container orchestration - see kubernetes-operations skill

When to Use This Skill

Use when deploying custom applications, troubleshooting slow systems, investigating service failures, optimizing workloads, managing users, configuring SSH, monitoring disk space, scheduling tasks, diagnosing network issues, or applying performance tuning.

Quick Start

Essential Commands

Service Management:

systemctl start nginx              # Start service
systemctl stop nginx               # Stop service
systemctl restart nginx            # Restart service
systemctl status nginx             # Check status
systemctl enable nginx             # Enable at boot
journalctl -u nginx -f             # Follow service logs

Process Monitoring:

top                                # Interactive process monitor
htop                               # Enhanced process monitor
ps aux | grep process_name         # Find specific process
kill -15 PID                       # Graceful shutdown (SIGTERM)
kill -9 PID                        # Force kill (SIGKILL)

Disk Usage:

df -h                              # Filesystem usage
du -sh /path/to/dir                # Directory size
ncdu /path                         # Interactive disk analyzer

Log Analysis:

journalctl -f                      # Follow all logs
journalctl -u service -f           # Follow service logs
journalctl --since "1 hour ago"    # Filter by time
journalctl -p err                  # Show errors only

User Management:

useradd -m -s /bin/bash username   # Create user with home dir
passwd username                    # Set password
usermod -aG sudo username          # Add to sudo group
userdel -r username                # Delete user and home dir

Core Concepts

Systemd Architecture

Systemd is the standard init system and service manager. Systemd units define services, timers, targets, and other system resources.

Unit File Locations (priority order):

• /etc/systemd/system/ - Custom units (highest priority)
• /run/systemd/system/ - Runtime units (transient)
• /lib/systemd/system/ - System-provided units (don't modify)

Key Unit Types: .service (services), .timer (scheduled tasks), .target (unit groups), .socket (socket-activated)

Essential systemctl Commands:

systemctl daemon-reload            # Reload unit files after changes
systemctl list-units --type=service
systemctl list-timers              # Show all timers
systemctl cat nginx.service        # Show unit file content
systemctl edit nginx.service       # Create override file

For detailed systemd reference, see references/systemd-guide.md.

Process Management

Processes are running programs with unique PIDs. Understanding process states, signals, and resource usage is essential for troubleshooting.

Process States: R (running), S (sleeping), D (uninterruptible sleep/I/O), Z (zombie), T (stopped)

Common Signals: SIGTERM (15) graceful, SIGKILL (9) force, SIGHUP (1) reload config

Process Priority:

nice -n 10 command                 # Start with lower priority
renice -n 5 -p PID                 # Change priority of running process

Filesystem Hierarchy

Essential directories: / (root), /etc/ (config), /var/ (variable data), /opt/ (optional software), /usr/ (user programs), /home/ (user directories), /tmp/ (temporary), /boot/ (boot loader)

Filesystem Types Quick Reference:

• ext4 - General purpose (default)
• XFS - Large files, databases (RHEL default)
• Btrfs - Snapshots, copy-on-write
• ZFS - Enterprise, data integrity, NAS

For filesystem management details including LVM and RAID, see references/filesystem-management.md.

Package Management

Ubuntu/Debian (apt):

apt update && apt upgrade          # Update system
apt install package                # Install package
apt remove package                 # Remove package
apt search keyword                 # Search packages

RHEL/CentOS/Fedora (dnf):

dnf update                         # Update all packages
dnf install package                # Install package
dnf remove package                 # Remove package
dnf search keyword                 # Search packages

Use native package managers for system services; snap/flatpak for desktop apps and cross-distro compatibility.

Decision Frameworks

Troubleshooting Performance Issues

Investigation Workflow:

1. Identify bottleneck:

   top                             # Quick overview
   uptime                          # Load averages

2. CPU Issues (usage >80%):

   top                             # Press Shift+P to sort by CPU
   ps aux --sort=-%cpu | head

3. Memory Issues (swap used):

   free -h                         # Memory usage
   top                             # Press Shift+M to sort by memory

4. Disk I/O Issues (high wa%):

   iostat -x 1                     # Disk statistics
   iotop                           # I/O by process

5. Network Issues:

   ss -tunap                       # Active connections
   iftop                           # Bandwidth monitor

For comprehensive troubleshooting, see references/troubleshooting-guide.md.

Filesystem Selection

Quick Decision:

• Default/General → ext4
• Database servers → XFS
• Large file storage → XFS or ZFS
• NAS/File server → ZFS
• Need snapshots → Btrfs or ZFS

Common Workflows

Creating a Systemd Service

Step 1: Create unit file

sudo nano /etc/systemd/system/myapp.service

Step 2: Unit file content

[Unit]
Description=My Web Application
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment="PORT=8080"
ExecStart=/opt/myapp/bin/server
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
StandardOutput=journal

# Security hardening
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target

Step 3: Deploy and start

sudo useradd -r -s /bin/false myapp
sudo mkdir -p /var/lib/myapp
sudo chown myapp:myapp /var/lib/myapp
sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service
sudo systemctl status myapp.service

For complete examples, see examples/systemd-units/.

Systemd Timer (Cron Replacement)

Create service and timer units for scheduled tasks. Timer unit specifies OnCalendar= schedule and Persistent=true for missed jobs. Service unit has Type=oneshot. See examples/systemd-units/backup.timer and backup.service for complete examples.

SSH Hardening

Generate SSH key:

ssh-keygen -t ed25519 -C "admin@example.com"
ssh-copy-id admin@server

Harden sshd_config:

sudo nano /etc/ssh/sshd_config

Key settings:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
AllowUsers admin deploy
X11Forwarding no
Port 2222                          # Optional

Apply changes:

sudo sshd -t                       # Test
sudo systemctl restart sshd        # Apply (keep backup session!)

For complete SSH configuration, see examples/configs/sshd_config.hardened and references/security-hardening.md.

Performance Tuning

Configure sysctl parameters in /etc/sysctl.d/99-custom.conf for network tuning (tcp buffers, BBR congestion control), memory management (swappiness, cache pressure), and file descriptors. Set ulimits in /etc/security/limits.conf for nofile and nproc. Configure I/O schedulers and CPU governors. For comprehensive tuning, see references/performance-tuning.md and examples/configs/ for templates.

Log Investigation

Use systemctl status myapp and journalctl -u myapp to investigate issues. Filter logs by time --since, severity -p err, or search patterns with grep. Correlate with system metrics using top, df -h, free -h. Check for OOM kills with journalctl -k | grep -i oom. For detailed workflows, see references/troubleshooting-guide.md.

Essential Commands

Interface Management:

ip addr show                       # Show all interfaces
ip link set eth0 up                # Bring interface up
ip addr add 192.168.1.100/24 dev eth0

Routing:

ip route show                      # Show routing table
ip route get 8.8.8.8               # Show route to IP
ip route add 10.0.0.0/24 via 192.168.1.1

Socket Statistics:

ss -tunap                          # All TCP/UDP connections
ss -tlnp                           # Listening TCP ports
ss -ulnp                           # Listening UDP ports
ss -tnp state established          # Established connections

Firewall Configuration

Ubuntu (ufw):

sudo ufw status
sudo ufw enable
sudo ufw allow 22/tcp              # Allow SSH
sudo ufw allow 80/tcp              # Allow HTTP
sudo ufw allow from 192.168.1.0/24 # Allow from subnet
sudo ufw default deny incoming

RHEL/CentOS (firewalld):

firewall-cmd --state
firewall-cmd --list-all
firewall-cmd --add-service=http --permanent
firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --reload

For complete network configuration including netplan, NetworkManager, and DNS, see references/network-configuration.md.

Scheduled Tasks

Cron Syntax

crontab -e                         # Edit user crontab

# Format: minute hour day month weekday command
0 2 * * * /usr/local/bin/backup.sh              # Daily at 2:00 AM
*/5 * * * * /usr/local/bin/check-health.sh      # Every 5 minutes
0 3 * * 0 /usr/local/bin/weekly-cleanup.sh      # Weekly Sunday 3 AM
@reboot /usr/local/bin/startup-script.sh        # Run at boot

Systemd Timer Calendar Syntax

OnCalendar=daily                   # Every day at midnight
OnCalendar=*-*-* 02:00:00          # Daily at 2:00 AM
OnCalendar=Mon *-*-* 09:00:00      # Every Monday at 9 AM
OnCalendar=*-*-01 00:00:00         # 1st of every month
OnBootSec=5min                     # 5 minutes after boot

Essential Tools

Process Monitoring

• top, htop - Real-time process monitor
• ps - Report process status
• pgrep/pkill - Find/kill by name

Log Analysis

• journalctl - Query systemd journal
• grep - Search text patterns
• tail -f - Follow log files

Disk Management

• df - Disk space usage
• du - Directory space usage
• lsblk - List block devices
• ncdu - Interactive disk analyzer

Network Tools

• ip - Network configuration
• ss - Socket statistics
• ping - Test connectivity
• dig/nslookup - DNS queries
• tcpdump - Packet capture

System Monitoring

• Netdata - Real-time web dashboard
• Prometheus + Grafana - Metrics collection
• ELK Stack - Centralized logging

Integration with Other Skills

Kubernetes Operations

Linux administration is the foundation for Kubernetes node management. Node optimization (sysctl tuning), kubelet as systemd service, container logs via journald, cgroups for resource limits.

Example:

# /etc/sysctl.d/99-kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

For Kubernetes-specific operations, see kubernetes-operations skill.

Configuration Management

Linux administration provides knowledge; configuration management automates it. Ansible playbooks automate systemd service creation and system tuning.

For automation at scale, see configuration-management skill.

Security Hardening

This skill covers SSH and firewall basics. For advanced security (MFA, certificates, CIS benchmarks, compliance), see security-hardening skill.

CI/CD Pipelines

CI/CD pipelines deploy to Linux servers using these skills. Uses systemctl for deployment and journalctl for monitoring.

For deployment automation, see building-ci-pipelines skill.

Reference Materials

Detailed Guides

• `references/systemd-guide.md` - Comprehensive systemd reference (unit files, dependencies, targets)
• `references/performance-tuning.md` - Complete sysctl, ulimits, cgroups, I/O scheduler guide
• `references/filesystem-management.md` - LVM, RAID, filesystem types, permissions
• `references/network-configuration.md` - ip/ss commands, netplan, NetworkManager, DNS, firewall
• `references/security-hardening.md` - SSH hardening, firewall, SELinux/AppArmor basics
• `references/troubleshooting-guide.md` - Common issues, diagnostic workflows, solutions

Examples

• `examples/systemd-units/` - Service, timer, and target unit files
• `examples/scripts/` - Backup, health check, and maintenance scripts
• `examples/configs/` - sshd_config, sysctl.conf, logrotate examples

Distribution-Specific Notes

Ubuntu/Debian

Package Manager: apt, Network: netplan, Firewall: ufw, Repositories: /etc/apt/sources.list

RHEL/CentOS/Fedora

Package Manager: dnf, Network: NetworkManager, Firewall: firewalld, Repositories: /etc/yum.repos.d/, SELinux enabled by default

Arch Linux

Package Manager: pacman, Network: NetworkManager, Rolling release, AUR for community packages

Additional Resources

Official Documentation:

• systemd: https://systemd.io/
• Linux kernel: https://kernel.org/doc/

Related Skills:

• kubernetes-operations - Container orchestration on Linux
• configuration-management - Automate Linux admin at scale
• security-hardening - Advanced security and compliance
• building-ci-pipelines - Deploy via CI/CD
• performance-engineering - Deep performance analysis

Filesystem Management Guide

Complete reference for managing Linux filesystems, LVM, RAID, permissions, and storage.

Table of Contents

1. Filesystem Types 2. Logical Volume Manager (LVM) 3. RAID Configuration 4. Mounting and fstab 5. Permissions and ACLs 6. Disk Usage Management

Filesystem Types

Comparison

Creating Filesystems

ext4:

sudo mkfs.ext4 /dev/sdb1
sudo mkfs.ext4 -L mylabel /dev/sdb1     # With label

XFS:

sudo mkfs.xfs /dev/sdb1
sudo mkfs.xfs -L mylabel /dev/sdb1

Btrfs:

sudo mkfs.btrfs /dev/sdb1
sudo mkfs.btrfs -L mylabel /dev/sdb1

Logical Volume Manager (LVM)

LVM Concepts

Three layers: 1. Physical Volumes (PV) - Raw disks/partitions 2. Volume Groups (VG) - Pool of PVs 3. Logical Volumes (LV) - Virtual partitions from VG

Creating LVM Setup

Step 1: Create Physical Volume

sudo pvcreate /dev/sdb
sudo pvcreate /dev/sdc

# View PVs
sudo pvdisplay
sudo pvs

Step 2: Create Volume Group

sudo vgcreate vg_data /dev/sdb /dev/sdc

# View VGs
sudo vgdisplay
sudo vgs

Step 3: Create Logical Volume

# Fixed size
sudo lvcreate -L 10G -n lv_data vg_data

# Percentage of VG
sudo lvcreate -l 100%FREE -n lv_data vg_data

# View LVs
sudo lvdisplay
sudo lvs

Step 4: Create Filesystem

sudo mkfs.ext4 /dev/vg_data/lv_data

Step 5: Mount

sudo mkdir /mnt/data
sudo mount /dev/vg_data/lv_data /mnt/data

Extending LVM Volumes

Extend LV:

# Add 5GB
sudo lvextend -L +5G /dev/vg_data/lv_data

# Use all free space
sudo lvextend -l +100%FREE /dev/vg_data/lv_data

Resize Filesystem:

# ext4
sudo resize2fs /dev/vg_data/lv_data

# XFS
sudo xfs_growfs /mnt/data

# Btrfs
sudo btrfs filesystem resize max /mnt/data

One-step extend and resize:

sudo lvextend -L +5G --resizefs /dev/vg_data/lv_data

Reducing LVM Volumes

WARNING: Can cause data loss if not careful!

For ext4 only (XFS cannot shrink):

# Unmount first
sudo umount /mnt/data

# Check filesystem
sudo e2fsck -f /dev/vg_data/lv_data

# Resize filesystem first
sudo resize2fs /dev/vg_data/lv_data 8G

# Then reduce LV
sudo lvreduce -L 8G /dev/vg_data/lv_data

# Remount
sudo mount /dev/vg_data/lv_data /mnt/data

LVM Snapshots

# Create snapshot (10% of original size for changes)
sudo lvcreate -L 1G -s -n lv_data_snap /dev/vg_data/lv_data

# Mount snapshot
sudo mkdir /mnt/snapshot
sudo mount /dev/vg_data/lv_data_snap /mnt/snapshot

# Restore from snapshot
sudo lvconvert --merge /dev/vg_data/lv_data_snap

# Remove snapshot
sudo lvremove /dev/vg_data/lv_data_snap

RAID Configuration

RAID Levels

Creating Software RAID with mdadm

Install mdadm:

sudo apt install mdadm              # Ubuntu/Debian
sudo dnf install mdadm              # RHEL/Fedora

Create RAID 1 (mirroring):

sudo mdadm --create /dev/md0 \
    --level=1 \
    --raid-devices=2 \
    /dev/sdb /dev/sdc

# Monitor creation
watch cat /proc/mdstat

Create RAID 5:

sudo mdadm --create /dev/md0 \
    --level=5 \
    --raid-devices=3 \
    /dev/sdb /dev/sdc /dev/sdd

Create filesystem and mount:

sudo mkfs.ext4 /dev/md0
sudo mkdir /mnt/raid
sudo mount /dev/md0 /mnt/raid

Save RAID configuration:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

Check RAID status:

cat /proc/mdstat
sudo mdadm --detail /dev/md0

Add spare disk:

sudo mdadm --add /dev/md0 /dev/sde

Remove failed disk:

sudo mdadm --fail /dev/md0 /dev/sdb
sudo mdadm --remove /dev/md0 /dev/sdb
# Replace disk
sudo mdadm --add /dev/md0 /dev/sdf

Mounting and fstab

Manual Mounting

# Mount filesystem
sudo mount /dev/sdb1 /mnt/data

# Mount with options
sudo mount -o rw,noexec,nosuid /dev/sdb1 /mnt/data

# Mount by label
sudo mount LABEL=mylabel /mnt/data

# Mount by UUID
sudo mount UUID=xxxx-xxxx /mnt/data

# Remount with different options
sudo mount -o remount,ro /mnt/data

# Unmount
sudo umount /mnt/data

/etc/fstab Configuration

Format:

<device> <mount_point> <type> <options> <dump> <pass>

Examples:

# /etc/fstab

# By device
/dev/sdb1  /mnt/data  ext4  defaults  0  2

# By UUID (recommended)
UUID=xxx-xxx  /mnt/data  ext4  defaults  0  2

# By label
LABEL=mylabel  /mnt/data  ext4  defaults  0  2

# With specific options
UUID=xxx  /mnt/data  ext4  rw,noexec,nosuid  0  2

# NFS mount
server:/export  /mnt/nfs  nfs  defaults  0  0

# Temporary filesystem
tmpfs  /tmp  tmpfs  defaults,noatime,mode=1777  0  0

Common mount options:

• defaults - rw, suid, dev, exec, auto, nouser, async
• ro - Read-only
• rw - Read-write
• noexec - Don't allow program execution
• nosuid - Ignore SUID bits
• nodev - Don't interpret block special devices
• noatime - Don't update access time (performance)
• nodiratime - Don't update directory access time
• nofail - Don't fail boot if device missing

Apply fstab changes:

sudo mount -a                      # Mount all in fstab
sudo findmnt --verify              # Verify fstab syntax

Permissions and ACLs

Standard Permissions

Permission types:

• r (4) - Read
• w (2) - Write
• x (1) - Execute

Three groups:

• Owner
• Group
• Others

Examples:

# Symbolic
chmod u+x file                     # Add execute for user
chmod g+w file                     # Add write for group
chmod o-r file                     # Remove read for others
chmod a+x file                     # Add execute for all

# Numeric
chmod 644 file                     # rw-r--r--
chmod 755 file                     # rwxr-xr-x
chmod 600 file                     # rw-------
chmod 777 file                     # rwxrwxrwx (avoid!)

# Recursive
chmod -R 755 directory

Change ownership:

chown user file                    # Change owner
chown user:group file              # Change owner and group
chown -R user:group directory      # Recursive
chgrp group file                   # Change group only

Special permissions:

# SUID (Set User ID) - 4000
chmod u+s executable               # Run as file owner
chmod 4755 executable

# SGID (Set Group ID) - 2000
chmod g+s executable               # Run as file group
chmod g+s directory                # New files inherit directory group
chmod 2755 directory

# Sticky bit - 1000
chmod +t directory                 # Only owner can delete files
chmod 1777 /tmp                    # Typical for /tmp

Access Control Lists (ACLs)

Extended permissions beyond standard owner/group/other.

View ACLs:

getfacl file

Set ACLs:

# Give user specific permissions
setfacl -m u:username:rw file

# Give group specific permissions
setfacl -m g:groupname:rx file

# Remove ACL
setfacl -x u:username file

# Remove all ACLs
setfacl -b file

# Default ACLs for directory (inherited by new files)
setfacl -d -m u:username:rw directory

# Recursive
setfacl -R -m u:username:rw directory

Copy ACLs:

getfacl file1 | setfacl --set-file=- file2

Disk Usage Management

Checking Disk Usage

Filesystem usage:

df -h                              # Human-readable
df -i                              # Inode usage
df -T                              # Show filesystem type
df -h /path                        # Specific mount point

Directory usage:

du -sh /path                       # Summary
du -h --max-depth=1 /path          # One level deep
du -sh /* | sort -h                # Sort by size
ncdu /path                         # Interactive (requires install)

Find large files:

find /path -type f -size +100M     # Files > 100MB
find /path -type f -size +100M -exec ls -lh {} \;

# Top 10 largest files
find /path -type f -exec du -h {} + | sort -rh | head -10

Find large directories:

du -h /path | sort -rh | head -20

Cleaning Up Disk Space

Log files:

# Find large logs
find /var/log -type f -size +10M

# Truncate logs (don't delete - may break apps)
sudo truncate -s 0 /var/log/large.log

# Rotate logs
sudo logrotate -f /etc/logrotate.conf

# Clean systemd journal
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

Package caches:

# Ubuntu/Debian
sudo apt clean
sudo apt autoremove

# RHEL/Fedora
sudo dnf clean all

Temp files:

sudo find /tmp -type f -atime +7 -delete
sudo find /var/tmp -type f -atime +30 -delete

Deleted files still open:

# Find processes holding deleted files
sudo lsof | grep deleted

# Restart service to release
systemctl restart service_name

Best Practices

1. Always use UUIDs in fstab (device names can change) 2. Test fstab with `mount -a` before rebooting 3. Backup data before LVM operations 4. Use LVM for flexibility (easy to resize) 5. Monitor RAID arrays regularly 6. Set appropriate permissions (principle of least privilege) 7. Use noatime/nodiratime for performance 8. Regular filesystem checks (fsck during maintenance windows)

References

• mount(8): man mount
• fstab(5): man fstab
• lvm(8): man lvm
• mdadm(8): man mdadm
• chmod(1): man chmod
• setfacl(1): man setfacl

Network Configuration Guide

Network configuration, troubleshooting, and firewall management for Linux systems.

Table of Contents

1. ip Command Reference 2. ss Socket Statistics 3. Network Managers 4. DNS Configuration 5. Firewall Configuration 6. Network Troubleshooting

ip Command Reference

Modern replacement for ifconfig, route, and arp.

Interface Management

# Show interfaces
ip link show
ip addr show
ip a                               # Short form

# Show specific interface
ip addr show eth0

# Bring interface up/down
sudo ip link set eth0 up
sudo ip link set eth0 down

# Add IP address
sudo ip addr add 192.168.1.100/24 dev eth0

# Delete IP address
sudo ip addr del 192.168.1.100/24 dev eth0

# Change MTU
sudo ip link set eth0 mtu 9000

Routing

# Show routing table
ip route show
ip route list
ip r                               # Short form

# Show route to specific IP
ip route get 8.8.8.8

# Add route
sudo ip route add 10.0.0.0/24 via 192.168.1.1
sudo ip route add default via 192.168.1.1

# Delete route
sudo ip route del 10.0.0.0/24
sudo ip route del default

# Add route via interface
sudo ip route add 10.0.0.0/24 dev eth0

Neighbor (ARP) Table

# Show ARP cache
ip neigh show
ip n

# Add static ARP entry
sudo ip neigh add 192.168.1.10 lladdr 00:11:22:33:44:55 dev eth0

# Delete ARP entry
sudo ip neigh del 192.168.1.10 dev eth0

# Flush ARP cache
sudo ip neigh flush all

ss Socket Statistics

Modern replacement for netstat.

Basic Usage

# All connections
ss -a

# TCP connections
ss -t

# UDP connections
ss -u

# Listening ports
ss -l

# Combine options
ss -tunap                          # TCP+UDP, numeric, all, processes
ss -tlnp                           # TCP, listening, numeric, processes
ss -ulnp                           # UDP, listening, numeric, processes

Filtering

# Established connections
ss -tnp state established

# Listening ports
ss -tlnp

# Specific port
ss -tlnp | grep :80
ss -tunap '( dport = :80 )'

# Specific host
ss dst 192.168.1.100

# Show process names
ss -tp

# Show memory usage
ss -tm

Common Filters

# By state
ss state established
ss state time-wait
ss state close-wait
ss state syn-sent

# By port
ss sport = :22                     # Source port 22
ss dport = :80                     # Destination port 80
ss dport \> :1024                  # Ports > 1024

# By IP
ss src 192.168.1.0/24
ss dst 10.0.0.0/8

Network Managers

netplan (Ubuntu 18.04+)

Configuration: /etc/netplan/*.yaml

DHCP:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      dhcp6: false

Static IP:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      addresses:
        - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4
        search:
          - example.com

Multiple IPs:

network:
  version: 2
  ethernets:
    eth0:
      addresses:
        - 192.168.1.100/24
        - 192.168.1.101/24

Apply configuration:

sudo netplan apply
sudo netplan try                   # Test with auto-rollback
sudo netplan --debug apply         # Verbose

NetworkManager (RHEL/Fedora)

Command-line (nmcli):

# Show status
nmcli device status
nmcli connection show

# Show details
nmcli device show eth0
nmcli connection show "Wired connection 1"

# Configure static IP
nmcli con mod eth0 ipv4.addresses 192.168.1.100/24
nmcli con mod eth0 ipv4.gateway 192.168.1.1
nmcli con mod eth0 ipv4.dns "8.8.8.8 8.8.4.4"
nmcli con mod eth0 ipv4.method manual

# Apply changes
nmcli con up eth0

# Configure DHCP
nmcli con mod eth0 ipv4.method auto
nmcli con up eth0

# Add connection
nmcli con add type ethernet con-name eth0 ifname eth0

# Delete connection
nmcli con del "Wired connection 1"

Interactive TUI:

nmtui

DNS Configuration

systemd-resolved (Modern)

Status:

resolvectl status
resolvectl query example.com
resolvectl flush-caches

Configuration: /etc/systemd/resolved.conf

[Resolve]
DNS=8.8.8.8 8.8.4.4
FallbackDNS=1.1.1.1
Domains=example.com

Apply changes:

sudo systemctl restart systemd-resolved

Traditional resolv.conf

Configuration: /etc/resolv.conf

nameserver 8.8.8.8
nameserver 8.8.4.4
search example.com local.example.com
options timeout:2 attempts:3

For static configuration (prevent overwrite):

sudo chattr +i /etc/resolv.conf    # Make immutable
sudo chattr -i /etc/resolv.conf    # Remove immutable

DNS Tools

# Query DNS
dig example.com
dig @8.8.8.8 example.com           # Specific DNS server
dig +short example.com             # Brief output
dig -x 8.8.8.8                     # Reverse DNS

# nslookup
nslookup example.com
nslookup example.com 8.8.8.8

# host
host example.com
host 8.8.8.8                       # Reverse DNS

Firewall Configuration

ufw (Ubuntu)

Basic usage:

# Status
sudo ufw status
sudo ufw status verbose
sudo ufw status numbered           # Show rule numbers

# Enable/disable
sudo ufw enable
sudo ufw disable

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw default deny routed

Allow/deny rules:

# By port
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw deny 23/tcp

# By service name
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https

# Port ranges
sudo ufw allow 6000:6007/tcp

# Specific interface
sudo ufw allow in on eth0 to any port 22

Advanced rules:

# From specific IP
sudo ufw allow from 192.168.1.100

# From subnet
sudo ufw allow from 192.168.1.0/24

# From IP to specific port
sudo ufw allow from 192.168.1.100 to any port 22

# Delete rules
sudo ufw delete allow 80/tcp
sudo ufw delete 3                  # By rule number

# Reset firewall
sudo ufw reset

Application profiles:

# List profiles
sudo ufw app list

# Allow application
sudo ufw allow 'Nginx Full'
sudo ufw allow 'OpenSSH'

# Show app info
sudo ufw app info 'Nginx Full'

firewalld (RHEL/CentOS/Fedora)

Basic usage:

# Status
sudo firewall-cmd --state
sudo firewall-cmd --list-all
sudo firewall-cmd --list-all-zones

# Get default zone
sudo firewall-cmd --get-default-zone
sudo firewall-cmd --get-active-zones

Services:

# List available services
firewall-cmd --get-services

# Allow service
sudo firewall-cmd --add-service=http
sudo firewall-cmd --add-service=https
sudo firewall-cmd --add-service=http --permanent

# Remove service
sudo firewall-cmd --remove-service=http
sudo firewall-cmd --remove-service=http --permanent

# Reload (apply permanent changes)
sudo firewall-cmd --reload

Ports:

# Add port
sudo firewall-cmd --add-port=8080/tcp
sudo firewall-cmd --add-port=8080/tcp --permanent

# Port range
sudo firewall-cmd --add-port=6000-6007/tcp --permanent

# Remove port
sudo firewall-cmd --remove-port=8080/tcp --permanent

Sources:

# Allow from IP/subnet
sudo firewall-cmd --zone=public --add-source=192.168.1.0/24 --permanent
sudo firewall-cmd --zone=public --add-source=192.168.1.100 --permanent

# Remove source
sudo firewall-cmd --zone=public --remove-source=192.168.1.100 --permanent

Zones:

# List zones
firewall-cmd --get-zones

# Change default zone
sudo firewall-cmd --set-default-zone=home

# Add interface to zone
sudo firewall-cmd --zone=public --add-interface=eth0 --permanent

Rich rules:

# Allow SSH from specific IP
sudo firewall-cmd --add-rich-rule='rule family="ipv4" source address="192.168.1.100" service name="ssh" accept' --permanent

# Block IP
sudo firewall-cmd --add-rich-rule='rule family="ipv4" source address="10.0.0.50" reject' --permanent

# Rate limiting
sudo firewall-cmd --add-rich-rule='rule service name="ssh" limit value="10/m" accept' --permanent

iptables (Low-level)

Basic usage:

# List rules
sudo iptables -L
sudo iptables -L -n -v             # Numeric, verbose
sudo iptables -L -n --line-numbers

# Allow port
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Block IP
sudo iptables -A INPUT -s 10.0.0.50 -j DROP

# Delete rule
sudo iptables -D INPUT 3           # By line number

# Flush all rules
sudo iptables -F

# Save rules
sudo iptables-save > /etc/iptables/rules.v4
sudo netfilter-persistent save     # Ubuntu

Network Troubleshooting

Connectivity Tests

# Ping
ping -c 4 8.8.8.8                  # 4 packets
ping -c 4 google.com               # Test DNS too

# Traceroute
traceroute 8.8.8.8
traceroute -n 8.8.8.8              # Numeric (faster)
mtr 8.8.8.8                        # Combined ping/traceroute

# Port connectivity
telnet server 80
nc -zv server 80                   # netcat
curl -v telnet://server:80

# DNS lookup
dig google.com
nslookup google.com
host google.com

Bandwidth Testing

# iperf3
# Server:
iperf3 -s

# Client:
iperf3 -c server_ip -t 60          # 60 second test
iperf3 -c server_ip -R             # Reverse (download)
iperf3 -c server_ip -u -b 100M     # UDP, 100 Mbps

Network Statistics

# Interface statistics
ip -s link show eth0
ifconfig eth0                      # Legacy

# Error counts
netstat -i                         # Legacy
ip -s link

# Protocol statistics
netstat -s
ss -s

Packet Capture

# tcpdump
sudo tcpdump -i eth0                      # All traffic
sudo tcpdump -i eth0 port 80              # HTTP
sudo tcpdump -i eth0 host 192.168.1.100   # Specific host
sudo tcpdump -i eth0 -w capture.pcap      # Save to file
sudo tcpdump -r capture.pcap              # Read from file

# More specific
sudo tcpdump -i eth0 'tcp port 80 and host 192.168.1.100'

Common Issues

No network connectivity:

# Check interface status
ip link show eth0
# Should show "state UP"

# Check IP address
ip addr show eth0

# Check routes
ip route show

# Check DNS
cat /etc/resolv.conf
dig google.com

# Check firewall
sudo iptables -L -n -v
sudo ufw status
sudo firewall-cmd --list-all

Slow network:

# Check errors
ip -s link show eth0
# Look for RX/TX errors, drops

# Check MTU
ip link show eth0 | grep mtu

# Test bandwidth
iperf3 -c server

# Check latency
ping -c 100 server | tail -1

Port already in use:

# Find process
sudo ss -tlnp | grep :8080
sudo lsof -i :8080

# Kill process
sudo kill -9 PID

Best Practices

1. Use static IP for servers (easier management) 2. Document firewall rules (know what's open) 3. Default deny policy (explicit allow) 4. Test configuration changes before applying 5. Monitor network metrics regularly 6. Keep firewall rules minimal (only necessary ports) 7. Use fail2ban for SSH brute-force protection

References

• ip(8): man ip
• ss(8): man ss
• iptables(8): man iptables
• firewalld documentation: https://firewalld.org/
• ufw documentation: https://help.ubuntu.com/community/UFW

Performance Tuning Guide

Complete reference for optimizing Linux system performance through sysctl, ulimits, cgroups, I/O schedulers, and workload-specific tuning.

Table of Contents

1. Performance Analysis Methodology 2. sysctl Kernel Parameter Tuning 3. ulimit Resource Limits 4. Control Groups (cgroups) 5. I/O Schedulers 6. CPU Governors 7. Workload-Specific Tuning 8. Monitoring and Validation

Performance Analysis Methodology

Baseline First

Before tuning, establish baseline:

1. Measure current performance

   # CPU metrics
   mpstat 1 10                     # 10 samples, 1 second apart

   # Memory metrics
   free -h && vmstat 1 10

   # Disk I/O
   iostat -x 1 10

   # Network
   sar -n DEV 1 10

2. Identify bottleneck

• CPU: High user/system time, low idle
• Memory: High swap usage, low free/available
• Disk: High I/O wait (wa%), high service times
• Network: High packet loss, retransmissions

3. Apply one change at a time 4. Measure impact 5. Document everything

Performance Investigation Tools

# Overview
top                              # Real-time
htop                             # Enhanced
uptime                           # Load averages

# CPU
mpstat 1                         # Per-CPU stats
pidstat -u 1                     # Per-process CPU
perf top                         # CPU profiling

# Memory
free -h                          # Memory usage
vmstat 1                         # Virtual memory stats
smem -t                          # Memory by process

# Disk I/O
iostat -x 1                      # Extended disk stats
iotop -oPa                       # I/O by process
lsof | grep deleted              # Deleted files still open

# Network
ss -tunap                        # Connections
iftop                            # Bandwidth
nethogs                          # Per-process network

sysctl Kernel Parameter Tuning

sysctl Basics

View current value:

sysctl vm.swappiness
sysctl -a | grep tcp             # All TCP parameters

Set temporarily (until reboot):

sudo sysctl -w vm.swappiness=10

Set permanently:

# Create file in /etc/sysctl.d/
sudo nano /etc/sysctl.d/99-custom.conf

# Add parameters
vm.swappiness = 10
net.ipv4.tcp_congestion_control = bbr

# Apply immediately
sudo sysctl -p /etc/sysctl.d/99-custom.conf

# Or reload all
sudo sysctl --system

Memory Management Parameters

Swappiness:

# vm.swappiness (0-100, default 60)
# Lower = prefer RAM, higher = more aggressive swapping

vm.swappiness = 10                # Good for servers (prefer RAM)
vm.swappiness = 60                # Default (balanced)
vm.swappiness = 1                 # Minimal swapping (databases)
vm.swappiness = 100               # Aggressive swapping (unusual)

VFS Cache Pressure:

# vm.vfs_cache_pressure (default 100)
# Lower = keep dentry/inode cache longer
# Higher = reclaim faster

vm.vfs_cache_pressure = 50        # Keep more cached (file servers)
vm.vfs_cache_pressure = 100       # Default
vm.vfs_cache_pressure = 200       # Aggressive reclaim (memory-constrained)

Dirty Page Writeback:

# Controls when dirty pages written to disk

# vm.dirty_ratio (default 20)
# % of RAM that can be dirty before blocking writes
vm.dirty_ratio = 15               # Start blocking at 15% RAM

# vm.dirty_background_ratio (default 10)
# % of RAM that triggers background writeback
vm.dirty_background_ratio = 5     # Earlier background writes

# vm.dirty_expire_centisecs (default 3000 = 30 seconds)
# Age before dirty page written
vm.dirty_expire_centisecs = 1500  # Write after 15 seconds

# vm.dirty_writeback_centisecs (default 500 = 5 seconds)
# Interval for writeback daemon
vm.dirty_writeback_centisecs = 300  # Check every 3 seconds

Overcommit:

# vm.overcommit_memory (default 0)
# 0 = Heuristic (default, sane overcommit)
# 1 = Always overcommit (no checks)
# 2 = Never overcommit (strict accounting)

vm.overcommit_memory = 0          # Default (recommended)

# vm.overcommit_ratio (default 50)
# % of RAM to allow overcommit (when mode = 2)
vm.overcommit_ratio = 80          # Allow 80% overcommit

Transparent Huge Pages (THP):

# Check status
cat /sys/kernel/mm/transparent_hugepage/enabled

# Disable (recommended for databases like MongoDB, Redis)
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Make persistent (add to /etc/rc.local or systemd service)

Network Parameters

TCP Congestion Control:

# View available algorithms
sysctl net.ipv4.tcp_available_congestion_control

# Set BBR (modern, high-performance)
net.ipv4.tcp_congestion_control = bbr

# Alternatives:
# cubic (default) - Good for most cases
# reno - Classical TCP
# htcp - High-speed TCP

Enable BBR (requires kernel 4.9+):

# Load modules
sudo modprobe tcp_bbr
echo "tcp_bbr" | sudo tee -a /etc/modules-load.d/modules.conf

# Enable BBR
echo "net.ipv4.tcp_congestion_control=bbr" | sudo tee -a /etc/sysctl.d/99-bbr.conf
sudo sysctl -p /etc/sysctl.d/99-bbr.conf

TCP Buffer Sizes:

# net.ipv4.tcp_rmem (min default max)
# Read buffer sizes
net.ipv4.tcp_rmem = 4096 87380 16777216   # 4KB min, 85KB default, 16MB max

# net.ipv4.tcp_wmem (min default max)
# Write buffer sizes
net.ipv4.tcp_wmem = 4096 65536 16777216   # 4KB min, 64KB default, 16MB max

# net.core.rmem_max / net.core.wmem_max
# Maximum socket buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# net.core.rmem_default / net.core.wmem_default
# Default socket buffer sizes
net.core.rmem_default = 262144
net.core.wmem_default = 262144

Connection Queue Sizes:

# net.core.somaxconn
# Max listen() backlog
net.core.somaxconn = 4096         # Default 128 (too low for busy servers)

# net.ipv4.tcp_max_syn_backlog
# Max SYN queue size
net.ipv4.tcp_max_syn_backlog = 8192  # Default 1024

# net.core.netdev_max_backlog
# Max packets in input queue
net.core.netdev_max_backlog = 250000  # Default 1000

TCP Tuning:

# TIME_WAIT reuse (connections in TIME_WAIT state)
net.ipv4.tcp_tw_reuse = 1         # Reuse TIME_WAIT sockets (safe)

# Keepalive settings
net.ipv4.tcp_keepalive_time = 600     # Send keepalive after 10 min idle
net.ipv4.tcp_keepalive_intvl = 60     # Probe interval
net.ipv4.tcp_keepalive_probes = 3     # Max probes before drop

# Fast open (reduce latency)
net.ipv4.tcp_fastopen = 3         # 3 = client and server

# Slow start after idle
net.ipv4.tcp_slow_start_after_idle = 0  # Disable for persistent connections

# Timestamps (helps with high-bandwidth connections)
net.ipv4.tcp_timestamps = 1       # Enable (adds 12 bytes per packet)

# SACK (Selective Acknowledgment)
net.ipv4.tcp_sack = 1             # Enable (improves performance on lossy networks)

# Window scaling (essential for high-bandwidth)
net.ipv4.tcp_window_scaling = 1   # Enable

# MTU probing (avoid fragmentation)
net.ipv4.tcp_mtu_probing = 1      # Enable

Security:

# SYN flood protection
net.ipv4.tcp_syncookies = 1       # Enable SYN cookies

# IP forwarding (routers/NAT)
net.ipv4.ip_forward = 1           # Enable forwarding

# Reverse path filtering (prevent IP spoofing)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Ignore source routed packets
net.ipv4.conf.all.accept_source_route = 0

# Ignore broadcast pings
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore bogus ICMP errors
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log martians (packets with impossible addresses)
net.ipv4.conf.all.log_martians = 1

Filesystem Parameters

# fs.file-max
# Maximum file handles
fs.file-max = 2097152             # Default ~100k

# fs.nr_open
# Per-process file descriptor limit
fs.nr_open = 2097152

# fs.aio-max-nr
# Maximum async I/O requests
fs.aio-max-nr = 1048576           # Default 65536

Kernel Parameters

# kernel.pid_max
# Maximum PID value
kernel.pid_max = 4194304          # Default 32768

# kernel.threads-max
# Maximum number of threads
kernel.threads-max = 4194304

# kernel.panic
# Seconds before reboot after kernel panic
kernel.panic = 10                 # Reboot after 10 seconds

# kernel.panic_on_oops
# Panic on kernel oops
kernel.panic_on_oops = 1          # Recommended for production

ulimit Resource Limits

Understanding ulimits

User-level resource limits per process.

View current limits:

ulimit -a                         # All limits
ulimit -n                         # Open files
ulimit -u                         # Max processes

Set temporarily (current session):

ulimit -n 65536                   # Max open files
ulimit -u 4096                    # Max user processes

Permanent ulimit Configuration

Edit `/etc/security/limits.conf`:

# Format: <domain> <type> <item> <value>
# domain: username, @groupname, *
# type: soft (warning), hard (enforced)
# item: nofile, nproc, memlock, stack, etc.

# Example: Web server user
nginx  soft  nofile  100000
nginx  hard  nofile  100000

# Example: All users
*      soft  nofile  65536
*      hard  nofile  65536
*      soft  nproc   4096
*      hard  nproc   8192

# Example: Database user (PostgreSQL, MySQL)
postgres  soft  nofile  200000
postgres  hard  nofile  200000
postgres  soft  nproc   16384
postgres  hard  nproc   16384

# Example: Allow memory locking (for databases)
postgres  soft  memlock  unlimited
postgres  hard  memlock  unlimited

Drop-in files (recommended):

# Create file per application
sudo nano /etc/security/limits.d/99-myapp.conf

myapp  soft  nofile  100000
myapp  hard  nofile  100000

Common ulimit Items

Systemd Service Limits

Override ulimits in systemd units:

[Service]
LimitNOFILE=100000                # Max open files
LimitNPROC=16384                  # Max processes
LimitMEMLOCK=infinity             # Memory locking
LimitCORE=infinity                # Core dump size

View service limits:

systemctl show myapp.service | grep ^Limit
cat /proc/$(pidof myapp)/limits

Control Groups (cgroups)

cgroups v2 (Modern)

Check version:

mount | grep cgroup
stat -fc %T /sys/fs/cgroup
# cgroup2fs = v2, tmpfs = v1

Resource Limits via Systemd

Systemd manages cgroups automatically.

CPU Limits:

[Service]
# CPUQuota - Percentage of CPU time (200% = 2 CPUs)
CPUQuota=50%                      # Limit to 50% of one CPU core

# CPUWeight - Relative CPU share (1-10000, default 100)
CPUWeight=200                     # 2x priority
CPUWeight=50                      # 0.5x priority

# Enable CPU accounting
CPUAccounting=yes

Memory Limits:

[Service]
# MemoryMax - Hard limit (kills process if exceeded)
MemoryMax=2G

# MemoryHigh - Soft limit (throttles before hard limit)
MemoryHigh=1.5G

# MemoryMin - Guaranteed memory
MemoryMin=512M

# Enable memory accounting
MemoryAccounting=yes

I/O Limits:

[Service]
# IOWeight - I/O priority (1-10000, default 100)
IOWeight=500

# IOReadBandwidthMax - Read bandwidth limit
IOReadBandwidthMax=/dev/sda 10M

# IOWriteBandwidthMax - Write bandwidth limit
IOWriteBandwidthMax=/dev/sda 5M

# IOReadIOPSMax - Read IOPS limit
IOReadIOPSMax=/dev/sda 1000

# IOWriteIOPSMax - Write IOPS limit
IOWriteIOPSMax=/dev/sda 500

# Enable I/O accounting
IOAccounting=yes

Task Limits:

[Service]
# TasksMax - Maximum number of tasks (processes/threads)
TasksMax=100                      # Limit to 100 tasks
TasksMax=50%                      # 50% of system TasksMax
TasksMax=infinity                 # No limit

View cgroup usage:

systemctl status myapp.service    # Shows CPU/memory usage
systemd-cgtop                     # Interactive cgroup monitor

Custom Slices

Group services with shared resources:

# /etc/systemd/system/myapp.slice
[Unit]
Description=My Application Resource Slice

[Slice]
CPUQuota=200%                     # 2 CPUs for all services in slice
MemoryMax=4G                      # 4GB total

Assign services to slice:

[Service]
Slice=myapp.slice

I/O Schedulers

Available Schedulers

Modern Linux uses multi-queue (blk-mq) schedulers:

Legacy (single-queue) schedulers:

• cfq - Completely Fair Queueing (deprecated)
• deadline - Deadline-based (deprecated)
• noop - No scheduling (deprecated)

Check Current Scheduler

cat /sys/block/sda/queue/scheduler
# Output: [mq-deadline] none kyber bfq
# [ ] = currently active

Change Scheduler Temporarily

echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Change Scheduler Permanently

Method 1: udev rule

sudo nano /etc/udev/rules.d/60-scheduler.rules

# NVMe: use none
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"

# SSDs: use mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"

# HDDs: use bfq
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Method 2: kernel boot parameter

# Edit /etc/default/grub
GRUB_CMDLINE_LINUX="elevator=mq-deadline"

# Update GRUB
sudo update-grub                  # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg  # RHEL/CentOS

Scheduler Recommendations

NVMe SSDs:

echo none > /sys/block/nvme0n1/queue/scheduler

SATA SSDs:

echo mq-deadline > /sys/block/sda/queue/scheduler

HDDs:

echo bfq > /sys/block/sda/queue/scheduler

Low-latency workloads:

echo kyber > /sys/block/sda/queue/scheduler

CPU Governors

Available Governors

Control CPU frequency scaling:

Check Current Governor

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Or
cpupower frequency-info

Change Governor

Temporary:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Permanent (systemd):

sudo nano /etc/systemd/system/cpufreq-performance.service

[Unit]
Description=Set CPU Governor to Performance

[Service]
Type=oneshot
ExecStart=/bin/bash -c "echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor"

[Install]
WantedBy=multi-user.target

sudo systemctl enable cpufreq-performance.service
sudo systemctl start cpufreq-performance.service

Using cpupower:

sudo apt install linux-tools-common  # Ubuntu
sudo dnf install kernel-tools         # RHEL/Fedora

# Set governor
sudo cpupower frequency-set -g performance

# View info
cpupower frequency-info
cpupower monitor

Workload-Specific Tuning

Web Server Tuning

# /etc/sysctl.d/99-web-server.conf

# Network
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1

# Memory
vm.swappiness = 10
vm.vfs_cache_pressure = 50

# File descriptors
fs.file-max = 2097152

# /etc/security/limits.d/99-nginx.conf
nginx  soft  nofile  100000
nginx  hard  nofile  100000

I/O Scheduler: none (NVMe) or mq-deadline (SSD) CPU Governor: schedutil or performance

Database Server Tuning

# /etc/sysctl.d/99-database.conf

# Memory (minimal swapping)
vm.swappiness = 1
vm.vfs_cache_pressure = 50
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

# Huge pages (for large databases)
vm.nr_hugepages = 1024  # Calculate based on memory needs

# Overcommit (strict)
vm.overcommit_memory = 2
vm.overcommit_ratio = 80

# Network (if applicable)
net.core.somaxconn = 4096
net.ipv4.tcp_congestion_control = bbr

# File descriptors
fs.file-max = 2097152
fs.aio-max-nr = 1048576

# /etc/security/limits.d/99-postgres.conf
postgres  soft  nofile  200000
postgres  hard  nofile  200000
postgres  soft  nproc   16384
postgres  hard  nproc   16384
postgres  soft  memlock  unlimited
postgres  hard  memlock  unlimited

Disable Transparent Huge Pages:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

I/O Scheduler: none (NVMe) or mq-deadline (SSD) CPU Governor: performance

High-Throughput File Server

# /etc/sysctl.d/99-file-server.conf

# Memory (cache files aggressively)
vm.swappiness = 10
vm.vfs_cache_pressure = 30    # Keep dentry/inode cache longer
vm.dirty_ratio = 40           # Allow more dirty pages
vm.dirty_background_ratio = 10

# Network
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# File descriptors
fs.file-max = 4194304

I/O Scheduler: none (NVMe) or mq-deadline (SSD)

Monitoring and Validation

Before and After Comparison

CPU:

# Before tuning
mpstat 1 60 > before_cpu.txt

# After tuning
mpstat 1 60 > after_cpu.txt

# Compare
diff before_cpu.txt after_cpu.txt

Memory:

vmstat 1 60 > before_memory.txt
# ... apply changes ...
vmstat 1 60 > after_memory.txt

Network:

sar -n DEV 1 60 > before_network.txt
# ... apply changes ...
sar -n DEV 1 60 > after_network.txt

Continuous Monitoring

sysstat (sar):

sudo apt install sysstat
sudo systemctl enable sysstat

# View historical data
sar -u                            # CPU
sar -r                            # Memory
sar -b                            # Disk I/O
sar -n DEV                        # Network

Prometheus Node Exporter:

# Install node_exporter
# Expose metrics at :9100/metrics
# Visualize in Grafana

Performance Testing

CPU stress test:

sudo apt install stress-ng
stress-ng --cpu 4 --timeout 60s

Memory stress test:

stress-ng --vm 2 --vm-bytes 2G --timeout 60s

Disk I/O benchmark:

sudo apt install fio

# Sequential read
fio --name=seqread --rw=read --bs=1M --size=1G --numjobs=1

# Random read
fio --name=randread --rw=randread --bs=4k --size=1G --numjobs=4

# Sequential write
fio --name=seqwrite --rw=write --bs=1M --size=1G --numjobs=1

Network benchmark:

sudo apt install iperf3

# Server
iperf3 -s

# Client
iperf3 -c server_ip -t 60

Best Practices

1. Establish baseline before changes 2. Apply one change at a time 3. Test in staging first 4. Monitor impact continuously 5. Document all changes 6. Keep configuration in version control 7. Use workload-specific tuning 8. Don't tune without measuring

References

• sysctl.conf(5) man page: man sysctl.conf
• limits.conf(5) man page: man limits.conf
• systemd.resource-control(5): man systemd.resource-control
• Linux kernel documentation: https://kernel.org/doc/Documentation/sysctl/

Security Hardening Guide

Security best practices for Linux servers including SSH, firewall, user management, and SELinux/AppArmor basics.

Table of Contents

1. SSH Hardening 2. Firewall Best Practices 3. User and Access Control 4. SELinux Basics 5. AppArmor Basics 6. System Hardening

SSH Hardening

SSH Key Setup

Generate strong SSH key:

# Ed25519 (modern, recommended)
ssh-keygen -t ed25519 -C "user@example.com"

# RSA (broader compatibility)
ssh-keygen -t rsa -b 4096 -C "user@example.com"

Copy to server:

ssh-copy-id user@server
# Or manually:
cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Set correct permissions:

# Client
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub

# Server
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

sshd_config Hardening

Essential settings: /etc/ssh/sshd_config

# Disable root login
PermitRootLogin no

# Key-based authentication only
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Protocol
Protocol 2

# Login restrictions
MaxAuthTries 3
LoginGraceTime 30s
MaxSessions 10

# User/group restrictions
AllowUsers deploy admin
AllowGroups sshusers

# Disable features
X11Forwarding no
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
GatewayPorts no

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Keep connections alive
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable GSSAPI (if not needed)
GSSAPIAuthentication no

Apply changes:

# Test configuration
sudo sshd -t

# Restart (keep backup session open!)
sudo systemctl restart sshd

Fail2ban

Install:

sudo apt install fail2ban         # Ubuntu
sudo dnf install fail2ban         # RHEL/Fedora

Configure: /etc/fail2ban/jail.local

[DEFAULT]
# Ban for 1 hour
bantime = 3600

# Find window of 10 minutes
findtime = 600

# Ban after 3 failures
maxretry = 3

# Email notifications (optional)
destemail = admin@example.com
sendername = Fail2Ban
action = %(action_mwl)s

[sshd]
enabled = true
port = ssh,2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

Manage fail2ban:

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Check status
sudo fail2ban-client status
sudo fail2ban-client status sshd

# Unban IP
sudo fail2ban-client set sshd unbanip 192.168.1.100

Firewall Best Practices

Principles

1. Default deny - Block everything by default 2. Explicit allow - Only open necessary ports 3. Principle of least privilege - Minimum access required 4. Regular review - Audit rules periodically

Ubuntu (ufw) Hardening

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw default deny routed

# Essential services only
sudo ufw allow 22/tcp comment 'SSH'
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'

# Restrict SSH to specific IPs
sudo ufw delete allow 22/tcp
sudo ufw allow from 192.168.1.0/24 to any port 22 comment 'SSH from office'

# Rate limiting (prevent brute force)
sudo ufw limit 22/tcp

# Enable
sudo ufw enable

# Logging
sudo ufw logging on
sudo ufw logging medium

RHEL/CentOS (firewalld) Hardening

# Set default zone
sudo firewall-cmd --set-default-zone=drop

# Create custom zone
sudo firewall-cmd --permanent --new-zone=servers
sudo firewall-cmd --permanent --zone=servers --set-target=DROP

# Add services
sudo firewall-cmd --permanent --zone=servers --add-service=ssh
sudo firewall-cmd --permanent --zone=servers --add-service=http
sudo firewall-cmd --permanent --zone=servers --add-service=https

# Restrict SSH to specific IPs
sudo firewall-cmd --permanent --zone=servers --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" service name="ssh" accept'

# Rate limiting
sudo firewall-cmd --permanent --add-rich-rule='rule service name="ssh" limit value="10/m" accept'

# Apply
sudo firewall-cmd --reload

User and Access Control

User Management

Create system user (for services):

sudo useradd -r -s /bin/false -M username
# -r = system user
# -s /bin/false = no login shell
# -M = no home directory

Create regular user:

sudo useradd -m -s /bin/bash username
sudo passwd username

Disable account:

sudo usermod -L username          # Lock account
sudo usermod -s /usr/sbin/nologin username  # Disable shell

Remove inactive users:

# Find users not logged in for 90 days
lastlog -b 90

# Lock old accounts
sudo passwd -l username

sudo Configuration

Edit sudoers safely:

sudo visudo

Common configurations:

# Full sudo access
username ALL=(ALL:ALL) ALL

# Passwordless sudo (use carefully!)
username ALL=(ALL) NOPASSWD: ALL

# Specific commands only
username ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx

# Group-based
%wheel ALL=(ALL:ALL) ALL          # RHEL/CentOS
%sudo ALL=(ALL:ALL) ALL           # Ubuntu/Debian

# Require password every time (no caching)
Defaults timestamp_timeout=0

# Log all sudo commands
Defaults logfile="/var/log/sudo.log"
Defaults log_year, log_host, log_input, log_output

Password Policies

Configure PAM: /etc/pam.d/common-password (Ubuntu) or /etc/pam.d/system-auth (RHEL)

# Password quality requirements
password requisite pam_pwquality.so retry=3 minlen=12 ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1

# Options:
# retry=3 - 3 attempts
# minlen=12 - 12 characters minimum
# ucredit=-1 - require 1 uppercase
# lcredit=-1 - require 1 lowercase
# dcredit=-1 - require 1 digit
# ocredit=-1 - require 1 special character

Password aging: /etc/login.defs

PASS_MAX_DAYS 90                  # Max 90 days
PASS_MIN_DAYS 1                   # Min 1 day between changes
PASS_WARN_AGE 7                   # Warn 7 days before expiry

Set for existing user:

sudo chage -M 90 -m 1 -W 7 username

# View settings
sudo chage -l username

SELinux Basics

SELinux Modes

Check status:

getenforce
sestatus

Modes:

• Enforcing - SELinux actively enforces policy
• Permissive - SELinux logs violations but doesn't enforce
• Disabled - SELinux completely disabled

Change mode:

# Temporary
sudo setenforce 0                 # Permissive
sudo setenforce 1                 # Enforcing

# Permanent: /etc/selinux/config
SELINUX=enforcing
SELINUX=permissive
SELINUX=disabled

SELinux Contexts

View contexts:

ls -Z /var/www/html
ps auxZ                           # Process contexts

Common contexts:

• httpd_sys_content_t - Web server readable content
• httpd_sys_rw_content_t - Web server writable content
• httpd_sys_script_exec_t - Web server executable scripts

Change context:

# Set specific context
sudo chcon -t httpd_sys_content_t /var/www/html/index.html

# Restore default contexts
sudo restorecon -Rv /var/www/html

# Make permanent (file context rules)
sudo semanage fcontext -a -t httpd_sys_content_t "/var/www/html(/.*)?"
sudo restorecon -Rv /var/www/html

SELinux Troubleshooting

Check denials:

# Recent denials
sudo ausearch -m avc -ts recent

# All denials
sudo grep avc /var/log/audit/audit.log

# Human-readable
sudo sealert -a /var/log/audit/audit.log

Create policy module:

# Generate policy from denials
sudo audit2allow -a -M mypolicy

# Install policy
sudo semodule -i mypolicy.pp

# View installed modules
sudo semodule -l

Booleans:

# List booleans
getsebool -a
getsebool -a | grep httpd

# Set boolean
sudo setsebool -P httpd_can_network_connect on
# -P = permanent

AppArmor Basics

AppArmor Status

Check status:

sudo aa-status

Modes:

• Enforce - Policy is enforced
• Complain - Policy violations logged but not blocked
• Unconfined - No policy applied

Manage Profiles

Profile locations: /etc/apparmor.d/

Set mode:

# Complain mode (testing)
sudo aa-complain /usr/sbin/nginx

# Enforce mode
sudo aa-enforce /usr/sbin/nginx

# Disable profile
sudo ln -s /etc/apparmor.d/usr.sbin.nginx /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.nginx

# Enable profile
sudo rm /etc/apparmor.d/disable/usr.sbin.nginx
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

Troubleshooting

Check denials:

# System logs
sudo dmesg | grep -i apparmor
sudo journalctl | grep -i apparmor

# Audit log
sudo grep DENIED /var/log/syslog
sudo grep DENIED /var/log/audit/audit.log

Update profile:

# Edit profile
sudo nano /etc/apparmor.d/usr.sbin.nginx

# Reload profile
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

System Hardening

Disable Unnecessary Services

# List all services
systemctl list-unit-files --type=service

# Disable service
sudo systemctl disable service_name
sudo systemctl mask service_name  # Prevent accidental start

Common services to disable (if not needed):

• bluetooth.service
• cups.service (printing)
• avahi-daemon.service (zeroconf)

Kernel Hardening

sysctl settings: /etc/sysctl.d/99-security.conf

# IP forwarding (disable if not router)
net.ipv4.ip_forward = 0

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Ignore source routed packets
net.ipv4.conf.all.accept_source_route = 0

# Ignore broadcast pings
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore bogus ICMP errors
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Enable reverse path filtering (anti-spoofing)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Log martians
net.ipv4.conf.all.log_martians = 1

# SYN flood protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2

# Disable IPv6 (if not used)
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Apply:

sudo sysctl -p /etc/sysctl.d/99-security.conf

Automatic Updates

Ubuntu/Debian:

sudo apt install unattended-upgrades

# Configure: /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};

# Enable
sudo dpkg-reconfigure -plow unattended-upgrades

RHEL/CentOS/Fedora:

sudo dnf install dnf-automatic

# Configure: /etc/dnf/automatic.conf
[commands]
upgrade_type = security
apply_updates = yes

# Enable
sudo systemctl enable --now dnf-automatic.timer

File Integrity Monitoring

AIDE (Advanced Intrusion Detection Environment):

# Install
sudo apt install aide              # Ubuntu
sudo dnf install aide              # RHEL/Fedora

# Initialize database
sudo aideinit

# Check integrity
sudo aide --check

# Update database
sudo aide --update

Rootkit Detection

rkhunter:

# Install
sudo apt install rkhunter

# Update database
sudo rkhunter --update

# Scan system
sudo rkhunter --check

# Schedule daily scan
sudo systemctl enable rkhunter.timer

Audit Logging

auditd:

# Install
sudo apt install auditd

# Status
sudo systemctl status auditd

# View logs
sudo ausearch -m USER_LOGIN
sudo ausearch -m EXECVE            # Command execution

# Add rule
sudo auditctl -w /etc/passwd -p wa -k passwd_changes
# -w = watch file
# -p wa = write, attribute change
# -k = key for searching

Security Checklist

• [ ] SSH key-based authentication only
• [ ] Root login disabled
• [ ] Firewall configured with default deny
• [ ] Fail2ban installed and configured
• [ ] Strong password policies
• [ ] Automatic security updates enabled
• [ ] Unnecessary services disabled
• [ ] SELinux/AppArmor enabled
• [ ] Regular system updates
• [ ] Log monitoring configured
• [ ] User accounts reviewed regularly
• [ ] sudo access limited
• [ ] File integrity monitoring
• [ ] Kernel hardening applied

Best Practices

1. Defense in depth - Multiple security layers 2. Principle of least privilege - Minimal access 3. Regular updates - Apply security patches promptly 4. Monitor logs - Watch for suspicious activity 5. Backup regularly - Before making changes 6. Test changes - Staging environment first 7. Document security policies - Know what's configured 8. Regular audits - Review security posture

References

• CIS Benchmarks: https://www.cisecurity.org/cis-benchmarks/
• NIST Guidelines: https://csrc.nist.gov/
• SELinux documentation: https://selinuxproject.org/
• AppArmor documentation: https://wiki.ubuntu.com/AppArmor

Systemd Comprehensive Guide

Complete reference for systemd service management, unit files, dependencies, targets, and advanced configurations.

Table of Contents

1. Unit File Structure 2. Service Unit Directives 3. Timer Units 4. Dependencies and Ordering 5. Systemd Targets 6. Advanced Configurations 7. Security Hardening 8. Troubleshooting

Unit File Structure

File Locations and Priority

Systemd searches for unit files in this order (highest to lowest priority):

1. `/etc/systemd/system/` - System administrator units (highest priority)

• Place custom units and override files here
• Takes precedence over all other locations

2. `/run/systemd/system/` - Runtime units (volatile)

• Transient units created at runtime
• Cleared on reboot

3. `/lib/systemd/system/` - Distribution-provided units (lowest priority)

• Installed by package manager
• Never modify directly - use overrides instead

Basic Unit File Format

[Unit]
# Description and dependencies

[Service]
# Service-specific configuration

[Install]
# Installation information (for enable/disable)

Creating Override Files

Override without modifying original:

# Create override directory and file
sudo systemctl edit nginx.service

# This creates: /etc/systemd/system/nginx.service.d/override.conf
# Add your overrides:
[Service]
MemoryLimit=1G
CPUQuota=50%

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart nginx

View merged configuration:

systemctl cat nginx.service

Service Unit Directives

[Unit] Section

Metadata and dependencies for the unit.

Description and Documentation:

[Unit]
Description=My Web Application
Documentation=https://docs.example.com
Documentation=man:myapp(8)

Dependencies:

# Hard dependency (fails if dependency fails)
Requires=postgresql.service

# Soft dependency (continues if dependency fails)
Wants=redis.service

# Conflicts (stop if other unit starts)
Conflicts=apache2.service

Ordering:

# Start after these units
After=network.target postgresql.service

# Start before these units
Before=nginx.service

# Conditions (skip if condition fails)
ConditionPathExists=/etc/myapp/config.yml
ConditionFileNotEmpty=/etc/myapp/secret.key

[Service] Section

Service-specific configuration.

Service Type:

[Service]
# Type determines how systemd tracks the service

# simple (default) - Main process doesn't fork
Type=simple

# forking - Main process forks, systemd tracks parent
Type=forking
PIDFile=/var/run/myapp.pid

# oneshot - Process completes and exits (for scripts)
Type=oneshot
RemainAfterExit=yes

# notify - Process signals systemd when ready (sd_notify)
Type=notify

# dbus - Service acquires DBus name
Type=dbus
BusName=com.example.myapp

# idle - Wait until other services start
Type=idle

Execution Commands:

[Service]
# Main command to start service
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yml

# Pre-start commands (preparation)
ExecStartPre=/usr/bin/myapp-check-config
ExecStartPre=/bin/mkdir -p /var/run/myapp

# Post-start commands (verification)
ExecStartPost=/usr/bin/myapp-notify-monitoring

# Reload command (SIGHUP alternative)
ExecReload=/bin/kill -HUP $MAINPID

# Stop command (default: SIGTERM)
ExecStop=/usr/bin/myapp-shutdown

# Post-stop cleanup
ExecStopPost=/bin/rm -rf /var/run/myapp

Restart Policies:

[Service]
# When to restart
Restart=on-failure    # Restart only on failures
# Restart=always      # Always restart (even on clean exit)
# Restart=on-success  # Restart only on successful exits
# Restart=on-abnormal # Restart on watchdog, signal, or timeout
# Restart=no          # Never restart (default)

# Delay before restart
RestartSec=5s         # Wait 5 seconds before restart

# Maximum restart attempts
StartLimitBurst=5     # Max 5 restarts
StartLimitIntervalSec=10m  # Within 10 minutes

# Action when start limit hit
StartLimitAction=reboot    # Reboot system
# StartLimitAction=none    # Do nothing (default)

User and Group:

[Service]
# Run as specific user/group (security best practice)
User=myapp
Group=myapp

# Supplementary groups
SupplementaryGroups=docker ssl-cert

# Working directory
WorkingDirectory=/opt/myapp

# Root directory (chroot)
RootDirectory=/srv/myapp-root

Environment Variables:

[Service]
# Set environment variables
Environment="PORT=8080"
Environment="LOG_LEVEL=info"
Environment="DATABASE_URL=postgresql://localhost/myapp"

# Load from file
EnvironmentFile=/etc/myapp/environment
EnvironmentFile=-/etc/myapp/optional.env  # - prefix = optional

# Unset variables
UnsetEnvironment=DEBUG

Logging:

[Service]
# Output to journald (default)
StandardOutput=journal
StandardError=journal

# Syslog identifier (for filtering)
SyslogIdentifier=myapp

# Log level
SyslogLevel=info

# Output to file
StandardOutput=append:/var/log/myapp/output.log
StandardError=append:/var/log/myapp/error.log

# Null (discard)
StandardOutput=null

[Install] Section

Installation information for systemctl enable.

[Install]
# Target to enable service with
WantedBy=multi-user.target
# WantedBy=graphical.target  # For desktop services

# Alternative names
Alias=myapp-server.service
Alias=web-app.service

# Required by other units
RequiredBy=nginx.service

Timer Units

Systemd timers replace cron for scheduled tasks.

Basic Timer Structure

Timer Unit (myapp.timer):

[Unit]
Description=Run My Application Daily
Requires=myapp.service

[Timer]
# Calendar-based scheduling
OnCalendar=daily
OnCalendar=*-*-* 02:00:00

# Run missed timers after boot
Persistent=true

# Accuracy (allow slack for batch scheduling)
AccuracySec=5min

# Random delay (distribute load)
RandomizedDelaySec=30min

[Install]
WantedBy=timers.target

Service Unit (myapp.service):

[Unit]
Description=My Application Task

[Service]
Type=oneshot
User=myapp
ExecStart=/usr/local/bin/myapp-task
StandardOutput=journal
StandardError=journal

Timer Scheduling

Calendar Specifications:

# Predefined schedules
OnCalendar=minutely         # Every minute
OnCalendar=hourly           # Every hour
OnCalendar=daily            # Every day at 00:00
OnCalendar=weekly           # Every Monday at 00:00
OnCalendar=monthly          # 1st of month at 00:00
OnCalendar=yearly           # January 1st at 00:00

# Custom schedules
OnCalendar=*-*-* 02:00:00              # Daily at 2:00 AM
OnCalendar=Mon *-*-* 09:00:00          # Monday at 9:00 AM
OnCalendar=*-*-01 00:00:00             # 1st of month
OnCalendar=Mon,Fri *-*-* 08:00:00      # Monday and Friday 8:00 AM
OnCalendar=*-01,06,12-01 00:00:00      # Jan, Jun, Dec 1st
OnCalendar=*-*-* 00/2:00:00            # Every 2 hours

# Multiple schedules (runs on any match)
OnCalendar=Mon *-*-* 09:00:00
OnCalendar=Fri *-*-* 17:00:00

Relative Timers:

# Time after boot
OnBootSec=5min              # 5 minutes after boot

# Time after systemd starts
OnStartupSec=10min          # 10 minutes after systemd

# Time after unit activation
OnActiveSec=1h              # 1 hour after timer activated

# Time after unit last ran
OnUnitActiveSec=30min       # 30 minutes after last run
OnUnitInactiveSec=1h        # 1 hour after last finished

Time Units:

• s, sec, seconds
• m, min, minutes
• h, hr, hours
• d, days
• w, weeks
• M, months
• y, years

Timer Management

# List all timers
systemctl list-timers

# Show timer details
systemctl status backup.timer

# Show next activation
systemctl list-timers backup.timer

# Enable timer (not service!)
systemctl enable backup.timer
systemctl start backup.timer

# Manually trigger timer
systemctl start backup.service

# View logs
journalctl -u backup.service
journalctl -u backup.timer

Dependencies and Ordering

Dependency Types

Requires (Hard Dependency):

[Unit]
Requires=postgresql.service
After=postgresql.service

# If postgresql fails, this unit fails
# After= ensures ordering

Wants (Soft Dependency):

[Unit]
Wants=redis.service
After=redis.service

# If redis fails, this unit continues
# Recommended for optional dependencies

Requisite (Must Already Be Active):

[Unit]
Requisite=network.target
After=network.target

# Fails if network.target not already active

BindsTo (Stronger Than Requires):

[Unit]
BindsTo=special-device.mount
After=special-device.mount

# Stopped when dependency stops
# Used for device/mount dependencies

PartOf (Propagates Stop/Restart):

[Unit]
PartOf=nginx.service

# When nginx stops/restarts, this unit does too
# Useful for helper services

Conflicts:

[Unit]
Conflicts=apache2.service

# Cannot run simultaneously with apache2

Ordering Directives

After:

[Unit]
After=network.target postgresql.service

# Start after these units
# Does NOT imply dependency (combine with Requires/Wants)

Before:

[Unit]
Before=nginx.service

# Start before nginx

Ordering Without Dependencies:

# BAD: Requires without After (race condition)
[Unit]
Requires=postgresql.service

# GOOD: Explicit ordering
[Unit]
Requires=postgresql.service
After=postgresql.service

Condition Directives

Skip service if condition not met:

[Unit]
# Path checks
ConditionPathExists=/etc/myapp/config.yml
ConditionPathIsDirectory=/var/lib/myapp
ConditionFileNotEmpty=/etc/myapp/secret.key

# Filesystem checks
ConditionFileSystem=/mnt/data=ext4

# Host checks
ConditionHost=webserver01
ConditionHost=!database-server

# Virtualization checks
ConditionVirtualization=yes    # Running in VM
ConditionVirtualization=kvm
ConditionVirtualization=docker

# Architecture checks
ConditionArchitecture=x86-64

# Kernel checks
ConditionKernelVersion=>=5.10

# User checks
ConditionUser=root
ConditionGroup=admin

Assert vs. Condition:

• Condition - Skip unit if false (no error)
• Assert - Fail unit if false (error logged)

# Use Assert for required conditions
AssertPathExists=/etc/critical-config.yml

Systemd Targets

Targets group units and define system states (like runlevels).

Common Targets

Target Management

# Show current target
systemctl get-default

# Set default target
sudo systemctl set-default multi-user.target

# Switch to target
sudo systemctl isolate rescue.target

# Show units in target
systemctl list-dependencies multi-user.target

# Show active targets
systemctl list-units --type=target

Creating Custom Targets

# /etc/systemd/system/myapp-stack.target
[Unit]
Description=My Application Stack
Requires=postgresql.service redis.service myapp.service
After=postgresql.service redis.service

[Install]
WantedBy=multi-user.target

Usage:

sudo systemctl start myapp-stack.target
sudo systemctl enable myapp-stack.target

Advanced Configurations

Resource Limits (Control Groups)

[Service]
# CPU limits
CPUQuota=50%                    # Limit to 50% of one CPU
CPUWeight=500                   # Relative weight (1-10000, default 100)
CPUAccounting=yes               # Enable CPU accounting

# Memory limits
MemoryLimit=1G                  # Hard limit
MemoryHigh=800M                 # Soft limit (throttle before hard limit)
MemoryMax=1G                    # Maximum (same as MemoryLimit)
MemoryAccounting=yes            # Enable memory accounting

# Task limits
TasksMax=100                    # Max number of processes/threads

# I/O limits
IOWeight=500                    # I/O weight (1-10000)
IOReadBandwidthMax=/dev/sda 10M # Read bandwidth limit
IOWriteBandwidthMax=/dev/sda 5M # Write bandwidth limit

# Device access
DeviceAllow=/dev/null rw
DeviceAllow=/dev/zero rw
DevicePolicy=strict             # Only allow explicitly listed devices

# Slice assignment
Slice=myapp.slice               # Custom cgroup slice

Watchdog

Monitor service health and restart on failure:

[Service]
Type=notify
WatchdogSec=30s                 # Expect notification every 30s
Restart=on-watchdog             # Restart if watchdog timeout

# Application must call sd_notify periodically:
# sd_notify(0, "WATCHDOG=1");

Socket Activation

Start service on-demand when socket accessed:

Socket Unit (myapp.socket):

[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

Service Unit (myapp.service):

[Unit]
Description=My Application

[Service]
ExecStart=/usr/bin/myapp
StandardInput=socket

Benefits:

• Delayed service start (faster boot)
• Automatic service activation
• Zero-downtime restarts

Path Units

Trigger service on filesystem changes:

# /etc/systemd/system/myapp-watch.path
[Unit]
Description=Watch Config Changes

[Path]
PathChanged=/etc/myapp/config.yml
Unit=myapp-reload.service

[Install]
WantedBy=multi-user.target

Security Hardening

Sandboxing Directives

[Service]
# Filesystem protection
PrivateTmp=true                 # Private /tmp and /var/tmp
ProtectSystem=strict            # Read-only /usr, /boot, /etc
ProtectHome=true                # Inaccessible /home
ReadWritePaths=/var/lib/myapp   # Exception for write access
ReadOnlyPaths=/etc/myapp
InaccessiblePaths=/proc/sys

# Privilege restrictions
NoNewPrivileges=true            # Prevent privilege escalation
PrivateDevices=true             # Private /dev (only pseudo devices)
ProtectKernelTunables=true      # Read-only /proc/sys, /sys
ProtectKernelModules=true       # Deny module loading
ProtectControlGroups=true       # Read-only cgroups

# Network restrictions
PrivateNetwork=true             # Private network namespace (no network)
RestrictAddressFamilies=AF_INET AF_INET6  # Only IPv4/IPv6

# Capability restrictions
CapabilityBoundingSet=CAP_NET_BIND_SERVICE  # Specific capabilities
AmbientCapabilities=CAP_NET_BIND_SERVICE

# System call filtering
SystemCallFilter=@system-service  # Whitelist common syscalls
SystemCallFilter=~@privileged     # Blacklist privileged syscalls
SystemCallErrorNumber=EPERM       # Return EPERM on denied syscall

# Execution restrictions
LockPersonality=true            # Prevent personality changes
RestrictRealtime=true           # Deny realtime scheduling
RestrictSUIDSGID=true           # Deny SUID/SGID execution
RemoveIPC=true                  # Remove IPC objects on exit

Security Analysis

Check security restrictions:

systemd-analyze security myapp.service

This shows:

• Current security settings
• Recommendations for hardening
• Security score (lower is better)

Troubleshooting

Debugging Service Failures

Check service status:

systemctl status myapp.service

View logs:

journalctl -u myapp.service
journalctl -u myapp.service --since today
journalctl -u myapp.service -n 100        # Last 100 lines
journalctl -u myapp.service -f            # Follow
journalctl -u myapp.service -p err        # Errors only

Check dependencies:

systemctl list-dependencies myapp.service
systemctl list-dependencies --reverse myapp.service

Test unit file:

systemd-analyze verify myapp.service

Show loaded unit:

systemctl cat myapp.service              # Show file
systemctl show myapp.service             # Show all properties
systemctl show myapp.service -p ExecStart -p Restart

Common Issues

Service fails to start:

# Check syntax
systemd-analyze verify myapp.service

# Check file permissions
ls -la /etc/systemd/system/myapp.service
# Should be: -rw-r--r-- root root

# Check executable exists
which myapp
ls -la /usr/bin/myapp

# Check user/group exists
id myapp

Service starts but stops immediately:

# Type=forking but no PIDFile
[Service]
Type=forking
PIDFile=/var/run/myapp.pid  # Must specify PID file

# Type=oneshot but should be simple
[Service]
Type=simple                  # For long-running processes

# Missing RemainAfterExit for oneshot
[Service]
Type=oneshot
RemainAfterExit=yes         # Keep active after exit

Dependency timeout:

# Increase timeout
[Service]
TimeoutStartSec=300         # 5 minutes (default 90s)

# Or make dependency optional
[Unit]
Wants=slow-service.service  # Instead of Requires

Service killed by OOM:

# Check for OOM kills
journalctl -k | grep -i "out of memory"
dmesg | grep -i oom

# Increase memory limit
[Service]
MemoryMax=2G                # Increase limit

# Or disable limit
MemoryMax=infinity

Systemd Boot Analysis

Analyze boot time:

systemd-analyze                          # Total boot time
systemd-analyze blame                    # Time per unit
systemd-analyze critical-chain           # Critical path
systemd-analyze plot > boot.svg          # Visual timeline

Find slow services:

systemd-analyze blame | head -20

Emergency Mode Recovery

Booting into rescue mode: 1. At GRUB, edit boot entry (press 'e') 2. Add systemd.unit=rescue.target to kernel line 3. Boot with Ctrl+X

Reset failed units:

systemctl reset-failed

Mask service (prevent start):

systemctl mask myapp.service     # Symlink to /dev/null
systemctl unmask myapp.service

Best Practices

Unit File Organization

1. Use override files for customization

   systemctl edit myapp.service  # Don't modify original

2. Drop-in directories for modular config

   /etc/systemd/system/myapp.service.d/
   ├── 10-resources.conf    # Resource limits
   ├── 20-security.conf     # Security hardening
   └── 30-monitoring.conf   # Monitoring hooks

3. Document with Description and Documentation

   [Unit]
   Description=My Web Application Server
   Documentation=https://docs.example.com
   Documentation=man:myapp(8)

Dependency Management

1. Always combine Requires/Wants with After

   [Unit]
   Requires=postgresql.service
   After=postgresql.service      # Explicit ordering

2. Use Wants for optional dependencies

   Wants=redis.service           # Continues if redis fails

3. Check dependency loops

   systemctl list-dependencies myapp.service --all

Security Hardening

1. Run as non-root

   [Service]
   User=myapp
   Group=myapp

2. Apply sandboxing

   PrivateTmp=true
   ProtectSystem=strict
   NoNewPrivileges=true

3. Analyze security

   systemd-analyze security myapp.service

Logging

1. Use journal for centralized logs

   [Service]
   StandardOutput=journal
   StandardError=journal
   SyslogIdentifier=myapp

2. Set appropriate log levels

   SyslogLevel=info

3. Query logs efficiently

   journalctl -u myapp -f -n 100 -p warning

References

• Official systemd documentation: https://systemd.io/
• systemd.service(5) man page: man systemd.service
• systemd.timer(5) man page: man systemd.timer
• systemd.unit(5) man page: man systemd.unit
• systemd.exec(5) man page: man systemd.exec

Troubleshooting Guide

Comprehensive guide for diagnosing and resolving common Linux system issues.

Table of Contents

1. Service Failures 2. Performance Issues 3. Network Problems 4. Disk Space Issues 5. Memory Problems 6. Permission Errors 7. Boot Issues 8. Process Problems

Service Failures

Symptom: Service Won't Start

Investigation Steps:

1. Check service status

   systemctl status myapp.service

2. View full logs

   journalctl -u myapp.service -n 100
   journalctl -u myapp.service --since "5 minutes ago"

3. Check unit file syntax

   systemd-analyze verify myapp.service

4. Test executable manually

   sudo -u myapp /usr/bin/myapp

Common Causes:

Missing executable:

# Symptom: "Failed to execute command: No such file or directory"
# Solution: Check path
which myapp
ls -la /usr/bin/myapp

Permission denied:

# Symptom: "Permission denied"
# Solution: Fix permissions
sudo chmod +x /usr/bin/myapp
sudo chown myapp:myapp /usr/bin/myapp

User doesn't exist:

# Symptom: "Failed to look up user"
# Solution: Create user
sudo useradd -r -s /bin/false myapp

Missing directory:

# Symptom: "No such file or directory" for WorkingDirectory
# Solution: Create directory
sudo mkdir -p /var/lib/myapp
sudo chown myapp:myapp /var/lib/myapp

Port already in use:

# Symptom: "Address already in use"
# Solution: Find process using port
sudo ss -tlnp | grep :8080
sudo kill PID

Symptom: Service Starts Then Stops

Investigation:

journalctl -u myapp.service -f    # Follow logs in real-time

Common Causes:

Type mismatch:

# Wrong: Type=forking but process doesn't fork
[Service]
Type=simple                        # Use simple for most cases

# Wrong: Type=simple but process forks
[Service]
Type=forking
PIDFile=/var/run/myapp.pid        # Must specify PID file

Missing RemainAfterExit:

# For oneshot services that should stay "active"
[Service]
Type=oneshot
RemainAfterExit=yes

Application crash:

# Check for segfaults, core dumps
journalctl -k | grep -i segfault
coredumpctl list
coredumpctl info PID

Memory limit:

# Check for OOM kills
journalctl -k | grep -i "out of memory"
dmesg | grep oom

# Solution: Increase limit
[Service]
MemoryMax=2G

Symptom: Dependency Timeout

Investigation:

systemctl list-dependencies myapp.service
systemctl list-dependencies --reverse myapp.service

Solution:

# Increase timeout
[Service]
TimeoutStartSec=300               # 5 minutes (default 90s)

# Or make dependency optional
[Unit]
Wants=slow-service.service        # Instead of Requires

Performance Issues

High CPU Usage

Investigation:

top                               # Press Shift+P for CPU sort
htop                              # Enhanced view
ps aux --sort=-%cpu | head -10   # Top 10 CPU consumers
mpstat -P ALL 1                   # Per-CPU statistics

Common Causes:

Runaway process:

# Find and kill
pgrep -a process_name
kill -15 PID                      # SIGTERM (graceful)
kill -9 PID                       # SIGKILL (force)

CPU-bound application:

# Reduce priority (nice)
renice -n 10 -p PID               # Lower priority

# Limit CPU with systemd
systemctl edit myapp.service
[Service]
CPUQuota=50%                      # Limit to 50% of one core

High system time (kernel):

# Check I/O wait
top                               # Look at wa% column
iostat -x 1                       # Disk I/O stats

# Check context switches
vmstat 1                          # cs = context switches/sec
pidstat -w 1                      # Per-process context switches

High Memory Usage / Swap

Investigation:

free -h                           # Memory overview
top                               # Press Shift+M for memory sort
ps aux --sort=-%mem | head -10   # Top memory consumers
smem -tk                          # Memory by process (with swap)

Common Causes:

Memory leak:

# Monitor memory over time
watch -n 5 'ps aux --sort=-%mem | head -10'

# Restart affected service
systemctl restart myapp.service

Excessive swap:

# Check swap usage
swapon --show
cat /proc/swaps

# Reduce swappiness
sudo sysctl -w vm.swappiness=10
echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.d/99-swap.conf

Insufficient memory:

# Check OOM kills
journalctl -k | grep -i oom

# Solutions:
# 1. Add more RAM
# 2. Reduce application memory
# 3. Add swap (temporary)

Page cache using memory:

# This is normal! Linux uses free memory for cache
# Check "available" memory, not "free"
free -h
# "available" is what matters

# Clear cache if needed (usually not necessary)
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

High Disk I/O Wait

Investigation:

iostat -x 1                       # Extended disk stats
iotop -oPa                        # I/O by process (requires install)
lsof | grep deleted               # Deleted files still open

Common Causes:

Heavy disk activity:

# Find I/O-intensive processes
iotop -oPa

# Check specific disk
iostat -x 1 /dev/sda

# High %util = disk saturated
# High await = high latency

Slow storage:

# Benchmark disk
sudo hdparm -Tt /dev/sda          # Sequential read

# Or use fio
fio --name=read --rw=read --bs=1M --size=1G

Wrong I/O scheduler:

# Check scheduler
cat /sys/block/sda/queue/scheduler

# For SSDs/NVMe
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

Deleted files still open:

# Find processes holding deleted files
lsof | grep deleted

# Restart services to release
systemctl restart service_name

High Load Average

Understanding Load Average:

• 1-minute, 5-minute, 15-minute averages
• Load of 1.0 = one CPU fully utilized
• Load > number of CPUs = some processes waiting

Investigation:

uptime                            # Load averages
cat /proc/loadavg                 # Detailed load info
mpstat 1                          # CPU utilization

High load but low CPU:

# Processes in uninterruptible sleep (I/O wait)
ps aux | grep D                   # D state = waiting for I/O
iostat -x 1                       # Check disk I/O

Network Problems

Cannot Connect to Service

Investigation:

# Check if service listening
ss -tlnp | grep :80
netstat -tlnp | grep :80          # Legacy alternative

# Check connectivity
ping server_ip
telnet server_ip 80
curl -v http://server_ip

Common Causes:

Service not running:

systemctl status nginx
systemctl start nginx

Firewall blocking:

# Ubuntu (ufw)
sudo ufw status
sudo ufw allow 80/tcp

# RHEL/CentOS (firewalld)
sudo firewall-cmd --list-all
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --reload

# iptables
sudo iptables -L -n -v

Wrong port/IP:

# Check what IP service binds to
ss -tlnp | grep :80
# 0.0.0.0:80 = all interfaces
# 127.0.0.1:80 = localhost only

# Fix in application config or use:
nginx -t                          # Test nginx config

DNS issues:

# Test DNS resolution
nslookup example.com
dig example.com
host example.com

# Check resolver
cat /etc/resolv.conf
resolvectl status

Slow Network Performance

Investigation:

# Check bandwidth
iftop                             # Real-time bandwidth
nethogs                           # Per-process bandwidth
iperf3 -c server                  # Bandwidth test

# Check latency
ping server
mtr server                        # Combined ping/traceroute

# Check dropped packets
netstat -s | grep -i drop
ip -s link show eth0

Common Causes:

Network congestion:

# Check interface statistics
ip -s link show eth0

# Look for:
# RX errors, dropped
# TX errors, dropped

MTU issues:

# Check MTU
ip link show eth0

# Test path MTU
ping -M do -s 1472 server         # 1472 + 28 = 1500

TCP tuning needed:

# Increase TCP buffers (for high-bandwidth links)
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

Port Conflict

Symptom: "Address already in use"

Investigation:

# Find process using port
sudo ss -tlnp | grep :8080
sudo lsof -i :8080
sudo fuser 8080/tcp

# Kill process
sudo kill -9 PID

# Or change service port

Disk Space Issues

Disk Full

Investigation:

df -h                             # Filesystem usage
df -i                             # Inode usage (can be full!)
du -sh /*                         # Usage by top-level dir
du -sh /var/* | sort -h           # Find large directories
ncdu /var                         # Interactive disk analyzer

Common Causes:

Log files:

# Find large log files
find /var/log -type f -size +100M

# Check log size
du -sh /var/log/*

# Rotate logs manually
sudo logrotate -f /etc/logrotate.conf

# Clear systemd journal
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

Deleted files still open:

# Find processes holding deleted files
sudo lsof | grep deleted | grep -v /tmp

# Restart service to release
systemctl restart service_name

Package cache:

# Ubuntu/Debian
sudo apt clean
sudo apt autoremove

# RHEL/CentOS/Fedora
sudo dnf clean all

Temp files:

# Clear /tmp (be careful!)
sudo find /tmp -type f -atime +7 -delete

# Clear user caches
rm -rf ~/.cache/*

Docker images/containers:

docker system df                  # Docker disk usage
docker system prune -a            # Remove unused data
docker volume prune               # Remove unused volumes

Inode Exhaustion

Symptom: "No space left on device" but df shows space available

Investigation:

df -i                             # Check inode usage

Solution:

# Find directories with many files
for dir in /*; do echo $dir; find $dir -xdev | wc -l; done

# Common culprits:
# - /var/spool/postfix (mail queue)
# - /tmp (many small files)
# - /var/lib/php/sessions

# Remove unnecessary files
find /tmp -type f -delete

Memory Problems

OOM (Out of Memory) Killer

Symptom: Processes killed randomly

Investigation:

# Check for OOM kills
journalctl -k | grep -i oom
dmesg | grep -i oom
grep -i oom /var/log/kern.log

View OOM scores:

# Higher score = more likely to be killed
for proc in /proc/[0-9]*; do
  printf "%2d %s\n" "$(cat $proc/oom_score 2>/dev/null)" "$(cat $proc/cmdline 2>/dev/null | tr '\000' ' ')";
done | sort -rn | head -10

Solutions:

1. Add more RAM or swap 2. Reduce application memory usage 3. Adjust OOM score (discourage killing critical services)

   echo -1000 | sudo tee /proc/PID/oom_score_adj  # Less likely to kill

4. Systemd service protection

   [Service]
   OOMScoreAdjust=-500            # Less likely to kill

Memory Leak Detection

Monitor memory over time:

# Watch specific process
watch -n 5 'ps aux --sort=-%mem | head -10'

# Or use pidstat
pidstat -r -p PID 5               # Memory stats every 5 seconds

# Log to file for analysis
while true; do
  date >> mem.log
  ps aux --sort=-%mem | head -10 >> mem.log
  sleep 60
done

Tools:

• Valgrind (for C/C++ applications)
• heaptrack (heap memory profiler)
• Application-specific tools (Node.js heap snapshots, Python tracemalloc)

Permission Errors

Permission Denied

Common Scenarios:

File permissions:

ls -la /path/to/file              # Check permissions
# Fix:
sudo chmod 644 file               # rw-r--r--
sudo chmod 755 script             # rwxr-xr-x
sudo chown user:group file

Directory permissions:

# Execute bit required to enter directory
chmod +x directory

SELinux (RHEL/CentOS):

# Check SELinux status
getenforce

# Check SELinux denials
ausearch -m avc -ts recent
grep avc /var/log/audit/audit.log

# Temporary: Set permissive mode
sudo setenforce 0

# Fix SELinux contexts
restorecon -Rv /path
chcon -t httpd_sys_content_t /var/www/html

# Create SELinux policy (if needed)
audit2allow -a -M mypolicy
semodule -i mypolicy.pp

AppArmor (Ubuntu):

# Check AppArmor status
sudo aa-status

# Check denials
sudo dmesg | grep -i apparmor
sudo journalctl | grep -i apparmor

# Disable profile (temporary)
sudo aa-complain /etc/apparmor.d/usr.bin.nginx

# Edit profile
sudo nano /etc/apparmor.d/usr.bin.nginx

sudo access:

# Check sudo config
sudo visudo

# Add user to sudo group
sudo usermod -aG sudo username    # Debian/Ubuntu
sudo usermod -aG wheel username   # RHEL/CentOS

Boot Issues

System Won't Boot

Emergency shell: 1. At GRUB, press 'e' to edit boot entry 2. Add systemd.unit=rescue.target or systemd.unit=emergency.target to kernel line 3. Press Ctrl+X to boot

Common causes:

Filesystem error:

# Run filesystem check
fsck /dev/sda1

fstab error:

# Comment out problematic line
nano /etc/fstab

Failed service blocking boot:

# Check failed units
systemctl --failed

# Mask service temporarily
systemctl mask failing-service.service

Slow Boot

Investigation:

systemd-analyze                   # Total boot time
systemd-analyze blame             # Time per unit
systemd-analyze critical-chain    # Critical path

Common causes:

• Slow network service (waiting for DHCP)
• Filesystem check
• Slow service startup

Solutions:

• Make services optional with Wants= instead of Requires=
• Use After=network-online.target for network services
• Increase timeout for slow services

Process Problems

Zombie Processes

Symptom: Processes in Z state

Investigation:

ps aux | grep Z                   # Find zombies
ps aux | grep 'Z'

Cause: Parent process hasn't reaped child process

Solution:

• Usually harmless (parent will reap eventually)
• If many zombies: restart parent process
• Killing zombie won't work (already dead)

Hung Process

Symptom: Process not responding

Investigation:

# Check process state
ps aux | grep process_name        # Look for D state

# D state = uninterruptible sleep (usually I/O)
# Check I/O
iotop
iostat -x 1

# Trace syscalls
strace -p PID                     # See what process doing
lsof -p PID                       # See open files

# Check locks
lslocks                           # Show file locks

Solution:

• Wait for I/O to complete (D state usually temporary)
• Kill if truly hung: kill -9 PID
• Fix underlying I/O issue

Too Many Open Files

Symptom: "Too many open files"

Investigation:

# Check current limit
ulimit -n

# Check process usage
lsof -p PID | wc -l
cat /proc/PID/limits

Solution:

# Increase limit temporarily
ulimit -n 65536

# Increase permanently
# Edit /etc/security/limits.conf
username soft nofile 65536
username hard nofile 65536

# For systemd service
[Service]
LimitNOFILE=65536

General Troubleshooting Workflow

1. Define the problem clearly

• What's not working?
• What's the expected behavior?
• When did it start?

2. Gather information

• Check logs: journalctl, /var/log/
• Check status: systemctl status, top, df -h
• Check network: ss, ping

3. Form hypothesis

• Based on symptoms and logs
• Consider recent changes

4. Test hypothesis

• Make one change at a time
• Document changes
• Verify impact

5. Fix and document

• Apply permanent fix
• Document in runbook
• Monitor for recurrence

Useful Commands Reference

System overview:

uptime                            # Load averages
top                               # Real-time monitor
htop                              # Enhanced monitor
df -h                             # Disk usage
free -h                           # Memory usage

Logs:

journalctl -xe                    # Recent logs with explanations
journalctl -f                     # Follow all logs
journalctl -u service -f          # Follow service logs
dmesg                             # Kernel ring buffer

Network:

ss -tunap                         # All connections
ip addr                           # IP addresses
ip route                          # Routing table
ping server                       # Connectivity

Processes:

ps aux                            # All processes
pgrep -a name                     # Find by name
kill -15 PID                      # Graceful kill
kill -9 PID                       # Force kill

Files:

lsof                              # Open files
lsof -p PID                       # Files for process
lsof -i :80                       # Process using port
find / -name filename             # Find file

References

• systemd.service(5): man systemd.service
• journalctl(1): man journalctl
• ps(1): man ps
• ss(8): man ss
• Linux troubleshooting guides: https://www.kernel.org/doc/