Proxmox

Proxmox VE

Production infrastructure demands production discipline. Every Proxmox configuration must be secure by default, redundant where it matters, and automated where possible.

Route to Reference

• VM, LXC, and OCI container management — [${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md]: VM vs LXC vs OCI

comparison, OCI container support (PVE 9.1 tech preview), Docker-on-Proxmox decision guidance, configuration options, template workflows, linked vs full clone trade-offs

• Storage backends — [${CLAUDE_SKILL_DIR}/references/storage-backends.md]: Backend capability matrix, ZFS tuning

(ARC/L2ARC/SLOG/volblocksize), Ceph configuration, LVM-Thin monitoring, storage selection decision tree

• Networking — [${CLAUDE_SKILL_DIR}/references/networking.md]: Bridge configuration, VLAN layout, bonding modes,

SDN zones (VXLAN/EVPN), MTU considerations, OVS vs Linux bridge, firewall lockout prevention

• Clustering and HA — [${CLAUDE_SKILL_DIR}/references/clustering-and-ha.md]: Corosync configuration, quorum math,

QDevice setup, split-brain prevention, fencing methods, HA groups, migration, quorum loss recovery

• API and automation — [${CLAUDE_SKILL_DIR}/references/api-automation.md]: REST API architecture, pvesh/qm/pct

reference, Terraform patterns, cloud-init customization (cicustom, network v2), hookscript lifecycle, CI/CD pipelines

• Backup strategies — [${CLAUDE_SKILL_DIR}/references/backup-strategies.md]: vzdump modes, PBS architecture,

encryption key management, garbage collection safety, verification jobs, retention policies, off-site sync patterns

Guest Management

<guest-management>

VM vs LXC vs OCI Decision

• Default to LXC for trusted Linux workloads — near-zero overhead, high density
• Use VMs when the workload requires: non-Linux OS, full kernel isolation, PCIe passthrough, or live migration
• Use OCI containers (PVE 9.1+, tech preview) for single-purpose microservices from Docker Hub/GHCR — lightweight

deployment without a Docker VM. Not suitable for multi-container stacks or workloads requiring Docker Compose

• Use a Docker VM for full Docker/Compose/Kubernetes workflows, multi-container stacks, or advanced networking

(macvlan, overlay). Still the most flexible and compatible option for container-heavy workloads

• Use unprivileged containers (default) — they map container UID 0 to a non-privileged host UID, preventing

container escape attacks

• Only use privileged containers when unprivileged mode is incompatible (specific device access, certain NFS mounts)

VM Configuration

• Use VirtIO drivers for disk and network — mandatory for production performance
• Enable QEMU Guest Agent inside every VM for graceful shutdown, snapshot consistency, and IP address reporting
• Use OVMF (UEFI) + Q35 machine type for PCIe passthrough and Secure Boot
• Set CPU type to host for maximum performance in homogeneous clusters; use x86-64-v2-AES or the lowest common CPU

generation when live migration across different CPU generations is required

• Enable memory ballooning for dynamic RAM management
• Enable NUMA topology for VMs with many cores on multi-socket hosts

Container Configuration

• Set explicit memory limits — containers without limits can exhaust host RAM
• Enable nesting (features: nesting=1) only when required (Docker inside LXC)
• For Docker workloads in LXC: unprivileged + nesting=1 + keyctl=1 — but note this is unsupported and can break

on host updates (CVE-2025-52881 broke Docker-in-LXC setups; workaround: lxc.apparmor.profile: unconfined)

• Use bind mounts to share host directories, not NFS/CIFS mounts inside the container
• PVE 9.0 removed cgroup v1 entirely — containers requiring cgroup v1 must move to VMs

OCI Application Containers (PVE 9.1 — Tech Preview)

• Pull OCI images from Docker Hub/GHCR/Quay and run as LXC containers — no Docker engine required
• Limitations: no in-place updates, no Docker Compose, no orchestration, no shell in most containers
• Use for: single-purpose lightweight services; use a Docker VM for multi-container stacks
• See [${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md] for full OCI details, Docker-on-Proxmox decision guide, and

Docker VM best practices

Templates and Cloning

• Build templates with the guest agent and cloud-init pre-installed
• Convert to template: qm template <vmid> or backup as .tar.gz for containers
• Use linked clones for development/ephemeral workloads (fast, shared base disk)
• Use full clones for production (independent, no template dependency)

</guest-management>

Storage

<storage>

Backend Selection

• Local redundancy, data integrity, snapshots → ZFS
• Local snapshots/clones without ZFS overhead → LVM-Thin
• Shared storage for HA clusters (3+ nodes) → Ceph
• Simple shared storage (existing NAS/SAN) → NFS or iSCSI
• Deduplicated backups → PBS

ZFS Rules

• Use HBA in IT mode — never hardware RAID controllers with write-back cache
• ARC sizing: cap explicitly in /etc/modprobe.d/zfs.conf; 25-30% of host RAM for mixed workloads. PVE 8.1+

defaults to 10% (max 16GB) for new installs. Rule of thumb: 2GB base + 1GB per TB of storage

• Set ashift=12 for 4K sector disks (most modern drives); incorrect ashift halves IOPS due to sector misalignment
• Volblocksize (VMs on zvols): 16k for mirrors/RAID10; increase for wide RAIDZ to reduce write amplification. Can

only be set at zvol creation

• Recordsize (containers on datasets): tune per workload — 8k for Postgres, 16k for MariaDB, 1M for large sequential

files

• Enable compression=lz4 — can actually increase I/O performance by writing less data to disk
• SLOG: 8-32GB enterprise NVMe with power-loss protection; only holds ~5s of write data. Benefits sync-heavy

workloads (databases, NFS)

• L2ARC: only if ARC hit ratio is low and adding RAM is not possible; size 5-20x RAM; budget 1GB RAM per 50GB of

L2ARC for metadata

• Schedule regular scrubs (weekly or monthly)
• Plan capacity upfront — ZFS pools cannot be shrunk
• Never create swap on a ZFS zvol — causes blocking I/O during backups

Ceph Rules

• Minimum 3 nodes for proper quorum and data distribution
• Dedicated 10GbE+ network for Ceph traffic (25GbE+ recommended for NVMe OSDs), separate from Corosync — Ceph rebalance

traffic saturates links and causes Corosync instability

• Use SSD/NVMe for OSD WAL/DB
• Dedicated disks for OSDs — never share with the host OS
• PVE 9.0 defaults to Ceph Squid (v19.2)

Storage Anti-Patterns

• Running ZFS or Ceph on top of hardware RAID with write-back cache — defeats data integrity guarantees
• Overprovisioning LVM-Thin without monitoring — a full thin pool causes I/O errors for all guests on that pool
• Storing backups on the same physical disks as production data
• Letting ZFS ARC consume unbounded RAM — causes VM crashes when the hypervisor and VMs compete for memory

</storage>

Networking

<networking>

Core Model

Every guest connects to a Linux bridge. Use VLAN-aware bridges (single bridge with 802.1Q tagging) instead of per-VLAN bridges. The VLAN-aware checkbox must be explicitly enabled — it is off by default.

VLAN Best Practices

• Place the management interface on a dedicated VLAN — never share with guest traffic
• Configure trunk ports on physical switches for the Proxmox host — frames with VLAN IDs not allowed by the switch

are silently dropped

• Assign VLAN tags per guest in the network configuration
• Verify no double-tagging mismatch between Proxmox and switch native VLAN

Traffic Separation

• Management / Corosync — 1GbE (dedicated)
• Ceph cluster + public — 10GbE (dedicated), 25GbE recommended
• Migration — 10GbE (recommended)
• Guest — depends on workload

Critical rule: Never combine Corosync traffic with high-bandwidth Ceph or migration traffic on a single 1GbE link. Corosync is latency-sensitive — network contention causes cluster instability and false fencing.

SDN

Use SDN (VXLAN zones) for overlay networking across nodes without physical switch changes. Use EVPN for advanced multi-tenant setups with BGP routing. SDN is fully supported and installed by default since PVE 8.1.

SDN gotchas:

• VXLAN adds 50-byte header — set VNet MTU to 1450 (or 1370 with IPSEC)
• SDN changes are staged, not live — click Apply at Datacenter level
• DHCP requires a gateway configured on the subnet
• Multiple EVPN exit nodes require disabling rp_filter in sysctl
• OVS vs Linux bridge: OVS is automatically VLAN-aware and may resolve 10GbE throughput bottlenecks seen with native

Linux bridge

SDN-Firewall integration (PVE 8.3+): SDN automatically generates IPSets for VNets and IPAM-managed guests — use these in firewall rules for simplified maintenance. The nftables firewall can filter forwarded traffic at host and VNet levels (e.g., restrict SNAT or inter-zone traffic).

Fabrics (PVE 9.0+): Automated routing between cluster nodes using FRRouting with OpenFabric (IS-IS-based) or OSPF. Fabrics simplify underlay network configuration for Ceph full-mesh and EVPN/VXLAN deployments.

</networking>

Clustering and High Availability

<clustering>

Cluster Rules

• Use a dedicated network for Corosync — latency under 5ms required
• Configure redundant Corosync links (up to 8 supported via Kronosnet) on separate physical networks
• For 2-node clusters, deploy a QDevice on a third machine for quorum — a 2-node cluster without QDevice is a

split-brain generator

• QDevice is discouraged for odd-numbered clusters — it becomes a single point of failure due to (N-1) vote allocation
• If using LACP bonds for Corosync, set bond-lacp-rate fast on both node and switch — default slow rate has 90s

failover, causing fencing after ~60s

• Avoid balance-rr, balance-xor, balance-tlb, balance-alb bond modes for Corosync — they cause asymmetric

connectivity and mass fencing

• Never join a node with existing VMs/containers to a cluster — start fresh
• Update nodes one at a time — the LRM requests a service freeze from the CRM during updates; if both are updating,

the watchdog fences the node

HA Requirements

• Shared or replicated storage accessible from all HA nodes
• Working fencing mechanism — test before relying on HA
• Minimum 3 quorum votes (3 nodes, or 2 nodes + QDevice)
• Configure HA groups with node priorities for controlled failover

Fencing

Fencing guarantees a failed node is offline before its services restart elsewhere. HA without fencing is a data corruption risk — two nodes writing to shared storage simultaneously causes irrecoverable damage.

• Verify watchdog status: ha-manager status
• Test fencing by simulating node failure before going to production
• Use hardware watchdog (iTCO_wdt via /etc/default/pve-ha-manager) when available, software watchdog (softdog) as

fallback

• Configure WATCHDOG_MODULE=iTCO_wdt in /etc/default/pve-ha-manager

Quorum Loss Recovery

If the cluster loses quorum, pmxcfs becomes read-only — no VM operations are possible. Emergency recovery:

• pvecm expected 1 forces single-node quorum — use only to restore vital guests or fix the quorum issue itself
• Never make cluster changes (add/remove nodes, storage, guests) while expected votes are overridden

Migration

• Live migration (VMs only): requires shared/replicated storage, brief pause at cutover
• vGPU live migration (PVE 8.4+): VMs using NVIDIA vGPU (mediated devices) can now be live-migrated between nodes

with compatible GPU hardware — previously required shutdown

• VMs with full PCIe passthrough devices still cannot be live-migrated — use cluster-wide resource mappings for HA
• Offline migration (VMs and containers): guest stops, data transfers, guest starts on target
• Use a dedicated high-bandwidth network for migration traffic

</clustering>

API and Automation

<automation>

Authentication

• Use API tokens with privilege separation for all automation — never root credentials
• Store token secrets in vault or environment variables — never in code
• Use ticket authentication only for interactive or short-lived tools

CLI Tools

• `pvesh` — direct REST API access from CLI
• `qm` — VM lifecycle management
• `pct` — container lifecycle management
• `ha-manager` — HA resource management
• `pvecm` — cluster management
• `pvesm` — storage management

Terraform

• Use API tokens with privilege separation — never root credentials
• Use cloud-init templates as the base for Terraform-managed VMs — templates must have qemu-guest-agent installed or

Terraform hangs on "still creating"

• Store Terraform state remotely (GitLab HTTP backend, S3, Consul) — never local state for shared infrastructure
• Use lifecycle { ignore_changes } for fields Proxmox modifies outside Terraform (e.g., disk size after manual resize)
• CI/CD pipeline: validate -> plan (save artifact) -> apply (manual trigger)
• Never commit API secrets to git — use CI/CD variables (TF_VAR_*)

Cloud-Init

• Prepare a base VM with qemu-guest-agent and cloud-init installed
• Add a Cloud-Init drive (IDE or SCSI CD-ROM)
• Configure network, SSH keys, and user data via the Cloud-Init panel or API
• Convert to template, deploy via linked clone
• Use SSH key authentication — cloud-init password storage is less secure
• cicustom for advanced needs: reference custom YAML snippets for user, network, and meta data from a

snippets-capable storage

• Store cicustom snippets on shared storage (CephFS) in clusters for HA
• Windows templates: use Cloudbase-Init with configdrive2 format + Sysprep

</automation>

Backups

<backups>

Backup Rules

• Use PBS for production — deduplication, incremental backups, verification, encryption
• Use snapshot mode for VMs (crash-consistent, no downtime); zstd compression
• Follow the 3-2-1 rule: 3 copies, 2 media types, 1 off-site
• Retention: keep-daily=7,keep-weekly=4,keep-monthly=6,keep-yearly=1

PBS Security

• Restrict PVE backup user/token to create-only access (no delete) on PBS
• Separate PBS admin credentials from PVE access
• Store encryption keys separately from the backed-up system — password manager + offline backup
• Never disable gc-atime-safety-check; use dedicated remote users per sync job

Non-Negotiable

• Test restores regularly. A backup that cannot be restored is worthless.
• Monitor backup jobs. A silently failing backup is worse than no backup.
• Document the restore procedure.

</backups>

PCIe Passthrough

<passthrough>

Requirements

• CPU: VT-d (Intel) or AMD-Vi enabled in BIOS/UEFI
• IOMMU enabled in kernel: intel_iommu=on or amd_iommu=on
• VFIO modules loaded: vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd
• Dedicated IOMMU group for the passthrough device

Configuration

• Use OVMF (UEFI) + Q35 machine type; SeaBIOS if GPU lacks UEFI ROM
• Blacklist host driver or bind via vfio-pci IDs in /etc/modprobe.d/
• Pass through all device functions — GPU requires video + audio; USB-C controllers must also be bound to vfio-pci
• For GPU: x-vga=1 for primary, vga: none; output via physical monitor, dummy plug, or Looking Glass
• VMs with full passthrough cannot be live-migrated — use resource mappings (/cluster/mapping/pci) for HA

GPU-Specific Issues

• NVIDIA Error 43 (Windows): set CPU type to host, add options kvm ignore_msrs=1 in /etc/modprobe.d/kvm.conf
• AMD reset bug (Vega, Polaris, some Navi): GPU fails to reset after VM shutdown, preventing reuse without host

reboot. Fix: install vendor-reset kernel module for vendor-specific reset quirks. RDNA2+ generally unaffected

• NVIDIA vGPU: officially supported since vGPU Software 18 on PVE. Requires valid NVIDIA entitlement. Ampere+ GPUs

need SR-IOV enabled first via pve-nvidia-vgpu-helper. PVE 8.4+ supports live migration of vGPU VMs

Virtiofs Directory Passthrough (PVE 8.4+)

Host-to-guest file sharing via virtiofs — bypasses network filesystems, provides near-native performance. Linux guests support virtiofs natively; Windows guests require a guest driver. Use for workloads requiring frequent host-guest file exchange without the overhead of NFS/SMB.

LXC Device Passthrough

Since PVE 8.2, device passthrough for containers is configurable via the UI. Limited compared to VM passthrough — no full PCIe passthrough, but supports specific device access (GPU rendering, USB devices).

Troubleshooting

• dmesg | grep -e DMAR -e IOMMU — verify IOMMU is enabled
• pvesh get /nodes/{node}/hardware/pci --pci-class-blacklist "" — list groups
• Multi-device IOMMU groups (B550/X570): pcie_acs_override=downstream,multifunction as workaround (not for production)
• IOMMU groups can change between kernel major versions — verify after updates

</passthrough>

Security

<security>

Certificates

• Replace self-signed certs with ACME (Let's Encrypt) — DNS-01 for nodes behind firewalls, HTTP-01 for

internet-reachable

• Trusted certificates required for reliable WebAuthn (FIDO2)

Firewall

Proxmox VE includes a distributed firewall (iptables-based, nftables opt-in since PVE 8.2) at datacenter, host, and guest levels. The nftables backend (PVE 8.3+) supports filtering forwarded traffic at host and VNet levels.

• Enable selectively — datacenter first, then host, then per-interface. Disabled by default at all levels
• Before enabling, create rules for management access (8006, 22, 3128) — keep an SSH session open as safety net
• Create IPSet `management` with trusted admin IPs — auto-generates management access rules
• Use security groups for reusable rule sets across VMs
• Use *`ipfilter-net` IPSets** per VM interface to prevent IP spoofing

User Management and RBAC

• Grant permissions to groups, not individuals
• Use resource pools to group related resources; assign permissions at pool level
• Use realms for auth: PAM, LDAP, AD, OpenID Connect (Keycloak, Authentik)
• Use privilege-separated API tokens for automation
• Enforce 2FA (TOTP, YubiKey, WebAuthn) at realm level

Hardening

• Use Enterprise repository for production
• Restrict management interface access (dedicated VLAN, firewall rules)
• Disable root SSH; use non-root with sudo
• Do not run Docker or other services directly on the PVE host

</security>

Monitoring

<monitoring>

• Export metrics to InfluxDB/Graphite (Datacenter > Metric Server); visualize with Grafana (dashboard 10048)
• Configure notification matchers for: backup failures (vzdump), fencing/HA events, replication failures
• Enable ZED for ZFS errors, smartmontools for disk health, Ceph health monitoring
• Notification targets: Sendmail, SMTP, Gotify, Webhooks (PVE 8.3+ — any HTTP endpoint with custom headers/body)

</monitoring>

Anti-Patterns

Non-obvious traps where the "right" approach is counterintuitive:

• Swap on ZFS zvol → partition a physical disk for swap; zvol swap causes blocking I/O during backups
• LACP bonds for Corosync with default rate → set bond-lacp-rate fast (default 90s failover > 60s fence timeout)
• Load-balancing bond modes for Corosync → use active-backup or LACP with fast rate; load-balancing modes cause

asymmetric connectivity and mass fencing

• PBS encryption key on backed-up system → store in password manager + offline backup; compromised host means

compromised backups

• Shared remote user for PBS sync jobs → dedicated remote user per sync job; shared users bypass

gc-atime-safety-check

Application

<application>

When Configuring Proxmox

• Apply conventions without narrating each rule; follow existing environment patterns
• Security practices are defaults, not optional add-ons
• Test changes in non-production when possible

When Reviewing Configuration

• Check guest type matches workload (VM vs LXC vs OCI decision)
• Verify storage backend matches workload characteristics (ZFS tuning, Ceph sizing, LVM-Thin monitoring)
• Verify network separation: management, Corosync, Ceph, guest traffic on appropriate interfaces
• Verify HA prerequisites: shared storage, fencing tested, quorum math correct, HA groups configured
• Verify backup coverage: PBS for production, retention policy set, restores tested, encryption keys stored separately
• Check security posture: API tokens with privsep, 2FA enforced, firewall enabled, management VLAN isolated
• Check monitoring: metric export configured, notification matchers for backup/fencing/replication failures

When Writing Automation

• Use pvesh or REST API — never screen-scrape the GUI
• Use API tokens with privilege separation; handle task status polling
• Use cloud-init templates; write idempotent scripts

</application>

Integration

This skill provides Proxmox VE discipline alongside sibling skills in the infrastructure plugin:

• `devops` — foundational discipline (IaC principles, change management, observability) that applies to all Proxmox

work

• `networking` — general network infrastructure (VLANs, firewalls, DNS) beyond PVE-specific networking covered here
• `ansible` — configuration management for automating Proxmox host setup and post-provisioning
• `containers` — Docker/Podman management inside PVE guests (VMs or LXC)

Proxmox Datacenter Manager (PDM 1.0): Centralized management for multiple independent PVE/PBS environments — aggregated views, cross-cluster live migration, EVPN configuration between clusters, centralized update overview. Written in Rust. Requires PVE 8.4+ / PBS 3.4+. Relevant for multi-site or large-scale deployments.

Critical Rules

• HA without fencing is a data corruption risk — never enable HA without a tested fencing mechanism
• Cap ZFS ARC explicitly — unbounded ARC causes VM OOM crashes
• Dedicated Corosync network — never share with Ceph or migration traffic on 1GbE
• Test restores regularly — a backup that cannot be restored is worthless
• API tokens with privilege separation for all automation — never root credentials
• Use VirtIO drivers for all production VM disks and network interfaces

Proxmox is infrastructure. Treat it with the same rigor as production code: version-controlled configuration, tested changes, monitored operations, documented decisions.

API and Automation

Proxmox VE exposes a comprehensive REST API that maps 1:1 to all GUI operations. Every action available in the web interface is available programmatically.

API Architecture

• RESTful JSON API over HTTPS (port 8006)
• Formally defined via JSON Schema — enables auto-generated clients
• Every API endpoint corresponds to a filesystem-like path (e.g., /nodes/{node}/qemu/{vmid})
• API documentation: https://<host>:8006/pve-docs/api-viewer/

Authentication

API Tokens (Recommended)

API tokens provide non-interactive authentication without exposing user credentials. Each token has an ID and a secret value shown once at creation.

pveum user token add root@pam automation --privsep=1

• Privilege separation (privsep=1, default): Token permissions are the intersection of the user's permissions and

the token's explicit permissions. Always use privilege separation for least-privilege access.

• No privilege separation (privsep=0): Token inherits full user permissions. Avoid for automation — overly broad

access.

• Tokens do not expire by default. Set expiration for temporary access.
• Store token secrets securely (vault, environment variables) — never in code or version control.

Using tokens in API calls:

Authorization: PVEAPIToken=root@pam!automation=<secret-uuid>

Ticket Authentication

For interactive or short-lived sessions, obtain a ticket via /access/ticket:

curl -k -d "username=root@pam&password=<password>" \
  https://<host>:8006/api2/json/access/ticket

Returns a CSRFPreventionToken (for PUT/POST/DELETE) and a PVEAuthCookie. Tickets expire after 2 hours.

Use API tokens for automation. Use tickets only for interactive tools or short-lived scripts.

pvesh — CLI API Client

pvesh is the built-in CLI tool that maps directly to the REST API. Every API endpoint is accessible via pvesh.

# List VMs on a node
pvesh get /nodes/pve1/qemu

# Create a VM
pvesh create /nodes/pve1/qemu -vmid 100 -name myvm -memory 2048 \
  -cores 2 -net0 virtio,bridge=vmbr0 -scsi0 local-lvm:32

# Start a VM
pvesh create /nodes/pve1/qemu/100/status/start

# Get VM status
pvesh get /nodes/pve1/qemu/100/status/current

# Delete a VM
pvesh delete /nodes/pve1/qemu/100

# Resize a disk
pvesh set /nodes/pve1/qemu/100/resize -disk scsi0 -size +10G

pvesh verbs map to HTTP methods:

• `get` → GET — read/list resources
• `create` → POST — create resources or trigger actions
• `set` → PUT — update resources
• `delete` → DELETE — remove resources

Output formats:

• Default: human-readable table
• --output-format json: machine-parseable JSON
• --output-format json-pretty: formatted JSON

qm and pct — Guest CLI Tools

qm (VM management)

qm list                          # List all VMs
qm create 100 --name myvm ...   # Create VM
qm start 100                    # Start VM
qm stop 100                     # Stop VM (hard)
qm shutdown 100                 # Graceful shutdown (requires guest agent)
qm destroy 100                  # Delete VM and disks
qm config 100                   # Show VM configuration
qm set 100 --memory 4096        # Update VM config
qm template 100                 # Convert to template
qm clone 100 101 --name clone1  # Clone VM
qm migrate 100 node2            # Migrate VM
qm snapshot 100 snap1           # Create snapshot
qm rollback 100 snap1           # Rollback to snapshot

pct (Container management)

pct list                         # List all containers
pct create 200 local:vztmpl/... # Create container from template
pct start 200                   # Start container
pct stop 200                    # Stop container
pct destroy 200                 # Delete container
pct config 200                  # Show container configuration
pct set 200 --memory 2048       # Update container config
pct enter 200                   # Open shell in container
pct push 200 /local /ct/path    # Copy file into container
pct pull 200 /ct/path /local    # Copy file from container
pct snapshot 200 snap1          # Create snapshot
pct clone 200 201 --hostname x  # Clone container

Terraform Integration

The Telmate Terraform provider enables Infrastructure-as-Code management of Proxmox resources.

Best practices:

• Use API tokens with privilege separation — never use root credentials
• Store state remotely (S3, Consul) — never in local files for shared infrastructure
• Use cloud-init templates as the base for Terraform-managed VMs
• Define resource pools in Proxmox to organize Terraform-managed resources
• Use lifecycle { ignore_changes } for fields that Proxmox modifies outside Terraform (e.g., disk size after manual

resize)

Automation Patterns

Bulk Operations

# Start all VMs in a pool
pvesh get /pools/production --output-format json | \
  jq -r '.members[] | select(.type=="qemu") | .vmid' | \
  xargs -I{} qm start {}

# Snapshot all VMs before maintenance
for vmid in $(qm list | awk 'NR>1 {print $1}'); do
  qm snapshot $vmid "pre-maintenance-$(date +%Y%m%d)"
done

Hookscripts

Hookscripts execute at guest lifecycle events (pre-start, post-start, pre-stop, post-stop). They are stored in a snippets-capable storage.

# Assign hookscript to VM
qm set 100 --hookscript local:snippets/myhook.pl

Hookscript receives: $vmid $phase where phase is pre-start, post-start, pre-stop, or post-stop.

Scheduled Tasks

Use systemd timers or cron on the Proxmox host for scheduled operations:

• Backup schedules (vzdump)
• ZFS scrubs
• Replication schedules
• Custom maintenance scripts

Prefer vzdump's built-in scheduler (configured via GUI or /etc/pve/jobs.cfg) over manual cron entries for backups.

API Performance Scaling

For automation-heavy workloads with high API request volume, increase the number of worker processes to prevent slow request handling or timeouts:

# /etc/default/pvedaemon
MAX_WORKERS=8

# /etc/default/pveproxy
MAX_WORKERS=8

Default is 3 workers for each daemon. Restart the relevant service after changes.

Resource Mappings

For clusters using PCI/USB passthrough with HA, define cluster-wide resource mappings to decouple device configuration from hardware-specific addresses:

pvesh create /cluster/mapping/pci --id mygpu --map "node=pve1,path=0000:01:00"

This assigns a cluster-unique ID to hardware, so HA failovers and automated processes reference the mapping ID rather than a host-specific PCI address. Proxmox verifies the hardware matches the mapping on each node.

Backup Strategies

Proxmox VE provides built-in backup via vzdump and integrates with Proxmox Backup Server (PBS) for advanced backup management. Reliable backups are a non-negotiable requirement for any production environment.

vzdump — Built-in Backup Tool

vzdump creates consistent backup archives of VMs and containers, including disk data and configuration files.

Backup Modes

• Snapshot mode (recommended for VMs): Uses QEMU's live snapshot capability. The VM continues running. Result is

crash-consistent — equivalent to a power loss recovery. Applications with journaling (databases, filesystems) handle this well.

• Suspend mode: Suspends the guest, copies data, then resumes. Brief downtime but guaranteed memory consistency.
• Stop mode: Stops the guest completely, backs up, restarts. Longest downtime but cleanest backup. Required for some

LXC containers.

PVE 8.3+ container backup optimization: When backing up containers to PBS, PVE can detect unchanged files since the last snapshot and skip them — significantly faster container backups for large, mostly-static filesystems.

PVE 8.4+ backup fleecing: Refined mechanism that reduces I/O impact on running VMs during live backup operations — prevents the I/O stalls that could occur with large, write-heavy VMs during snapshot backups.

Backup Compression

• `lzo` — fast speed, low ratio, low CPU
• `gzip` — medium speed, medium ratio, medium CPU
• `zstd` — fast speed, high ratio, medium CPU

Use zstd as the default — it provides the best balance of compression ratio and speed. Specify with --compress zstd.

Backup Scheduling

Configure backup jobs via the GUI (Datacenter > Backup) or /etc/pve/jobs.cfg:

vzdump: backup-daily
    enabled 1
    schedule daily 02:00
    storage pbs-store
    mailnotification always
    mailto admin@example.com
    mode snapshot
    compress zstd
    all 1
    exclude 9000,9001
    notes-template {{guestname}} - {{cluster}}

Scheduling best practices:

• Run backups during low-usage windows (overnight)
• Stagger backup start times across storage targets to avoid I/O contention
• Exclude templates and ephemeral VMs (exclude parameter)
• Enable email notifications (mailnotification always) for failure alerts
• Use prune-backups to define retention policy inline

Retention

Configure retention per backup job or per storage:

prune-backups: keep-daily=7,keep-weekly=4,keep-monthly=6,keep-yearly=1

• `keep-last` — keep the N most recent backups
• `keep-hourly` — keep one backup per hour for N hours
• `keep-daily` — keep one backup per day for N days
• `keep-weekly` — keep one backup per week for N weeks
• `keep-monthly` — keep one backup per month for N months
• `keep-yearly` — keep one backup per year for N years

Retention rules are applied in order. A backup kept by keep-daily is also counted toward keep-weekly if applicable. The pruning algorithm preserves the most diverse time distribution.

Proxmox Backup Server (PBS)

PBS is a dedicated backup solution that provides enterprise features beyond what vzdump-to-directory offers.

Key Advantages Over Directory Backups

PBS Architecture

PBS uses content-addressable storage — data is split into fixed-size chunks, each identified by its hash. Identical chunks across any backup (same VM, different VMs, different times) are stored only once.

Result: After the initial full backup, subsequent backups transfer and store only changed chunks. Storage usage grows proportionally to actual data change rate, not backup count.

Datastore Configuration

# Create a datastore
proxmox-backup-manager datastore create mystore /mnt/backup-storage

# Configure garbage collection schedule
proxmox-backup-manager datastore update mystore --gc-schedule daily

# Configure verification schedule
proxmox-backup-manager datastore update mystore --verify-schedule weekly

Encryption

PBS supports client-side encryption — data is encrypted before leaving the Proxmox VE node. The PBS server never sees plaintext data.

• Generate an encryption key: proxmox-backup-client key create --kdf scrypt
• Store the key securely — losing the key means losing access to encrypted backups
• The master key can optionally be split with Shamir's Secret Sharing for key recovery

Ransomware Protection

PBS provides architectural protection against ransomware that compromises the Proxmox VE host:

• Restricted API access: The PVE backup user/token can create backups and list snapshots but **cannot delete

backups** on PBS

• Separate admin credentials: PBS admin access is separate from PVE access — compromising PVE does not grant PBS

admin rights

• Verification jobs: PBS can verify backup integrity independently of PVE
• Immutable backups: Configure datastore namespaces with delete restrictions

Third-Party Backup API (PVE 8.4+)

PVE 8.4 introduces an official backup plugin API that allows external backup providers to integrate directly into the Proxmox UI and backup framework. Third-party solutions can register as backup plugins, enabling backup and restore operations through the same interface as native PBS — including scheduling, verification, and management features. This gives administrators flexibility to use non-Proxmox backup services with first-class integration.

Connecting PVE to PBS

Add PBS as a storage in PVE:

pvesm add pbs pbs-store \
  --server 10.0.0.50 \
  --datastore mystore \
  --username backup@pbs!automation \
  --password <token-secret> \
  --fingerprint <server-fingerprint>

Backup Best Practices

• Test restores regularly. A backup that cannot be restored is worthless. Schedule quarterly restore tests.
• Follow the 3-2-1 rule: 3 copies of data, on 2 different media types, with 1 off-site. PBS + local directory +

off-site PBS replication satisfies this.

• Use PBS for production. Directory/NFS backups lack deduplication, incrementals, and verification. They are

acceptable only for lab environments.

• Separate backup storage. Never store backups on the same physical disks as production data.
• Monitor backup jobs. Enable email notifications and regularly check backup job status. A silently failing backup

is worse than no backup.

• Document the restore procedure. Include: where backups are stored, credentials needed, the exact restore commands,

expected recovery time.

• Encrypt off-site backups. Any backup leaving your physical control must be encrypted with keys you manage.

Clustering and High Availability

Proxmox VE clustering connects multiple nodes into a unified management domain with shared configuration, live migration, and optional high availability (HA) for automatic failover of guests.

Cluster Fundamentals

Corosync

Corosync provides the cluster communication layer — membership, messaging, and quorum. All cluster nodes must be able to reach each other via Corosync.

Configuration rules:

• Use a dedicated network for Corosync traffic — never share with Ceph, guest, or high-bandwidth traffic on a single

1GbE link

• Configure redundant Corosync links (two separate networks) for fault tolerance
• Corosync uses UDP multicast or unicast — unicast is required in most cloud/VLAN setups
• Default Corosync port: 5405-5412 (UDP)
• Latency between nodes must be low (<2ms recommended) — Corosync is latency-sensitive
• Use static IP addresses for cluster addresses, not hostnames — DNS resolution delays or changes can cause cluster

communication failures

• If using LACP bonds for Corosync, set bond-lacp-rate fast on both node and switch. Default slow rate has a

90-second failover time — HA resources fence the node after ~60 seconds of lost quorum, so slow LACP triggers unnecessary fencing

• Avoid balance-rr, balance-xor, balance-tlb, and balance-alb bond modes for Corosync — they are known to cause

problems in failure scenarios

Quorum

A cluster requires a majority of votes (quorum) to operate. Without quorum, the cluster enters read-only mode to prevent split-brain.

Two-Node Clusters

Two-node clusters have no fault tolerance by default — losing one node means losing quorum. Solutions:

• QDevice (recommended): An external lightweight daemon (corosync-qdevice) running on a third machine provides a

tie-breaking vote. The QDevice host does not need to be a Proxmox node — any Linux machine works.

  pvecm qdevice setup <qdevice-ip>

• Manual quorum override (dangerous): pvecm expected 1 forces the remaining node to operate without quorum. Only

use during planned maintenance, never as a permanent solution.

Adding and Removing Nodes

Adding a node:

# On the new node:
pvecm add <existing-node-ip>

• The new node must have a fresh Proxmox installation — never join a node with existing VMs/containers to an existing

cluster

• Ensure network connectivity (Corosync, SSH) between all nodes before joining

Removing a node:

1. Migrate all guests off the node 2. On the node being removed: pvecm nodes to verify membership 3. Shut down the node 4. On a remaining node: pvecm delnode <nodename> 5. Remove SSH keys and known_hosts entries

High Availability (HA)

HA automatically restarts guests on another node when their host fails. It requires shared storage (Ceph, NFS, iSCSI) or replicated storage (ZFS replication).

How HA Works

1. Watchdog monitors node health via hardware watchdog timer 2. Fencing ensures a failed node is completely stopped before services restart elsewhere — this prevents simultaneous writes to shared storage (split-brain) 3. HA Manager (CRM) detects the failure and migrates affected HA resources to surviving nodes according to group and priority settings

Fencing

Fencing is the mechanism that guarantees a failed node is truly offline. Without reliable fencing, HA is unsafe — restarting a VM on another node while the original is still running causes data corruption.

Fencing methods:

• Hardware watchdog (default): Linux kernel software watchdog (softdog) or IPMI hardware watchdog resets the

failed node

• IPMI/iLO/iDRAC: Out-of-band management interface power-cycles the failed node
• External fencing agents: Custom scripts for PDU/UPS power control

Rules:

• Verify the watchdog is active: ha-manager status should show watchdog OK
• Test fencing before relying on HA: simulate a node failure and verify the node actually reboots/shuts down
• Never disable fencing — HA without fencing is a data corruption risk

HA Groups

HA groups define which nodes can run specific resources and set failover priority.

ha-group create mygroup -nodes "node1:2,node2:1,node3:1" -nofailback 0

• Priority (higher = preferred): Node with highest priority runs the resource under normal conditions
• Failback (nofailback: 0): Resource migrates back to the preferred node when it recovers. Set nofailback: 1 to

prevent automatic failback.

• Restricted groups: Only nodes in the group can run the resource. If all group nodes are down, the resource stays

down rather than running on an ungrouped node.

HA Resource Configuration

ha-manager add vm:100 -group mygroup -state started -max_restart 3 -max_relocate 2

• `state` — started, stopped, disabled, ignored
• `max_restart` — maximum restart attempts on the same node before relocating
• `max_relocate` — maximum relocations to other nodes before giving up
• `group` — HA group defining eligible nodes and priorities

HA Requirements Checklist

• [ ] Shared or replicated storage accessible from all HA nodes
• [ ] Minimum 3 nodes (or 2 nodes + QDevice) for quorum
• [ ] Working fencing mechanism tested and verified
• [ ] Dedicated Corosync network with redundant links
• [ ] HA groups configured with appropriate priorities
• [ ] Guest configuration stored on shared/replicated storage (automatic with pmxcfs)

Migration

Live Migration (VMs only)

• Requires shared storage or replicated local storage
• RAM contents transferred over the network while VM continues running
• Brief pause at cutover (typically milliseconds to seconds)
• Full PCIe passthrough: not possible — use cluster-wide resource mappings (/cluster/mapping/pci) for HA with

passthrough devices

• vGPU (mediated devices) live migration (PVE 8.4+): VMs using NVIDIA vGPU can now be live-migrated between nodes

with compatible GPU hardware and drivers. Previously required shutdown. Use pve-nvidia-vgpu-helper to simplify driver setup

• Use dedicated migration network for large-memory VMs

Virtiofs Directory Passthrough (PVE 8.4+)

PVE 8.4 introduces virtiofs for efficient host-to-guest file sharing:

• Bypasses network filesystems (NFS/SMB) — VM accesses host directories directly with minimal overhead
• Linux guests support virtiofs natively; Windows guests require a VirtIO guest driver
• Configured via VM hardware settings — add a virtiofs mount point referencing a host directory
• Use for development workflows, build artifacts, shared configuration, or any scenario with frequent host-guest file

exchange

Offline Migration

• Works for both VMs and containers
• Guest is stopped, disk data transferred, guest started on target node
• Required for LXC containers (no live migration support)
• Works with local non-replicated storage (disk is copied)

Migration Best Practices

• Use a dedicated high-bandwidth network (10GbE+) for migration traffic
• Set migration network in datacenter options: migration: network=10.0.1.0/24
• For large VMs, consider storage migration (move disk between storage backends) rather than node migration
• Test migration before relying on it for HA — verify network throughput and storage accessibility

Cluster Maintenance

Rolling Updates

Update cluster nodes one at a time — never simultaneously. The HA stack uses a request-acknowledge protocol between the Cluster Resource Manager (CRM) and Local Resource Manager (LRM). During an update, the LRM requests a service freeze from the CRM; if the CRM is also being updated, the request cannot be acknowledged, and the watchdog may fence the node.

Procedure:

1. Migrate HA resources off the target node (or let HA handle it) 2. Update the node: apt update && apt dist-upgrade 3. Reboot if a kernel update was applied 4. Verify the node is fully functional: pvecm status, ha-manager status 5. Proceed to the next node only after confirmation

Networking

Proxmox VE networking connects guests to each other and external networks via Linux bridges, VLANs, bonds, and Software-Defined Networking (SDN). Network configuration is defined in /etc/network/interfaces and applied with ifreload -a.

Network Model

Every Proxmox node has at least one Linux bridge (vmbr0) connecting guest virtual NICs to a physical NIC. Guests attach to bridges, not directly to physical interfaces.

Physical NIC (eno1) --- Bridge (vmbr0) -+- VM 100 (virtio)
                                        +- VM 101 (virtio)
                                        +- CT 200 (veth)

Linux Bridges

• One bridge per network segment. Create additional bridges for isolated networks or different VLANs.
• VLAN-aware bridges (recommended): A single bridge handles multiple VLANs via 802.1Q tagging, eliminating the need

for separate bridges per VLAN. The VLAN-aware checkbox must be explicitly enabled — it is off by default.

  auto vmbr0
  iface vmbr0 inet static
      address 10.0.0.1/24
      bridge-ports eno1
      bridge-stp off
      bridge-fd 0
      bridge-vlan-aware yes
      bridge-vids 2-4094

• Traditional bridges (legacy): One bridge per VLAN, using VLAN sub-interfaces. More complex to manage at scale.

OVS vs Linux Bridge

Open vSwitch (OVS) is an alternative to the native Linux bridge:

• Automatically VLAN-aware (no checkbox needed)
• Supports VXLAN and RSTP natively
• May resolve 10GbE+ throughput bottlenecks reported with native Linux bridge
• Preferred for complex SDN layouts

The native Linux bridge is simpler and adequate for most deployments. Consider OVS when hitting performance limits at 10GbE+ or needing advanced SDN features.

VLANs

VLANs (802.1Q) segment network traffic at Layer 2 without additional physical cabling.

Best practices:

• Use VLAN-aware bridges — simpler than per-VLAN bridges, configured once
• Assign VLANs per guest in the VM/container network configuration (tag field)
• Place the management interface on a dedicated VLAN — never share the management network with guest traffic
• Use a trunk port on the physical switch connected to the Proxmox host, allowing all required VLAN tags — frames

with unrecognized VLAN IDs are silently dropped

• Common VLAN layout:
• VLAN 1 (native) — Management / host access
• VLAN 10 — Server/infrastructure
• VLAN 20 — User/client network
• VLAN 30 — IoT/untrusted devices
• VLAN 50 — Storage (Ceph, NFS, iSCSI)
• VLAN 100 — DMZ / public-facing services

Troubleshooting VLANs:

• Verify no mismatch between Proxmox VLAN tagging and switch native/untagged VLAN — mismatches cause double-tagging and

dropped frames

• Ensure switch trunk allows all VLANs configured on the Proxmox side
• When moving management to a VLAN, verify bridge still allows management traffic before disconnecting

Network Bonding

Bonding combines multiple physical NICs for redundancy and/or throughput.

• `balance-rr` (0) — round-robin; throughput (requires switch support)
• `active-backup` (1) — active-backup; redundancy without switch config
• `balance-xor` (2) — XOR; balanced with switch LAG
• `802.3ad` (4) — LACP; best throughput + redundancy (requires switch LACP)
• `balance-alb` (6) — adaptive load balancing; throughput without switch config

Best practices:

• Use LACP (802.3ad) when the switch supports it — best combination of redundancy and throughput
• Use active-backup when switch configuration is not possible
• Bond before bridging: physical NICs -> bond -> bridge -> guests
• Use separate bonds for different traffic types (management, storage, guest)

Corosync bonding caveats:

• Avoid balance-rr, balance-xor, balance-tlb, balance-alb for Corosync — these modes cause asymmetric

connectivity when an interface fails but link state stays up, leading to mass fencing

• LACP with fast rate: If using 802.3ad for Corosync, set bond-lacp-rate fast on both node and switch. Default

slow rate has 90s failover — HA fences after ~60s. Fast rate reduces failover to ~3 seconds

• active-backup is safest for Corosync but may not fail over if the backup link detection is delayed

Software-Defined Networking (SDN)

SDN in Proxmox VE creates virtual network zones that span the cluster without requiring physical VLAN configuration on switches.

Zones

• Simple — L2, Linux bridge; single-node isolated networks
• VLAN — L2, 802.1Q tags; multi-node with physical VLAN support
• QinQ — L2, double-tagged VLANs; stacking VLANs over existing infrastructure
• VXLAN — L2/L3, UDP overlay; multi-node without physical VLAN changes
• EVPN — L2/L3, BGP + VXLAN; advanced routing, multi-tenancy

VNets

VNets are virtual networks within a zone. Each VNet gets a name and optional VLAN tag or VXLAN ID, and appears as a bridge-like interface for guest assignment.

SDN Configuration Gotchas

• Staged changes: SDN changes are not live — they are staged and require clicking Apply at the Datacenter

level to take effect cluster-wide

• VXLAN MTU: VXLAN encapsulation adds a 50-byte header. Set VNet MTU to 1450 (default auto setting). With IPSEC

encryption, reduce to 1370 (IPv4: -60 bytes, IPv6: -80 bytes). The physical network MTU must accommodate the overhead — if physical MTU is 1500, VNet MTU must be 1450 or lower

• DHCP prerequisite: Automatic DHCP within an SDN VNet requires a gateway configured on the subnet — without it,

DHCP silently fails

• Multiple EVPN exit nodes: Asymmetric routing causes Linux to drop packets. Disable reverse path filtering:

  # /etc/sysctl.conf (or /etc/sysctl.d/zzz-network.conf)
  net.ipv4.conf.all.rp_filter=0
  net.ipv4.conf.default.rp_filter=0

Note: default must be set before interfaces are created (use a zzz- prefix sysctl file to ensure late loading)

SDN-Firewall Integration (PVE 8.3+)

The SDN stack is now tightly integrated with the Proxmox firewall:

• Automatic IPSets: SDN auto-generates IPSets for each VNet (vnet-all, vnet-gateway, vnet-no-gateway) and for

IPAM-managed guests. Reference these in firewall rules for simplified maintenance.

• Forwarded traffic filtering (nftables): The opt-in nftables firewall can filter forwarded traffic at both host and

VNet levels — use to restrict SNAT traffic or traffic flowing between Simple Zones.

• IPAM integration: Guests managed by the IPAM plugin have their IPs automatically added to VNet IPSets.

EVPN Advanced Features

• Anycast IP/MAC: EVPN VNets support anycast addresses — the bridge IP is the same on every node, so guests use a

single gateway address regardless of which node they run on

• VRF routing: Routing between VNets in different zones works through Virtual Routing and Forwarding (VRF)

interfaces

• Inter-cluster EVPN: Proxmox Datacenter Manager (PDM 1.0) provides initial support for managing EVPN configurations

across physically distributed clusters

Fabrics (PVE 9.0+)

Fabrics automate routing between cluster nodes using FRRouting (FRR):

• OpenFabric — IS-IS-based, optimized for spine-leaf topologies
• OSPF — traditional link-state routing for flat or hierarchical networks

Fabrics provide the underlay network for:

• Ceph full-mesh connectivity between OSDs
• EVPN/VXLAN zone deployments
• Any scenario needing automatic, resilient inter-node routing

Fabrics auto-configure routing protocols on physical interfaces, creating a self-healing network that adapts to topology changes (node additions, link failures).

DHCP/IPAM Integration

SDN includes integrated DHCP management via dnsmasq, tied to the IPAM stack:

• IP allocation happens automatically during guest creation when using the PVE IPAM plugin
• MAC-to-IP mappings are created when the VM starts and cleaned up when destroyed
• Requires a gateway configured on the subnet — without it, DHCP silently fails

SDN Best Practices

• Use VXLAN zones for overlay networking across nodes without switch changes
• Use EVPN for advanced multi-tenant setups with BGP routing
• Use Fabrics to automate underlay routing — eliminates manual FRR configuration
• Apply SDN configuration changes via the GUI or pvesh — changes require explicit "Apply" to take effect cluster-wide
• Use SDN for tenant isolation in multi-tenant environments
• SDN is ideal when physical switch VLAN configuration is impractical or impossible

Dedicated Networks

Separate traffic types onto dedicated networks for performance and security:

• Management / Corosync — dedicated VLAN or NIC; 1GbE sufficient
• Ceph cluster — dedicated NIC(s); 10GbE minimum, 25GbE+ recommended
• Ceph public — dedicated NIC(s); 10GbE minimum
• VM/CT guest traffic — shared bridge; depends on workload
• Migration — dedicated or shared; 10GbE recommended for fast migration
• Backup — shared or dedicated; depends on backup window

Critical: Never combine Corosync cluster traffic with high-bandwidth Ceph or migration traffic on a single 1GbE link. Corosync requires latency under 5ms — network contention causes cluster instability, false fencing, and potential data loss. Ceph rebalance traffic (after disk/node failure) can saturate even 10GbE links with NVMe OSDs.

Firewall Lockout Prevention

Enabling the Proxmox firewall blocks all traffic by default except ports 8006 and 22 from the local network only.

Before enabling the firewall:

1. Create an IPSet named management with trusted admin IPs — Proxmox auto-generates management access rules from this set 2. Add explicit rules for ports 8006 (GUI), 22 (SSH), 3128 (SPICE) from remote IPs 3. Keep an active SSH session open as a safety net 4. Enable firewall at datacenter level first, then host, then per-interface

VM firewall: Each virtual NIC has its own firewall enable flag — the firewall must be enabled both globally and on each interface for rules to apply.

Storage Backends

Proxmox VE supports multiple storage backends, each with different capabilities and trade-offs. Storage selection directly impacts performance, redundancy, snapshot support, and cluster compatibility.

Storage Types Overview

PVE 9.0 change: LVM thick now supports VM snapshots via "snapshots as volume chains" — qcow2 layers separate volumes without the I/O degradation of traditional LVM snapshots.

Content Types

Each storage can host specific content types:

• `images` — VM disk images; all block/file storage
• `rootdir` — container root filesystems; Directory, NFS, ZFS, LVM-Thin
• `iso` — ISO images for VM installation; Directory, NFS, CIFS
• `vztmpl` — container templates; Directory, NFS, CIFS
• `backup` — backup files (vzdump); Directory, NFS, CIFS, PBS
• `snippets` — snippet files (cloud-init, hookscripts); Directory, NFS, CIFS

ZFS

ZFS provides software RAID, checksumming, copy-on-write snapshots, and built-in compression. It is the most feature-rich local storage option.

Strengths

• Data integrity via checksumming (detects and corrects silent corruption)
• Flexible RAID levels (mirror, RAIDZ1/2/3, dRAID)
• Efficient snapshots and clones (copy-on-write)
• Built-in compression (lz4 recommended — minimal CPU cost, good compression)
• Native replication to other nodes (pve-zsync or built-in replication)
• ARC cache provides excellent read performance

Constraints

• RAM-intensive: baseline 2GB + 1GB per TB of storage
• Requires HBA (IT mode) or direct disk access — never use hardware RAID controllers with write-back cache, as this

defeats ZFS's data integrity guarantees

• Cannot shrink a pool — plan capacity upfront
• Adding single disks to existing vdevs is supported only since OpenZFS 2.3+

ARC (Adaptive Replacement Cache) Sizing

The ARC is ZFS's primary read cache in RAM. If left uncapped, it silently consumes up to 50% of host memory, causing VM crashes from OOM contention.

• PVE 8.1+ defaults: New installs set ARC to 10% of physical memory, max 16GB
• Mixed workloads: Cap ARC to 25-30% of total host RAM
• Rule of thumb: 2GB base + 1GB per TB of storage
• Monitor: ARC hit ratio above 90% means the cache earns its RAM. Below 70%, diminishing returns
• Configuration:

  # /etc/modprobe.d/zfs.conf
  options zfs zfs_arc_max=8589934592  # 8GB

Run update-initramfs -u after changes. Reboot to apply.

• Temporary adjustment: echo "$[10 * 1024*1024*1024]" > /sys/module/zfs/parameters/zfs_arc_max
• If zfs_arc_max <= zfs_arc_min, zfs_arc_max is ignored — set zfs_arc_min lower first

SLOG (ZFS Intent Log)

The ZIL handles synchronous writes. A dedicated SLOG offloads this traffic, reducing latency for databases and frequent fsync operations.

• Size: 8-32GB is adequate — SLOG only holds ~5 seconds of write data
• Using more than half of installed RAM provides no benefit
• Requirements: enterprise NVMe with power-loss protection (PLP) — consumer SSDs without PLP defeat the purpose of

the ZIL

• Benefits: sync-heavy workloads (databases, NFS)

L2ARC (Level 2 Cache)

Second-level read cache on a dedicated fast device. Only use if ARC hit ratio is low and adding more RAM is not feasible.

• Sizing: 5-20x larger than RAM size
• RAM overhead: Budget 1GB of RAM per 50GB of L2ARC (for metadata in ARC)
• The L2ARC device must support more IOPS than the main pool to be effective
• A single SSD as L2ARC in front of 40 SSDs is useless — the pool already exceeds L2ARC IOPS

Block Size Tuning

Volblocksize (VMs on zvols) — sets the logical block size for VM disks. Can only be set at zvol creation. Match to RAID configuration to minimize write amplification:

• Mirror / RAID10 — 16k (PVE default)
• RAIDZ-1 (3-5 disks) — 16k-32k
• RAIDZ-1 (10+ disks) — 128k
• RAIDZ-2 (4 disks) — 16k
• RAIDZ-2 (11+ disks) — 128k

Recordsize (containers on datasets) — tune per workload:

• Postgres — 8k
• MariaDB/MySQL — 16k
• General purpose — 128k (default)
• Large sequential files (backups, video) — 1M

Other ZFS Settings

• ashift=12 for all modern SSDs and 4K drives. Incorrect ashift halves IOPS due to sector misalignment. Cannot be

changed after pool creation.

• compression=lz4 always — can increase I/O performance by writing less data. Use zstd if storage space is more

critical than CPU.

• atime=off — reduces metadata writes
• Schedule regular scrubs: zpool scrub <pool> (weekly or monthly)
• Never create swap on a ZFS zvol — use a partition on a physical disk

LVM and LVM-Thin

LVM (thick): Simple block storage. Each volume consumes its full allocated size immediately. PVE 9.0 adds snapshot support via volume chains (qcow2 layering).

LVM-Thin: Thin-provisioned volumes that allocate space on write. Supports snapshots and clones. Preferred over thick LVM for VM/container storage.

Configuration:

• Create thin pool on SSD/NVMe for VM storage
• Monitor thin pool usage — overprovisioning is expected, but the pool must not fill completely (causes I/O errors for

all guests on that pool)

• Set appropriate chunk size during thin pool creation (64K-512K)

Ceph (RBD and CephFS)

Ceph provides distributed, self-healing storage with no single point of failure. Proxmox VE integrates Ceph natively — it can manage the full Ceph lifecycle. Ceph Squid 19.2.1 is fully supported and stable since PVE 8.4 (was tech preview in 8.3). PVE 9.0+ defaults to Ceph Squid.

Strengths

• No single point of failure — data replicated across multiple nodes
• Self-healing — automatically recovers from disk/node failures
• Scales horizontally by adding nodes/disks
• Shared storage enables live migration and HA without external SAN
• Integrated management via Proxmox GUI and CLI

Requirements

• Minimum 3 nodes (for proper quorum and data distribution)
• Dedicated network for Ceph traffic: 10GbE minimum, 25GbE+ recommended for NVMe
• Separate Ceph cluster network from Proxmox corosync/management traffic — Ceph rebalance traffic saturates links and

triggers Corosync instability

• SSD or NVMe for OSD journals/WAL/DB (critical for performance)
• Dedicated disks for OSDs — never share with the OS

When to Choose Ceph vs ZFS

Rule of thumb: Use ZFS for single-node or small clusters without shared storage needs. Use Ceph when you need shared storage across 3+ nodes with HA.

Ceph Anti-Patterns

• Running Ceph on 1GbE — rebalance saturates the link, freezing client I/O and triggering Corosync fencing
• 2-node Ceph clusters — not HA, just a split-brain generator. Minimum 3 nodes for operational safety, 5+ for production

NFS and iSCSI

NFS: File-level shared storage. Easy to set up. Suitable for ISOs, backups, templates, and VM disks (as qcow2 files).

• Use NFSv4 when possible (better security, performance)
• Mount with noatime to reduce metadata writes
• Ensure server-side exports use no_root_squash for Proxmox operations

iSCSI: Block-level shared storage. Higher performance than NFS for VM workloads.

• Use multipath for redundancy
• Pair with LVM for thin provisioning on the Proxmox side
• Ensure proper CHAP authentication for security

Storage Selection Decision Tree

1. Need shared storage for HA/live migration?

• Yes, 3+ nodes, willing to manage complexity -> Ceph
• Yes, have existing SAN/NAS -> iSCSI or NFS
• No -> continue to local options

2. Need data integrity, snapshots, compression?

• Yes -> ZFS

3. Need snapshots/clones without ZFS overhead?

• Yes -> LVM-Thin

4. Simple block storage, no snapshots needed?

• Yes -> LVM (PVE 9+ adds snapshot support via volume chains)

5. Backup storage?

• Serious backups -> PBS (deduplication, encryption, verification)
• Simple backups -> NFS or Directory

VM and LXC Container Management

Virtualization Types

Proxmox VE supports two virtualization technologies: KVM virtual machines (full virtualization) and LXC containers (OS-level virtualization). Each serves different use cases with distinct trade-offs.

KVM Virtual Machines

VMs emulate a complete physical computer with dedicated virtual hardware. Each VM runs its own kernel, providing full isolation from the host and other guests.

• Isolation — full hardware-level isolation; separate kernel
• Overhead — higher; hypervisor layer, dedicated RAM allocation
• Guest OS — any OS (Linux, Windows, FreeBSD, etc.)
• Migration — live migration supported (except with PCIe passthrough)
• Use cases — untrusted workloads, non-Linux OS, GPU passthrough, full isolation

Configuration best practices:

• Use VirtIO drivers for disk (virtio-scsi-pci) and network (virtio) — they provide near-native performance vs.

emulated IDE/E1000

• Enable QEMU Guest Agent (qemu-guest-agent package inside VM) for proper shutdown, freeze/thaw for snapshots, and

IP reporting

• Use OVMF (UEFI) firmware with Q35 machine type for modern features (PCIe passthrough, Secure Boot support)
• Set CPU type to `host` for maximum performance when live migration between different CPU generations is not needed
• Use ballooning for dynamic memory management — set minimum memory below maximum to allow the hypervisor to reclaim

unused RAM

• Enable NUMA topology for VMs with many cores on multi-socket hosts

LXC Containers

Containers share the host kernel and provide lightweight, near-native performance isolation using Linux namespaces and cgroups.

• Isolation — namespace/cgroup isolation; shared kernel
• Overhead — minimal; no hypervisor layer, shared kernel
• Guest OS — Linux only (must be compatible with host kernel)
• Migration — offline migration only (restart required)
• Use cases — Linux services, high-density deployments, system containers

Configuration best practices:

• Use unprivileged containers (default) — they map container root (UID 0) to a non-privileged host UID (e.g.,

100000), preventing container escape attacks

• Only use privileged containers when unprivileged mode is incompatible with the workload (e.g., certain NFS mounts,

specific device access)

• Use bind mounts for sharing host directories into containers rather than NFS/CIFS mounts inside the container
• Set memory limits explicitly — containers without limits can consume all host RAM
• Enable nesting (features: nesting=1) only when running containers inside containers (e.g., Docker inside LXC)
• For Docker workloads, prefer: (1) a dedicated Docker VM (most compatible), (2) native OCI containers for

single-service apps (PVE 9.1+), or (3) Docker inside an unprivileged LXC with nesting=1 and keyctl=1 — but note this is unsupported by Proxmox and can break on host updates

• Docker-in-LXC breakage risk: CVE-2025-52881 (containerd patch) broke Docker in all LXC containers — workaround

requires setting lxc.apparmor.profile: unconfined, which weakens security. If using Docker-in-LXC, put Docker data on a separate mount point so the container OS is disposable and data persists across rebuilds

Choosing Between VM, LXC, and OCI Containers

Default to LXC for trusted Linux workloads. Use VMs when you need full isolation, non-Linux guests, PCIe passthrough, or live migration. Use OCI containers (PVE 9.1+) for single-purpose microservices from OCI registries. Use a Docker VM for multi-container stacks requiring Docker Compose or orchestration.

Template Management

VM Templates

1. Create a base VM with desired OS, packages, and configuration 2. Install qemu-guest-agent and cloud-init (if using cloud-init) 3. Remove machine-specific data (SSH host keys, machine-id) 4. Convert to template: qm template <vmid> 5. Deploy via linked clone (fast, shares base disk) or full clone (independent copy)

Container Templates

• Download official templates from Proxmox template repository via the GUI or:

  pveam update
  pveam available
  pveam download local <template-name>

• Create custom templates by configuring a container, then backing it up as a .tar.gz template

Linked vs Full Clone

Use linked clones for ephemeral or development workloads. Use full clones for production workloads that need independence from the template.

OCI Application Containers (PVE 9.1 — Technology Preview)

PVE 9.1 introduces native support for OCI (Open Container Initiative) images — the same format used by Docker Hub. This bridges the gap between traditional LXC system containers and Docker application containers.

How It Works

1. Pull from OCI Registry — a button in the storage interface (CT Templates section) downloads images from Docker Hub, GHCR, Quay, Harbor, or private registries using standard references (e.g., registry.hub.docker.com/library/nginx) 2. Layer squashing — PVE merges all OCI image layers into a single rootfs, creating an LXC-compatible template 3. Container creation — the OCI template is used like any other LXC template via the Create CT wizard 4. Host-managed DHCP — application containers without a traditional network stack get automatic DHCP via the host

Capabilities

• Managed as native Proxmox objects — snapshots, backups, permissions, storage, SDN all apply
• Compatible with any OCI-compliant registry
• Lower overhead than a Docker VM — shared kernel, no hypervisor layer
• More secure than running Docker directly on the PVE host

Limitations

• No in-place updates — layers are squashed at creation; to update, recreate the container with the newer image and

remap data volumes. The Proxmox team plans to add update support in a future release

• No Docker Compose — each OCI container is a single service; multi-container stacks require a Docker VM
• No shell in most containers — the console shows stdout of the main process, not a login shell
• No orchestration — no Swarm, no Kubernetes; for orchestrated workloads, use a Docker VM
• Technology Preview — not yet GA; expect rough edges

When to Use OCI Containers

• Single-purpose microservices: monitoring agents, reverse proxies, small databases, dashboards
• Quick deployments from Docker Hub without spinning up a Docker VM
• Reducing VM sprawl for lightweight, single-container workloads
• Environments where installing Docker/Podman is undesirable

Docker on Proxmox — Decision Guide

Docker VM best practices:

• Use Debian or Ubuntu Server as the base OS
• Install qemu-guest-agent for integration and stats reporting
• Use VirtIO network adapters
• Create a dedicated disk for Docker volumes (mounted to /opt/docker or /srv/docker) — separates app data from

OS, simplifies backups and migrations

• Back up volumes via Proxmox Backup Server or rsync/restic on the dedicated disk