Data Tools

Data Tools Skill

Critical Rule

NEVER use `grep`, `sed`, `awk`, or inline interpreters (`python3 -c`, `node -e`) on JSON, JSONL, YAML, TOML, XML, or CSV. Refuse and use the correct tool — text tools break on structure; inline scripts are verbose and fragile.

---

Tool Selection

Convert: dasel -w FORMAT or yq -o FORMAT.

---

Key Patterns

jq -- JSON (no in-place)

jq -r '.version' package.json
jq '.users[] | select(.role == "admin") | .name' users.json
jq '.version = "2.0.0"' pkg.json > pkg.json.tmp && mv pkg.json.tmp pkg.json
jq -s '.[0] * .[1]' base.json override.json

yq -- YAML (in-place -i)

yq '.services.web.image' docker-compose.yml
yq -i '.services.app.image = "node:20"' docker-compose.yml
yq -o json config.yml

dasel -- TOML / XML / Universal

dasel -f Cargo.toml '.package.version'
dasel put -f config.toml -t string -v "2.0" '.project.version'
dasel -f input.json -w yaml

qsv -- CSV / TSV

qsv headers data.csv && qsv stats data.csv --everything | qsv table
qsv search -s status "active" users.csv | qsv select name,email
qsv join user_id orders.csv id users.csv

mlr -- JSONL / multi-format / DSL (in-place -I)

mlr --c2j cat data.csv
mlr --jsonl filter '$level == "error"' logs.jsonl
mlr -I --csv put '$total = $qty * $price' orders.csv

GitHub CLI -- always --jq

gh api repos/owner/repo/releases/latest --jq '.tag_name'
gh pr list --json number,title --jq '.[] | [.number, .title] | @tsv'

---

Anti-Patterns

# BAD → GOOD: jq -r '.version' package.json
grep '"version"' package.json | sed 's/.*"\(.*\)".*/\1/'
# BAD → GOOD: yq -i '.services.app.image = "node:20"' compose.yml
sed -i 's/image: node:.*/image: node:20/' docker-compose.yml
# BAD → GOOD: qsv select 2 data.csv
awk -F',' '{print $2}' data.csv
# BAD → GOOD: jq -r '.version' plugin.json
python3 -c 'import json;print(json.load(open("plugin.json"))["version"])'

---

References

Docs: jq, yq, dasel, qsv, mlr.

CSV Processing with qsv

Patterns for fast, correct CSV/TSV processing using qsv.

---

Data Exploration Workflow

When you encounter a new CSV file, follow this sequence:

Step 1: Inspect Structure

# Column names
qsv headers data.csv

# Row count
qsv count data.csv

# First 5 rows in table format
qsv slice data.csv --len 5 | qsv table

Step 2: Profile Data

# Full statistics (min, max, mean, stddev, nullcount, etc.)
qsv stats data.csv --everything | qsv table

# Value distribution for categorical columns
qsv frequency data.csv --select category,status | qsv table

# Check for nulls/empty values
qsv stats data.csv --everything | qsv select field,nullcount | qsv search -s nullcount "[1-9]"

Step 3: Sample Data

# Random sample of 100 rows
qsv sample 100 data.csv

# Last 10 rows
qsv slice data.csv --start -10

# Every Nth row
qsv sample 50 data.csv --seed 42

---

Filtering and Selection

Column Selection

# By name
qsv select name,email,phone data.csv

# By index (1-based)
qsv select 1,3,5 data.csv

# Exclude columns
qsv select '!password,secret_key' data.csv

# Range of columns
qsv select 1-5 data.csv

# Reorder columns
qsv select email,name,id data.csv

Row Filtering

# Search in specific column
qsv search -s status "active" users.csv

# Regex search
qsv search -s email "@company\\.com$" contacts.csv

# Case-insensitive search
qsv search -i -s name "smith" people.csv

# Invert match (exclude rows)
qsv search -v -s status "deleted" records.csv

# Search across all columns
qsv search "error" logs.csv

Combined Filtering and Selection

# Filter then select columns
qsv search -s country "Germany" customers.csv | qsv select name,email,city

# Chain multiple filters
qsv search -s status "active" users.csv \
  | qsv search -s role "admin" \
  | qsv select name,email

---

Transformation

Sorting

# Sort by column (alphabetical)
qsv sort --select name employees.csv

# Numeric sort
qsv sort --select revenue --numeric sales.csv

# Reverse sort (descending)
qsv sort --select date --reverse events.csv

# Sort by multiple columns
qsv sort --select department,name employees.csv

Deduplication

# Remove exact duplicate rows
qsv dedup data.csv

# Deduplicate by specific columns
qsv dedup --select email contacts.csv

# Show only duplicates
qsv dedup --dupes-output dupes.csv data.csv

Renaming Columns

# Rename headers
qsv rename 'First Name,Last Name,Email Address' data.csv

# When you know current headers
qsv headers data.csv  # Check first
qsv rename 'id,name,email' data.csv

Adding Computed Columns

# Simple expression
qsv eval "total = price * quantity" orders.csv

# String concatenation
qsv eval "full_name = first_name + ' ' + last_name" people.csv

# Conditional
qsv eval "tier = if(revenue > 1000000, 'enterprise', 'standard')" accounts.csv

---

Joining Datasets

Inner Join

# Join on matching key
qsv join user_id orders.csv id users.csv

Left Join

# Keep all rows from left file
qsv join --left user_id orders.csv id users.csv

Join with Column Selection

# Join then select useful columns
qsv join user_id orders.csv id users.csv \
  | qsv select 'order_id,user_id,name,email,amount'

Cross-Reference Datasets

# Find orders from German customers
qsv search -s country "Germany" customers.csv | qsv select id > german_ids.csv
qsv join customer_id orders.csv id german_ids.csv

---

Statistical Analysis

Summary Statistics

# Basic stats for all columns
qsv stats data.csv | qsv table

# Full statistics including cardinality, mode, quartiles
qsv stats data.csv --everything | qsv table

# Stats for specific columns
qsv stats data.csv --select revenue,quantity --everything | qsv table

Frequency Analysis

# Value counts for a column
qsv frequency data.csv --select status

# Top N values
qsv frequency data.csv --select category --limit 20

# Frequency across multiple columns
qsv frequency data.csv --select 'status,category,region'

# Frequency as percentage (pipe to further processing)
qsv frequency data.csv --select status | qsv table

Grouping

# Count per group
qsv frequency data.csv --select department

# For more complex aggregations, combine with sort and dedup
qsv sort --select department data.csv | qsv dedup --select department

---

Large File Handling

Performance Tips

# Index for faster repeated operations
qsv index data.csv
# Creates data.csv.idx -- subsequent operations are faster

# Count rows (instant with index)
qsv count data.csv

# Random access with index
qsv slice data.csv --start 1000000 --len 100

Splitting Large Files

# Split into chunks of N rows
qsv split --size 100000 output_dir data.csv

# Split by column value
qsv partition region output_dir data.csv

Sampling for Analysis

# Analyze a sample instead of full dataset
qsv sample 10000 huge-file.csv > sample.csv
qsv stats sample.csv --everything | qsv table

---

Format Conversion

TSV to CSV

# Convert tab-delimited to CSV
qsv input --delimiter '\t' data.tsv > data.csv

CSV to JSON

# Each row becomes a JSON object
qsv tojsonl data.csv  # JSON Lines format (one object per line)

Excel to CSV

# qsv can read Excel files directly (if compiled with feature)
qsv excel data.xlsx > data.csv
qsv excel data.xlsx --sheet "Sheet2" > sheet2.csv

---

Validation and Cleaning

Check Data Quality

# Validate CSV structure
qsv validate data.csv

# Check for inconsistent row lengths
qsv validate data.csv 2>&1

# Count empty/null fields per column
qsv stats data.csv --everything | qsv select field,nullcount | qsv table

Clean Data

# Trim whitespace from all fields
qsv trim data.csv

# Fill empty cells with default
qsv fill --default "N/A" data.csv

# Remove rows with empty required fields
qsv search -s email ".+" data.csv  # Keep only rows with non-empty email

---

Anti-Patterns

# BAD: awk breaks on quoted fields containing commas
awk -F',' '{print $2}' data.csv

# GOOD: Proper CSV parsing
qsv select 2 data.csv

# BAD: sort does not understand CSV structure
sort -t',' -k3 -n data.csv

# GOOD: CSV-aware sort
qsv sort --select 3 --numeric data.csv

---

Comparison: qsv vs Alternatives

qsv vs awk

# Task: Sum a numeric column

# awk (breaks on quoted fields with commas)
awk -F',' '{sum += $3} END {print sum}' data.csv

# qsv (correct CSV parsing, handles quoting)
qsv stats data.csv --select 3 | qsv select sum

qsv vs Python pandas

# Task: Get top 10 values by revenue

# Python (requires script, slow startup, high memory)
python3 -c "
import pandas as pd
df = pd.read_csv('data.csv')
print(df.nlargest(10, 'revenue'))
"

# qsv (one-liner, instant, low memory)
qsv sort --select revenue --numeric --reverse data.csv | qsv slice --len 10 | qsv table

qsv vs csvkit

# qsv is significantly faster than csvkit for large files.
# csvkit is Python-based; qsv is compiled Rust.
# For files > 100MB, qsv is typically 10-100x faster.

# Both handle CSV correctly (quoting, escaping, encoding).
# Use qsv for performance; csvkit if qsv is unavailable.

---

Common Workflows

Log Analysis

# Parse structured logs exported as CSV
qsv search -s level "ERROR" logs.csv | qsv frequency --select source | qsv table

Data Pipeline

# Filter -> Transform -> Aggregate -> Export
qsv search -s status "completed" orders.csv \
  | qsv select customer_id,amount,date \
  | qsv sort --select date --reverse \
  | qsv slice --len 1000 \
  > recent_completed.csv

Report Generation

# Quick summary report
echo "=== Row Count ==="
qsv count data.csv
echo "=== Column Stats ==="
qsv stats data.csv --select revenue,quantity | qsv table
echo "=== Top Categories ==="
qsv frequency data.csv --select category --limit 5 | qsv table

dasel Cookbook

Universal selector for JSON, YAML, TOML, and XML using dasel v2 (TomWright/dasel).

---

Basic Usage

dasel auto-detects format from file extension:

# JSON
dasel -f config.json '.database.host'

# YAML
dasel -f config.yml '.server.port'

# TOML
dasel -f Cargo.toml '.package.version'

# XML
dasel -f pom.xml '.project.version'

---

In-Place Editing

# Set string value (auto-detects format, writes back)
dasel put -f config.json -t string -v "localhost" '.database.host'

# Set numeric value
dasel put -f config.json -t int -v 5432 '.database.port'

# Set boolean
dasel put -f config.toml -t bool -v true '.features.experimental'

---

When to Use dasel Over jq/yq

• TOML files: Cargo.toml, pyproject.toml, config.toml -- jq/yq cannot handle TOML
• XML files: pom.xml, web.xml -- simpler than xmlstarlet for basic operations
• Mixed-format pipelines: Read YAML, output JSON, etc.
• Simple reads/writes: Less syntax to remember than jq for basic operations

---

Format Conversion

# JSON to YAML
dasel -f input.json -w yaml

# TOML to JSON
dasel -f Cargo.toml -w json

# YAML to TOML
dasel -f config.yml -w toml

---

Common Workflows

Modifying package.json (alternative to jq)

# Simpler syntax for single-value edits
dasel put -f package.json -t string -v "2.1.0" '.version'

Updating Docker-Compose (alternative to yq)

dasel put -f docker-compose.yml -t string -v "postgres:16-alpine" '.services.postgres.image'

TOML Configuration Editing

# Read Cargo.toml version
dasel -f Cargo.toml '.package.version'

# Update pyproject.toml
dasel put -f pyproject.toml -t string -v "3.0.0" '.project.version'

jq Cookbook

Comprehensive patterns for JSON processing with jq.

---

Basic Extraction

# Single field
jq '.name' package.json

# Nested field
jq '.repository.url' package.json

# Array elements
jq '.items[]' response.json

# Array element by index
jq '.items[0]' response.json

# Multiple fields
jq '{name: .name, version: .version}' package.json

---

Filtering

# Select matching objects from array
jq '.users[] | select(.role == "admin")' users.json

# Multiple conditions
jq '.items[] | select(.age > 18 and .active == true)' data.json

# Null-safe access
jq '.items[] | select(.email != null)' data.json

# Regex matching
jq '.files[] | select(.name | test("^test.*\\.js$"))' manifest.json

---

Transformation

# Map over array
jq '[.items[] | {id: .id, label: .name}]' data.json

# Group by field
jq 'group_by(.category) | map({key: .[0].category, count: length})' items.json

# Sort
jq 'sort_by(.date) | reverse' events.json

# Flatten nested arrays
jq '[.groups[].members[]]' org.json

# Unique values
jq '[.items[].category] | unique' data.json

# Aggregate
jq '[.items[].price] | add' cart.json

---

In-Place Editing

jq does not support in-place editing natively. Use a temp file pattern:

# Safe in-place edit with temp file
jq '.version = "2.0.0"' package.json > package.json.tmp && mv package.json.tmp package.json

# Or with sponge (from moreutils)
jq '.version = "2.0.0"' package.json | sponge package.json

Note: sponge requires the moreutils package (apt install moreutils / brew install moreutils).

---

GitHub CLI Integration

Always prefer the `--jq` flag over piping to jq. It saves a process and is idiomatic.

# GOOD: --jq flag (preferred)
gh api repos/owner/repo/releases --jq '.[0].tag_name'
gh pr list --json number,title,author --jq '.[] | "\(.number)\t\(.title)\t\(.author.login)"'
gh run list --json status,conclusion --jq '.[] | select(.status == "completed")'

# BAD: piping to jq (wasteful, extra process)
gh api repos/owner/repo/releases | jq '.[0].tag_name'

The --jq flag works on both gh api and structured gh commands that support --json.

---

Anti-Patterns

# BAD: Fragile, breaks on formatting changes
grep '"version"' package.json | sed 's/.*: "\(.*\)".*/\1/'

# GOOD: Correct regardless of formatting
jq -r '.version' package.json

# BAD: Multi-line JSON defeats grep
grep '"name"' response.json

# GOOD: Handles any structure
jq '.items[].name' response.json

# BAD: sed on JSON changes (breaks on nested quotes, escapes)
sed -i 's/"version": "1.0.0"/"version": "2.0.0"/' package.json

# GOOD: Structural edit
jq '.version = "2.0.0"' package.json > package.json.tmp && mv package.json.tmp package.json

---

API Response Parsing

Extract Nested Data

# Get all repository names from GitHub org
gh api orgs/myorg/repos --jq '.[].full_name'

# Get release assets download URLs
gh api repos/owner/repo/releases/latest --jq '.assets[].browser_download_url'

# Extract paginated results (GitHub API)
gh api repos/owner/repo/issues --paginate --jq '.[].title'

Flatten Nested Responses

# Pull requests with review info
gh pr list --json number,title,reviews --jq '
  .[] | {
    number,
    title,
    approvals: [.reviews[] | select(.state == "APPROVED") | .author.login]
  }'

Handle Nullable Fields

# Default value for missing fields
jq '.items[] | {name, email: (.email // "N/A")}' users.json

# Filter out nulls
jq '[.items[] | select(.email != null)]' users.json

# Conditional field inclusion
jq '.items[] | {name} + (if .email then {email} else {} end)' users.json

---

Configuration File Manipulation

Read and Update package.json

# Read version
jq -r '.version' package.json

# Bump patch version
jq '.version |= (split(".") | .[2] = ((.[2] | tonumber) + 1 | tostring) | join("."))' \
  package.json > package.json.tmp && mv package.json.tmp package.json

# Add a dependency
jq '.dependencies["new-package"] = "^2.0.0"' package.json > package.json.tmp \
  && mv package.json.tmp package.json

# Remove a dependency
jq 'del(.devDependencies["old-package"])' package.json > package.json.tmp \
  && mv package.json.tmp package.json

# Sort dependencies alphabetically
jq '.dependencies = (.dependencies | to_entries | sort_by(.key) | from_entries)' \
  package.json > package.json.tmp && mv package.json.tmp package.json

Read and Update composer.json

# Get required PHP version
jq -r '.require.php' composer.json

# Add a requirement
jq '.require["vendor/package"] = "^3.0"' composer.json > composer.json.tmp \
  && mv composer.json.tmp composer.json

# Update autoload namespace
jq '.autoload."psr-4"["App\\"] = "src/"' composer.json > composer.json.tmp \
  && mv composer.json.tmp composer.json

Read and Update tsconfig.json

# Get compiler target
jq -r '.compilerOptions.target' tsconfig.json

# Enable strict mode
jq '.compilerOptions.strict = true' tsconfig.json > tsconfig.json.tmp \
  && mv tsconfig.json.tmp tsconfig.json

# Add path alias
jq '.compilerOptions.paths["@utils/*"] = ["src/utils/*"]' tsconfig.json > tsconfig.json.tmp \
  && mv tsconfig.json.tmp tsconfig.json

---

Data Transformation Recipes

Reshape Objects

# Rename keys
jq '.items[] | {id: .identifier, label: .display_name}' data.json

# Merge objects
jq '.defaults * .overrides' config.json

# Pick specific keys
jq '.items[] | {name, email}' users.json

# Exclude specific keys
jq '.items[] | del(.password, .internal_id)' users.json

Array Operations

# Flatten nested arrays
jq '[.departments[].employees[]]' org.json

# Zip two arrays
jq '[transpose[] | {key: .[0], value: .[1]}]' <<< '{"a": [["x","y"],["1","2"]]}'

# Chunk array into groups of N
jq '[range(0; length; 3) as $i | .[$i:$i+3]]' data.json

# Intersection of two arrays
jq --argjson a '["a","b","c"]' --argjson b '["b","c","d"]' \
  -n '[$a[] as $x | $b[] | select(. == $x)]'

Grouping and Aggregation

# Group by field and count
jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json

# Group and sum
jq 'group_by(.category) | map({
  category: .[0].category,
  total: (map(.amount) | add),
  count: length
})' transactions.json

# Pivot table style
jq 'group_by(.year) | map({
  year: .[0].year,
  quarters: group_by(.quarter) | map({
    quarter: .[0].quarter,
    revenue: (map(.revenue) | add)
  })
})' sales.json

String Operations

# Split and rejoin
jq '.path | split("/") | last' data.json

# String interpolation
jq '.items[] | "Item \(.id): \(.name) (\(.status))"' data.json

# Regex capture
jq '.version | capture("(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)")' package.json

# Replace in string
jq '.url | gsub("http://"; "https://")' config.json

---

GitHub CLI Integration Patterns

Pull Request Workflows

# List PRs with specific label, formatted as table
gh pr list --json number,title,author,labels --jq '
  .[] | select(.labels | map(.name) | index("bug"))
  | [.number, .author.login, .title] | @tsv'

# Get PR review status
gh pr view 42 --json reviews --jq '
  .reviews | group_by(.state) | map({state: .[0].state, count: length})'

# Find PRs by author
gh pr list --json number,title,author --jq '
  [.[] | select(.author.login == "username")] | length'

Repository Analysis

# List repos with specific topic
gh api orgs/myorg/repos --paginate --jq '
  [.[] | select(.topics | index("production"))] | map(.full_name)'

# Get branch protection rules
gh api repos/owner/repo/branches/main/protection --jq '{
  required_reviews: .required_pull_request_reviews.required_approving_review_count,
  ci_required: .required_status_checks.strict,
  contexts: .required_status_checks.contexts
}'

# Repo language breakdown
gh api repos/owner/repo/languages --jq 'to_entries | sort_by(-.value) | map("\(.key): \(.value)")'

Actions and Workflows

# List failed workflow runs
gh run list --json status,conclusion,name,headBranch --jq '
  [.[] | select(.conclusion == "failure")]
  | map({name, branch: .headBranch})'

# Get workflow run annotations (warnings/errors)
gh api "repos/owner/repo/check-runs/12345/annotations" --jq '
  .[] | {level: .annotation_level, message: .message, file: .path, line: .start_line}'

# Check run durations
gh run list --json databaseId,name,updatedAt,createdAt --jq '
  .[:10] | map({name, id: .databaseId})'

---

Advanced Patterns

Dynamic Key Access

# Variable key name
jq --arg key "production" '.environments[$key]' config.json

# Build object with dynamic keys
jq --arg env "staging" '{($env): .defaults}' config.json

# Iterate over all keys
jq 'to_entries[] | "\(.key) = \(.value)"' flat-config.json

Conditional Logic

# If-then-else
jq '.items[] | if .score >= 90 then "A" elif .score >= 80 then "B" else "C" end' grades.json

# Conditional field
jq '.items[] | . + {tier: (if .revenue > 1000000 then "enterprise" else "standard" end)}' accounts.json

# Default value (alternative operator for null/false)
jq '.items[] | .nested.field // "default"' data.json

Multi-File Operations

# Merge multiple JSON files
jq -s 'add' file1.json file2.json file3.json

# Combine into array
jq -s '.' file1.json file2.json file3.json

# Deep merge
jq -s '.[0] * .[1]' base.json override.json

# Compare two files
diff <(jq --sort-keys . a.json) <(jq --sort-keys . b.json)

Output Formatting

# Compact output (no whitespace)
jq -c '.' data.json

# Raw string output (no quotes)
jq -r '.name' data.json

# Tab-separated values
jq -r '.items[] | [.id, .name, .email] | @tsv' data.json

# CSV output
jq -r '.items[] | [.id, .name, .email] | @csv' data.json

# Custom delimiter
jq -r '.items[] | [.id, .name] | join(" | ")' data.json

# Pretty-print with sorted keys
jq --sort-keys '.' data.json

Streaming Large Files

# Process large files without loading entirely into memory
jq --stream 'select(.[0][-1] == "name") | .[1]' huge-file.json

# Truncate arrays for inspection
jq '.items = (.items[:5] + [{"...": "truncated"}])' large-response.json

---

Common Idioms

mlr Cookbook

Name-indexed record processing across CSV, TSV, JSON, JSON Lines, PPRINT, XTAB, and NIDX using Miller (mlr).

---

Basic Usage

mlr reads a stream of records, applies a chain of verbs (cat, filter, put, cut, sort, stats1, join, ...), then writes them out. Input and output formats are independent.

# Pretty-print a CSV
mlr --csv --opprint cat data.csv

# First N records
mlr --csv head -n 5 data.csv

# Tail
mlr --csv tail -n 5 data.csv

# List column names
head -n 1 data.csv
mlr --c2j cat data.csv | jq '.[0] | keys'   # via JSON (keys of first record)

---

Format Conversion

mlr's shorthand flags pair input and output formats: --c2j = CSV in, JSON out. The long form is --icsv --ojson.

# CSV <-> JSON / JSONL / TSV / Pretty-Print
mlr --c2j cat data.csv               # CSV  -> JSON array
mlr --icsv --ojsonl cat data.csv     # CSV  -> JSON Lines
mlr --c2t cat data.csv               # CSV  -> TSV
mlr --c2p cat data.csv               # CSV  -> PPRINT (aligned table)
mlr --j2c cat data.json              # JSON -> CSV
mlr --l2c cat events.jsonl           # JSONL -> CSV

# Long-form when shorthand doesn't cover the pair
mlr --ijsonl --oxtab cat events.jsonl
mlr --inidx --ifs comma --ojson cat raw.txt

---

DSL Filtering and Transformation

put adds/updates fields; filter keeps matching records. Both use the same DSL ($field references, types, regex, control flow).

# Filter rows
mlr --csv filter '$status == "active" && $amount > 100' txns.csv

# Compute a new column
mlr --csv put '$total = $qty * $price' orders.csv

# Conditional assignment
mlr --csv put '$tier = $revenue > 1e6 ? "enterprise" : "standard"' accounts.csv

# Regex
mlr --csv filter '$email =~ "@example\\.com$"' users.csv

# Chained verbs with `then`
mlr --csv filter '$status == "active"' then cut -f name,email then sort -f name users.csv

# Rename / reorder fields
mlr --csv rename id,user_id then reorder -f user_id,name users.csv

---

Stats and Group-By

# Mean, stddev, percentiles on a numeric field
mlr --csv stats1 -a mean,stddev,p50,p95 -f price sales.csv

# Group-by sum
mlr --csv stats1 -a sum,count -f amount -g category txns.csv

# Two-way group-by
mlr --csv stats1 -a mean -f latency -g region,endpoint requests.csv

# Frequency table for a categorical field
mlr --csv count-distinct -f status events.csv

# Top-N per group
mlr --csv top -n 3 -f amount -g category txns.csv

---

Joins

# Inner join on a shared key
mlr --csv join -j user_id -f users.csv orders.csv

# Left-outer join (keep unmatched left records)
mlr --csv join --ul -j user_id -f users.csv orders.csv

# Different key names on each side
mlr --csv join -l id -r user_id -f users.csv orders.csv

# Join across formats: left JSONL stream, right CSV lookup, output JSON
mlr --ijsonl --ojson join -i csv -f users.csv -j user_id then put '$joined = true' events.jsonl

---

In-Place Editing

-I rewrites files in place (across any supported format).

# Add a computed column to every row in a CSV
mlr -I --csv put '$total = $qty * $price' orders.csv

# Bulk update across many files
mlr -I --csv put '$updated_at = "2026-01-01"' data/*.csv

# JSONL in-place
mlr -I --jsonl put '$processed = true' events.jsonl

---

Multi-File and Pipeline Flows

# Concatenate CSVs with the same schema (headers merged)
mlr --csv cat sales-*.csv > all-sales.csv

# Per-file tagging then merge
mlr --csv put '$source = FILENAME' a.csv b.csv c.csv

# JSONL log analysis pipeline
mlr --jsonl filter '$level == "error"' then \
            cut -f ts,service,msg then \
            sort -f ts logs.jsonl

# Hand off to jq for deep JSON shaping
mlr --c2j cat data.csv | jq '[.[] | select(.score > 90)]'

---

Anti-Patterns

# BAD: grep/awk on JSONL drops structure, breaks on embedded quotes/commas
grep '"level":"error"' events.jsonl | awk -F'"msg":"' '{print $2}'
# GOOD
mlr --jsonl filter '$level == "error"' then cut -f msg events.jsonl

# BAD: awk on CSV breaks on quoted fields containing commas
awk -F',' '$3 == "active" {print $1,$2}' users.csv
# GOOD
mlr --csv filter '$status == "active"' then cut -f id,name users.csv

---

Common Idioms

yq Cookbook

YAML manipulation patterns using Mike Farah's yq (Go implementation).

---

Basic Operations

# Read a field
yq '.services.web.image' docker-compose.yml

# Read nested field
yq '.jobs.build.steps[0].uses' .github/workflows/ci.yml

# List all keys at a level
yq '.services | keys' docker-compose.yml

---

In-Place Editing

yq supports true in-place editing with -i:

# Set a value
yq -i '.version = "3.0.0"' chart.yaml

# Add an element to an array
yq -i '.services.web.ports += ["8080:8080"]' docker-compose.yml

# Delete a field
yq -i 'del(.services.debug)' docker-compose.yml

# Set nested value (creates intermediate keys)
yq -i '.jobs.test.env.CI = "true"' .github/workflows/ci.yml

---

Anti-Patterns

# BAD: sed does not understand YAML indentation or multi-line values
sed -i 's/image: node:.*/image: node:20/' docker-compose.yml

# GOOD: Structural edit that respects YAML semantics
yq -i '.services.app.image = "node:20"' docker-compose.yml

# BAD: grep on YAML misses context
grep "uses:" .github/workflows/ci.yml

# GOOD: Query with structure
yq '.jobs[].steps[].uses | select(. != null)' .github/workflows/ci.yml

---

GitHub Actions Workflow Editing

Update Action Versions

# Update a specific action
yq -i '(.jobs.build.steps[] | select(.uses == "actions/checkout@v3")).uses = "actions/checkout@v4"' \
  .github/workflows/ci.yml

# Update all occurrences of an action across all workflows
for f in .github/workflows/*.yml; do
  yq -i '(.jobs[].steps[] | select(.uses | test("^actions/checkout@"))).uses = "actions/checkout@v4"' "$f"
done

# Pin action to SHA
yq -i '
  (.jobs[].steps[] | select(.uses | test("^actions/setup-node@"))).uses =
  "actions/setup-node@1234567890abcdef1234567890abcdef12345678"
' .github/workflows/ci.yml

Modify Matrix Strategy

# Set matrix values
yq -i '.jobs.test.strategy.matrix.php-version = ["8.2", "8.3", "8.4"]' \
  .github/workflows/ci.yml

# Add to matrix include
yq -i '.jobs.test.strategy.matrix.include += [{"os": "ubuntu-24.04", "php": "8.4"}]' \
  .github/workflows/ci.yml

# Remove a matrix value
yq -i '.jobs.test.strategy.matrix.node-version -= ["16"]' \
  .github/workflows/ci.yml

Add or Modify Steps

# Append a step
yq -i '.jobs.build.steps += [{"name": "Run linter", "run": "npm run lint"}]' \
  .github/workflows/ci.yml

# Insert step at position (before index 2)
yq -i '.jobs.build.steps |= (
  .[:2] + [{"name": "Cache", "uses": "actions/cache@v4"}] + .[2:]
)' .github/workflows/ci.yml

# Add step with multi-line run command
yq -i '.jobs.build.steps += [{
  "name": "Build and test",
  "run": "npm ci\nnpm run build\nnpm test"
}]' .github/workflows/ci.yml

# Delete a step by name
yq -i 'del(.jobs.build.steps[] | select(.name == "Old step"))' \
  .github/workflows/ci.yml

Environment and Permissions

# Set workflow-level env
yq -i '.env.FORCE_COLOR = "1"' .github/workflows/ci.yml

# Set job-level permissions
yq -i '.jobs.deploy.permissions = {"contents": "read", "id-token": "write"}' \
  .github/workflows/ci.yml

# Add concurrency control
yq -i '.concurrency = {"group": "ci-${{ github.ref }}", "cancel-in-progress": true}' \
  .github/workflows/ci.yml

Triggers

# Set push trigger branches
yq -i '.on.push.branches = ["main", "release/*"]' .github/workflows/ci.yml

# Add workflow_dispatch with inputs
yq -i '.on.workflow_dispatch.inputs.environment = {
  "description": "Target environment",
  "required": true,
  "default": "staging",
  "type": "choice",
  "options": ["staging", "production"]
}' .github/workflows/ci.yml

# Set scheduled trigger
yq -i '.on.schedule = [{"cron": "0 6 * * 1"}]' .github/workflows/ci.yml

---

Docker-Compose Manipulation

Service Management

# Add a new service
yq -i '.services.redis = {
  "image": "redis:7-alpine",
  "ports": ["6379:6379"],
  "volumes": ["redis-data:/data"]
}' docker-compose.yml

# Update image tag
yq -i '.services.app.image = "myapp:2.5.0"' docker-compose.yml

# Add depends_on
yq -i '.services.app.depends_on += ["redis"]' docker-compose.yml

# Remove a service
yq -i 'del(.services.legacy)' docker-compose.yml

Environment Variables

# Set environment variable (map syntax)
yq -i '.services.app.environment.DATABASE_URL = "postgres://localhost/mydb"' \
  docker-compose.yml

# Add environment variable (list syntax)
yq -i '.services.app.environment += ["NEW_VAR=value"]' docker-compose.yml

# Read all environment variables for a service
yq '.services.app.environment' docker-compose.yml

Volumes and Networks

# Add a named volume
yq -i '.volumes.pgdata = {"driver": "local"}' docker-compose.yml

# Add volume mount to service
yq -i '.services.db.volumes += ["pgdata:/var/lib/postgresql/data"]' docker-compose.yml

# Add custom network
yq -i '.networks.backend = {"driver": "bridge"}' docker-compose.yml
yq -i '.services.app.networks += ["backend"]' docker-compose.yml

---

Kubernetes Manifest Editing

Deployment Updates

# Update container image
yq -i '(.spec.template.spec.containers[] | select(.name == "app")).image = "myapp:3.0"' \
  deployment.yml

# Set resource limits
yq -i '(.spec.template.spec.containers[] | select(.name == "app")).resources = {
  "requests": {"memory": "256Mi", "cpu": "250m"},
  "limits": {"memory": "512Mi", "cpu": "500m"}
}' deployment.yml

# Add environment variable
yq -i '(.spec.template.spec.containers[] | select(.name == "app")).env += [{
  "name": "LOG_LEVEL",
  "value": "info"
}]' deployment.yml

# Update replicas
yq -i '.spec.replicas = 3' deployment.yml

# Add annotation
yq -i '.metadata.annotations["app.kubernetes.io/version"] = "3.0.0"' deployment.yml

ConfigMap and Secret

# Update ConfigMap data
yq -i '.data["config.yaml"] = "key: new-value\nother: setting"' configmap.yml

# Add label to all resources in multi-doc
yq eval-all -i '.metadata.labels["managed-by"] = "automation"' manifests.yml

---

Multi-Document YAML

Select Specific Documents

# Select by kind
yq eval-all 'select(.kind == "Service")' k8s-all.yml

# Select by name
yq eval-all 'select(.metadata.name == "my-app")' manifests.yml

# Count documents
yq eval-all '[.] | length' manifests.yml

Modify Across Documents

# Add label to all documents
yq eval-all -i '.metadata.labels["team"] = "platform"' manifests.yml

# Update image in all Deployments
yq eval-all -i '
  select(.kind == "Deployment").spec.template.spec.containers[0].image = "newimage:latest"
' manifests.yml

Split and Merge

# Split multi-doc into individual files
yq eval-all -s '.kind + "-" + .metadata.name' manifests.yml

# Merge multiple files into one multi-doc
yq eval-all '.' deployment.yml service.yml configmap.yml > combined.yml

---

Format Conversion

YAML to JSON

# Single file
yq -o json config.yml

# Write to file
yq -o json config.yml > config.json

# Pretty-print JSON
yq -o json -P config.yml

JSON to YAML

# Convert JSON to YAML
yq -P config.json

# Pipe from stdin
curl -s https://api.example.com/config | yq -P

YAML to Properties

# Flat key=value format
yq -o props config.yml
# Output: database.host = localhost
#         database.port = 5432

---

Advanced Patterns

Variables and Environment

# Use shell variable in yq
VERSION="2.0.0"
yq -i ".version = \"${VERSION}\"" chart.yaml

# Using yq's env() function
export APP_VERSION="2.0.0"
yq -i '.image.tag = env(APP_VERSION)' values.yaml

# Using strenv() for string values
export DB_HOST="db.example.com"
yq -i '.database.host = strenv(DB_HOST)' config.yml

Conditional Operations

# Update only if field exists
yq -i '(.services[] | select(has("healthcheck"))).healthcheck.interval = "30s"' \
  docker-compose.yml

# Add field only if missing
yq -i '.services.app.restart //= "unless-stopped"' docker-compose.yml

Comments

# yq preserves YAML comments by default

# Add a comment before a key
yq -i '.database.host line_comment="Primary database host"' config.yml

# Read comments
yq '.database.host | line_comment' config.yml

Anchors and Aliases

# yq supports YAML anchors (&) and aliases (*)
# Read with anchor expansion
yq 'explode(.)' config-with-anchors.yml

---

Common Idioms

---

yq is not jq

The installed yq is mikefarah yq (v4) — a Go tool with its own expression language, not a jq wrapper. Porting jq idioms breaks, sometimes silently:

# BAD: `empty` is a jq builtin; mikefarah yq has no such function
yq '.items[] // empty' file.yml      # Error: lexer: invalid input text "empty"

# GOOD: default to an empty array before iterating — robust for a missing, null,
# or empty key (the `//` alternative operator exists; its right side is a value,
# not a jq builtin):
yq '(.items // [])[]' file.yml

Scripting trap: appending || true to a yq command hides the error and yields an empty result, silently dropping data (e.g. an exclude: list vanishing). Test the exact expression before relying on it — don't assume jq syntax carries over.