Bash Script Generator

Bash Script Generator

Overview

Generate production-ready Bash scripts with clear requirements capture, deterministic generation flow, and validation-first iteration.

Trigger Phrases

Use this skill when the user asks to:

• Create, generate, write, or build a Bash/shell script
• Convert manual CLI steps into an automation script
• Build a text-processing script using grep, awk, or sed
• Create an operations helper script, cron script, or CI utility script

Do not use this skill for validating an existing script only. Use devops-skills:bash-script-validator for validation-only requests.

Execution Model

Follow stages in order. Do not skip a stage; use the documented fallback when blocked.

Stage 0: Preflight

1. Confirm scope and target output path. 2. Confirm shell target:

• Default: bash
• If portability is requested: POSIX sh

3. Check capabilities and pick fallback path:

If a fallback path is used, state it explicitly in the final summary.

Stage 1: Capture Requirements

Collect only what is needed to generate the script correctly:

• Input source and format
• Output destination and format
• Error handling behavior (fail-fast/retry/continue)
• Security constraints (sensitive data, privilege level)
• Performance constraints (large files, parallelism)
• Portability requirement (Bash-only vs POSIX)

Then create a Captured Requirements table with stable IDs.

## Captured Requirements

| Requirement ID | Description | Source | Implementation Plan |
|---|---|---|---|
| REQ-001 | Parse nginx logs from file input | User | `parse_args()` + `validate_file()` + `awk` parser |
| REQ-002 | Output top 10 errors | User | `analyze_errors()` + `sort | uniq -c | head -10` |
| REQ-003 | Handle large files efficiently | Assumption | Single-pass `awk`; avoid multi-pass loops |

Rules:

• Every major design decision maps to at least one REQ-*.
• Keep assumptions explicit and minimal.

Stage 2: Choose Generation Path

Use this deterministic decision tree:

Need multi-command architecture, unusual control flow, or strict non-template conventions?
├─ Yes -> Custom generation
└─ No
   Need standard CLI skeleton (usage/logging/arg parsing/cleanup)?
   ├─ Yes -> Template-first generation
   └─ No -> Custom generation

Template-first is the default for single-purpose CLI utilities.

Stage 3: Load Only Relevant References

Use progressive disclosure. Read only docs needed for the current request.

Citation format (required):

• [Ref: docs/<file>.md -> <section>]

Stage 4: Generate Script

Path A: Template-first (default)

1. Generate scaffold:

bash scripts/generate_script_template.sh standard output-script.sh

2. Replace placeholders and add business logic. 3. Keep logging to stderr and data output to stdout unless requirements say otherwise. 4. Add comments only where logic is non-obvious.

Path B: Custom generation

Build a script with at least:

• Shebang and strict mode
• usage()
• parse_args()
• Input validation and dependency checks
• Main workflow function(s)
• Predictable exit codes

Stage 5: Validate and Iterate

Default validation path: 1. Invoke devops-skills:bash-script-validator 2. Apply fixes 3. Re-run validation 4. Repeat until checks pass or blocker is identified

Fallback validation path (when validator skill is unavailable):

# Deterministic local gate for this skill:
bash scripts/run_ci_checks.sh --skip-shellcheck

# CI gate (shellcheck required):
bash scripts/run_ci_checks.sh --require-shellcheck

If any check is skipped, include Skipped check, Reason, and Risk in the output.

Stage 6: Final Response Contract

Always return: 1. Generated script path 2. Requirements traceability (REQ-* -> implementation) 3. Validation results with rerun status 4. Citations in standard format 5. Any assumptions/fallbacks used

Canonical Example Flows

Example A: Full Flow (Template-first)

1. Clarify missing data format and output expectations. 2. Capture REQ-* table. 3. Choose template-first path. 4. Generate scaffold with scripts/generate_script_template.sh. 5. Implement logic and map functions to REQ-*. 6. Validate with devops-skills:bash-script-validator and rerun until clean. 7. Return final summary with citations.

Example B: Constrained Environment Flow

Use this when AskUserQuestion, validator skill, or shellcheck is unavailable: 1. Ask clarifying questions in chat. 2. Mark unresolved items as assumptions in Captured Requirements. 3. Generate from template script or template file copy fallback. 4. Run bash -n (and sh -n if relevant). 5. If shellcheck is missing, report skip with risk and mitigation.

Done Criteria

The task is complete only when all items are true:

• Trigger matched and scope confirmed
• Captured Requirements table exists with REQ-* IDs
• Template-first vs custom decision is documented
• Script is generated with deterministic structure
• Validation executed and rerun policy applied
• Any skipped checks include explicit reason and risk
• Final response includes traceability, citations, and assumptions

Helper Scripts and Assets

• Script generator: scripts/generate_script_template.sh
• Deterministic CI gate: scripts/run_ci_checks.sh
• Regression test suite: scripts/test_generator.sh
• Standard scaffold: assets/templates/standard-template.sh
• Example output style: examples/log-analyzer.sh

Reference Docs

• docs/bash-scripting-guide.md
• docs/script-patterns.md
• docs/generation-best-practices.md
• docs/text-processing-guide.md

External References

• GNU Bash Manual
• POSIX Shell Specification
• ShellCheck

Bash Scripting Guide

Table of Contents

1. Introduction 2. Bash vs POSIX sh 3. Strict Mode and Error Handling 4. Variables and Parameter Expansion 5. Functions and Scope 6. Arrays and Associative Arrays 7. Control Structures 8. Process and Command Substitution 9. Best Practices 10. Common Pitfalls

Introduction

Bash (Bourne Again Shell) is a powerful Unix shell and command language. This guide covers modern bash scripting practices and patterns for creating robust, maintainable scripts.

Bash vs POSIX sh

Key Differences

Bash-specific features (not in POSIX sh):

• Arrays: arr=(one two three)
• Associative arrays: declare -A map=([key]=value)
• [[ conditional expressions
• $(( )) arithmetic expansion with more operators
• ${var//pattern/replacement} parameter expansion
• Process substitution: <(command)
• select keyword for menus
• ** recursive globbing with shopt -s globstar

POSIX sh compatible:

• Basic variable assignment and substitution
• [ test command (single brackets)
• case statements
• Basic parameter expansion
• Command substitution with $()
• Functions (with different syntax)

When to Choose

Use Bash when:

• Script runs on modern Linux/macOS systems
• Need arrays or associative arrays
• Want advanced string manipulation
• Targeting bash-specific environments

Use POSIX sh when:

• Maximum portability required
• Running on minimal systems (embedded, containers)
• Need to run on different Unix variants
• Following strict POSIX compliance requirements

Strict Mode and Error Handling

Essential: set -euo pipefail

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Explanation:

• set -e (errexit): Exit immediately if a command exits with non-zero status
• set -u (nounset): Treat unset variables as an error
• set -o pipefail: Return value of pipeline is status of last command to exit with non-zero status
• IFS=$'\n\t': Set Internal Field Separator to newline and tab only (prevents word splitting issues)

When to Disable Strict Mode Temporarily

# Disable errexit for commands that are expected to fail
set +e
command_that_might_fail
exit_code=$?
set -e

# Or use || true for single commands
command_that_might_fail || true

# Or handle error explicitly
if ! command_that_might_fail; then
    echo "Command failed, but continuing..."
fi

Signal Handling with trap

# Cleanup function
cleanup() {
    local exit_code=$?
    echo "Cleaning up..." >&2
    rm -f "${temp_file}"
    exit "${exit_code}"
}

# Set traps
trap cleanup EXIT      # Always run cleanup on exit
trap cleanup ERR       # Run cleanup on error
trap cleanup INT TERM  # Run cleanup on interrupt or termination

# Create temp file
temp_file=$(mktemp)

# Rest of script...

Error Handling Patterns

# Pattern 1: Die function
die() {
    echo "ERROR: $*" >&2
    exit 1
}

[[ -f "${file}" ]] || die "File not found: ${file}"

# Pattern 2: Check function return values
if ! do_something; then
    echo "do_something failed" >&2
    return 1
fi

# Pattern 3: Command substitution with error handling
output=$(command 2>&1) || {
    echo "Command failed: ${output}" >&2
    exit 1
}

# Pattern 4: Validate prerequisites
check_command() {
    command -v "$1" &> /dev/null || die "Required command not found: $1"
}

check_command "jq"
check_command "curl"

Variables and Parameter Expansion

Variable Naming Conventions

# Constants - uppercase with readonly
readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/myapp/config.conf"

# Environment variables - uppercase
export PATH="${HOME}/bin:${PATH}"
export LOG_LEVEL="INFO"

# Local variables - lowercase
local counter=0
local temp_file=""

# Function names - lowercase with underscores
process_data() {
    local input="$1"
    # ...
}

Always Quote Variables

# Good - properly quoted
rm "${file}"
cp "${source}" "${destination}"
echo "Value: ${variable}"

# Bad - unquoted (prone to word splitting and globbing)
rm $file
cp $source $destination
echo "Value: $variable"

Parameter Expansion

# Default values
${var:-default}          # Use default if var is unset or empty
${var:=default}          # Set var to default if unset or empty
${var:?error message}    # Exit with error message if var is unset or empty
${var:+alternative}      # Use alternative if var is set

# String manipulation
${var#pattern}           # Remove shortest match from beginning
${var##pattern}          # Remove longest match from beginning
${var%pattern}           # Remove shortest match from end
${var%%pattern}          # Remove longest match from end
${var/pattern/replacement}    # Replace first match
${var//pattern/replacement}   # Replace all matches
${var^}                  # Uppercase first character
${var^^}                 # Uppercase all characters
${var,}                  # Lowercase first character
${var,,}                 # Lowercase all characters

# Length and substring
${#var}                  # Length of var
${var:offset}            # Substring from offset to end
${var:offset:length}     # Substring from offset with length

# Examples
file="/path/to/file.txt"
${file##*/}              # file.txt (basename)
${file%.*}               # /path/to/file (remove extension)
${file##*.}              # txt (extension only)
${file%/*}               # /path/to (dirname)

Functions and Scope

Function Definition

# POSIX style (portable)
function_name() {
    # function body
}

# Bash-specific (not portable to sh)
function function_name {
    # function body
}

# Recommended: POSIX style with local variables
process_file() {
    local input_file="$1"
    local output_file="$2"

    # Process file
    grep "pattern" "${input_file}" > "${output_file}"
}

Variable Scope

# Global variable
GLOBAL_VAR="global"

my_function() {
    # Local variable - only visible in function
    local local_var="local"

    # Modifying global variable
    GLOBAL_VAR="modified"

    # Function parameter access
    local param1="$1"
    local param2="$2"

    echo "Params: ${param1} ${param2}"
}

my_function "arg1" "arg2"

Return Values

# Functions return exit status (0-255)
check_file() {
    local file="$1"
    [[ -f "${file}" ]] && return 0 || return 1
}

# Use function return status
if check_file "data.txt"; then
    echo "File exists"
fi

# Return data via stdout
get_value() {
    echo "computed value"
}

# Capture output
result=$(get_value)

# Return data via variable (using nameref in bash 4.3+)
get_data() {
    local -n result_var=$1
    result_var="computed value"
}

get_data my_result
echo "${my_result}"

Arrays and Associative Arrays

Indexed Arrays (Bash-specific)

# Array creation
arr=()                          # Empty array
arr=(one two three)             # Initialize with values
arr[0]="first"                  # Assign to specific index

# Array operations
arr+=("four")                   # Append
${arr[0]}                       # Access element
${arr[@]}                       # All elements (as separate words)
${arr[*]}                       # All elements (as single word)
${#arr[@]}                      # Number of elements
${!arr[@]}                      # Indices

# Iterating over array
for item in "${arr[@]}"; do
    echo "${item}"
done

# Iterating with indices
for i in "${!arr[@]}"; do
    echo "Index $i: ${arr[i]}"
done

# Array slicing
${arr[@]:offset:length}         # Slice array

# Remove element
unset 'arr[1]'                  # Remove specific element

Associative Arrays (Bash 4.0+)

# Declaration required
declare -A map

# Assignment
map[key1]="value1"
map[key2]="value2"

# Or initialize
declare -A map=([key1]="value1" [key2]="value2")

# Access
${map[key1]}                    # Get value
${map[@]}                       # All values
${!map[@]}                      # All keys
${#map[@]}                      # Number of elements

# Check if key exists
if [[ -v map[key1] ]]; then
    echo "key1 exists"
fi

# Iterate over keys and values
for key in "${!map[@]}"; do
    echo "${key}: ${map[${key}]}"
done

POSIX Alternative to Arrays

# Use positional parameters
set -- one two three

# Access
echo "$1"  # one
echo "$2"  # two
echo "$#"  # count: 3

# Iterate
for item in "$@"; do
    echo "${item}"
done

# Add item
set -- "$@" "four"

# Remove first item
shift

Control Structures

Conditional Expressions

# Bash [[ ... ]] (recommended for bash)
if [[ -f "${file}" ]]; then
    echo "File exists"
fi

if [[ "${var}" == "value" ]]; then
    echo "Match"
fi

if [[ "${var}" =~ ^[0-9]+$ ]]; then
    echo "Numeric"
fi

# POSIX [ ... ] (portable)
if [ -f "${file}" ]; then
    echo "File exists"
fi

# File tests
[[ -e file ]]    # Exists
[[ -f file ]]    # Regular file
[[ -d file ]]    # Directory
[[ -L file ]]    # Symbolic link
[[ -r file ]]    # Readable
[[ -w file ]]    # Writable
[[ -x file ]]    # Executable
[[ -s file ]]    # Not empty

# String tests
[[ -z "${var}" ]]        # Empty string
[[ -n "${var}" ]]        # Non-empty string
[[ "${a}" == "${b}" ]]   # Equal
[[ "${a}" != "${b}" ]]   # Not equal
[[ "${a}" < "${b}" ]]    # Lexicographically less (bash only)

# Numeric tests
[[ "${a}" -eq "${b}" ]]  # Equal
[[ "${a}" -ne "${b}" ]]  # Not equal
[[ "${a}" -lt "${b}" ]]  # Less than
[[ "${a}" -le "${b}" ]]  # Less than or equal
[[ "${a}" -gt "${b}" ]]  # Greater than
[[ "${a}" -ge "${b}" ]]  # Greater than or equal

# Logical operators
[[ condition1 && condition2 ]]  # AND
[[ condition1 || condition2 ]]  # OR
[[ ! condition ]]                # NOT

case Statements

case "${var}" in
    pattern1)
        # commands
        ;;
    pattern2|pattern3)
        # Multiple patterns
        ;;
    *)
        # Default case
        ;;
esac

# Example with patterns
case "${file}" in
    *.txt)
        echo "Text file"
        ;;
    *.jpg|*.png)
        echo "Image file"
        ;;
    *)
        echo "Unknown type"
        ;;
esac

Loops

# while loop
while condition; do
    # commands
done

# until loop
until condition; do
    # commands
done

# for loop (C-style, bash only)
for ((i=0; i<10; i++)); do
    echo "${i}"
done

# for loop (iterating over values)
for item in one two three; do
    echo "${item}"
done

# for loop (iterating over files)
for file in *.txt; do
    echo "${file}"
done

# for loop (iterating over command output)
while IFS= read -r line; do
    echo "${line}"
done < file.txt

# Or with command substitution (avoid for large output)
for file in $(find . -name "*.txt"); do
    echo "${file}"
done

Process and Command Substitution

Command Substitution

# Recommended: $( ... )
result=$(command)
result=$(command arg1 arg2)

# Nested command substitution
outer=$(echo "Inner: $(echo "value")")

# Not recommended: backticks (legacy)
result=`command`

Process Substitution (Bash-specific)

# <( ... ) creates a named pipe/file descriptor
# Treat command output as a file

# Compare output of two commands
diff <(ls dir1) <(ls dir2)

# Use multiple inputs
paste <(cut -f1 file1) <(cut -f2 file2)

# Output redirection with process substitution
command > >(tee stdout.log) 2> >(tee stderr.log >&2)

Best Practices

Script Structure

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

# ============================================================================
# Script Name: example.sh
# Description: Brief description
# Author: Your Name
# Created: 2025-01-23
# ============================================================================

# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"

# Global variables
VERBOSE=false
DRY_RUN=false

# Functions
usage() {
    # ...
}

cleanup() {
    # ...
}

main() {
    # ...
}

# Signal handlers
trap cleanup EXIT ERR INT TERM

# Execute main
main "$@"

Always Use Quotes

# Good
echo "${variable}"
cp "${source}" "${dest}"
[[ -f "${file}" ]]

# Bad (unsafe)
echo $variable
cp $source $dest
[[ -f $file ]]

Use readonly for Constants

readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/config"

Prefer $() Over Backticks

# Good
output=$(command)
result=$(first $(second))

# Bad
output=`command`
result=`first \`second\``  # Hard to read

Check Command Existence

if ! command -v required_cmd &> /dev/null; then
    echo "Error: required_cmd not found" >&2
    exit 1
fi

Validate Inputs

# Check argument count
if [[ $# -lt 1 ]]; then
    echo "Usage: $0 <file>" >&2
    exit 1
fi

# Validate file exists
[[ -f "${file}" ]] || { echo "File not found: ${file}" >&2; exit 1; }

# Validate numeric input
[[ "${count}" =~ ^[0-9]+$ ]] || { echo "Count must be numeric" >&2; exit 1; }

Common Pitfalls

Word Splitting

# Problem: Filename with spaces
file="my file.txt"
rm $file           # Tries to remove "my" and "file.txt"

# Solution: Quote variables
rm "${file}"       # Correctly removes "my file.txt"

Globbing

# Problem: Pattern in variable
pattern="*.txt"
echo $pattern      # Expands to list of .txt files

# Solution: Quote to prevent globbing
echo "${pattern}"  # Prints "*.txt"

Useless Use of Cat (UUOC)

# Bad: Unnecessary cat
cat file.txt | grep "pattern"

# Good: Direct input
grep "pattern" file.txt

# Bad: cat in loop
cat file.txt | while read line; do
    echo "${line}"
done

# Good: redirect to while
while read -r line; do
    echo "${line}"
done < file.txt

Not Handling Spaces in Filenames

# Bad: Will break on filenames with spaces
for file in $(find . -name "*.txt"); do
    process "${file}"
done

# Good: Use while read
find . -name "*.txt" -print0 | while IFS= read -r -d '' file; do
    process "${file}"
done

# Or use globbing
for file in ./**/*.txt; do
    process "${file}"
done

Ignoring Command Exit Status

# Bad: Ignoring failure
command_that_might_fail
next_command

# Good: Check exit status
if command_that_might_fail; then
    next_command
else
    echo "Command failed" >&2
    exit 1
fi

# Or with errexit
command_that_might_fail || { echo "Failed" >&2; exit 1; }

---

References

• GNU Bash Manual
• Google Shell Style Guide
• ShellCheck - Script analysis tool
• Bash Guide for Beginners

Script Generation Best Practices

Guidelines for generating high-quality, maintainable bash scripts.

Core Principles

1. Security First - Validate inputs, quote variables, avoid injection 2. Fail Fast - Use strict mode, check errors immediately 3. Self-Documenting - Clear names, usage text, comments for complex logic 4. Testable - Modular functions, predictable behavior 5. Maintainable - Consistent style, organized structure

Script Structure Template

#!/usr/bin/env bash
#
# Script Name: descriptive-name.sh
# Description: What it does in one line
# Usage: script.sh [OPTIONS] ARGUMENTS
# Author: Name
# Created: Date
#

set -euo pipefail
IFS=$'\n\t'

# Constants (UPPERCASE, readonly)
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"

# Global variables (lowercase or Mixed_Case)
verbose=false
dry_run=false

# Functions (lowercase_with_underscores)
usage() { }
cleanup() { }
main() { }

# Signal handlers
trap cleanup EXIT ERR INT TERM

# Execute
main "$@"

Naming Conventions

# Constants - UPPERCASE with readonly
readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/app.conf"

# Environment variables - UPPERCASE
export PATH="${HOME}/bin:${PATH}"
export LOG_LEVEL="INFO"

# Global variables - lowercase or Mixed_Case
script_version="1.0.0"
temp_directory=""

# Functions - lowercase_with_underscores
process_file() { }
send_notification() { }

# Local variables - lowercase
local count=0
local file_path=""

Security Best Practices

# 1. Always quote variables
rm "${file}"                    # Good
rm $file                        # Bad

# 2. Validate all inputs
[[ "${input}" =~ ^[a-zA-Z0-9_-]+$ ]] || die "Invalid input"

# 3. Never use eval with user input
eval "${user_command}"          # Dangerous!

# 4. Validate file paths
[[ "${file}" =~ /etc ]] && die "Cannot modify /etc"
[[ -f "${file}" ]] || die "File not found"

# 5. Use $() instead of backticks
output=$(command)               # Good
output=`command`                # Bad

# 6. Set safe IFS
IFS=$'\n\t'

Error Handling Patterns

# Pattern 1: Die function
die() {
    echo "ERROR: $*" >&2
    exit 1
}

# Pattern 2: Check prerequisites
check_command() {
    command -v "$1" &> /dev/null || die "Required: $1"
}

# Pattern 3: Validate inputs
[[ $# -ge 1 ]] || die "Usage: $0 FILE"
[[ -f "$1" ]] || die "File not found: $1"

# Pattern 4: Cleanup on exit
cleanup() {
    [[ -n "${temp_dir:-}" ]] && rm -rf "${temp_dir}"
}
trap cleanup EXIT

Function Design

# Good function design
#######################################
# Process a log file and extract errors
# Globals:
#   LOG_LEVEL
# Arguments:
#   $1 - Path to log file
#   $2 - Output file (optional)
# Outputs:
#   Writes errors to stdout or file
# Returns:
#   0 on success, 1 on error
#######################################
process_log_file() {
    local log_file="$1"
    local output_file="${2:-}"

    # Validate
    [[ -f "${log_file}" ]] || return 1

    # Process
    local errors
    errors=$(grep "ERROR" "${log_file}")

    # Output
    if [[ -n "${output_file}" ]]; then
        echo "${errors}" > "${output_file}"
    else
        echo "${errors}"
    fi

    return 0
}

Code Organization

# Recommended order:
1. Shebang and header comments
2. Strict mode settings
3. Constants
4. Global variables
5. Helper functions (general → specific)
6. Main logic functions
7. Main function
8. Signal handlers
9. Main execution

Generated Code Quality Checklist

• [ ] Proper shebang: #!/usr/bin/env bash
• [ ] Strict mode enabled: set -euo pipefail
• [ ] All variables quoted: "${var}"
• [ ] Constants marked readonly
• [ ] Functions documented
• [ ] Error handling implemented
• [ ] Usage/help function included
• [ ] Input validation present
• [ ] Cleanup on exit (trap)
• [ ] No ShellCheck warnings
• [ ] Comments for complex logic
• [ ] Consistent formatting

References

• Google Shell Style Guide
• ShellCheck

Bash Script Patterns

Common patterns and templates for bash script generation.

Table of Contents

1. Argument Parsing Patterns 2. Configuration File Handling 3. Logging Frameworks 4. Parallel Processing 5. Lock Files 6. Signal Handling 7. Retry Logic

Argument Parsing Patterns

Simple getopts Pattern

#!/usr/bin/env bash
set -euo pipefail

usage() {
    cat << EOF
Usage: ${0##*/} [OPTIONS] FILE

Options:
    -h          Show this help
    -v          Verbose output
    -f FILE     Input file
    -o FILE     Output file
EOF
}

main() {
    local verbose=false
    local input_file=""
    local output_file=""

    while getopts ":hvf:o:" opt; do
        case ${opt} in
            h) usage; exit 0 ;;
            v) verbose=true ;;
            f) input_file="${OPTARG}" ;;
            o) output_file="${OPTARG}" ;;
            :) echo "Option -${OPTARG} requires an argument" >&2; exit 1 ;;
            \?) echo "Invalid option: -${OPTARG}" >&2; exit 1 ;;
        esac
    done
    shift $((OPTIND - 1))

    # Validation
    [[ -n "${input_file}" ]] || { echo "Error: -f required" >&2; exit 1; }

    # Process
    echo "Processing ${input_file}..."
}

main "$@"

Long Options Pattern

# Parse both short and long options
parse_args() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            -h|--help)
                usage
                exit 0
                ;;
            -v|--verbose)
                VERBOSE=true
                shift
                ;;
            -f|--file)
                INPUT_FILE="$2"
                shift 2
                ;;
            -o|--output)
                OUTPUT_FILE="$2"
                shift 2
                ;;
            --)
                shift
                break
                ;;
            -*)
                echo "Unknown option: $1" >&2
                exit 1
                ;;
            *)
                break
                ;;
        esac
    done

    # Remaining arguments
    REMAINING_ARGS=("$@")
}

Subcommand Pattern

#!/usr/bin/env bash
set -euo pipefail

cmd_start() {
    echo "Starting service..."
}

cmd_stop() {
    echo "Stopping service..."
}

cmd_status() {
    echo "Checking status..."
}

usage() {
    cat << EOF
Usage: ${0##*/} COMMAND [OPTIONS]

Commands:
    start       Start the service
    stop        Stop the service
    status      Check service status

Options:
    -h, --help  Show this help
EOF
}

main() {
    [[ $# -lt 1 ]] && { usage; exit 1; }

    local command="$1"
    shift

    case "${command}" in
        start)  cmd_start "$@" ;;
        stop)   cmd_stop "$@" ;;
        status) cmd_status "$@" ;;
        -h|--help) usage; exit 0 ;;
        *) echo "Unknown command: ${command}" >&2; usage; exit 1 ;;
    esac
}

main "$@"

Configuration File Handling

Source-based Configuration

# config.conf file
CONFIG_VALUE="something"
MAX_RETRIES=3
API_URL="https://api.example.com"

# In script
load_config() {
    local config_file="${1:-config.conf}"

    if [[ -f "${config_file}" ]]; then
        # shellcheck source=/dev/null
        source "${config_file}"
    else
        echo "Warning: Config file not found: ${config_file}" >&2
    fi
}

load_config "/etc/myapp/config.conf"

Key-Value Configuration Parser

# config.conf format:
# key=value
# # comments

load_config() {
    local config_file="$1"

    while IFS='=' read -r key value; do
        # Skip empty lines and comments
        [[ -z "${key}" || "${key}" =~ ^[[:space:]]*# ]] && continue

        # Trim whitespace
        key=$(echo "${key}" | tr -d '[:space:]')
        value=$(echo "${value}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')

        # Export as variable
        declare -g "${key}=${value}"
    done < "${config_file}"
}

INI-style Configuration Parser

# Parse INI format [section] key=value
parse_ini() {
    local file="$1"
    local section=""

    while IFS= read -r line; do
        # Skip empty lines and comments
        [[ -z "${line}" || "${line}" =~ ^[[:space:]]*[#;] ]] && continue

        # Section header
        if [[ "${line}" =~ ^\[([^]]+)\] ]]; then
            section="${BASH_REMATCH[1]}"
            continue
        fi

        # Key=value
        if [[ "${line}" =~ ^([^=]+)=(.*)$ ]]; then
            local key="${BASH_REMATCH[1]}"
            local value="${BASH_REMATCH[2]}"
            key=$(echo "${key}" | tr -d '[:space:]')
            value=$(echo "${value}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')

            # Store in associative array
            config["${section}.${key}"]="${value}"
        fi
    done < "${file}"
}

Logging Frameworks

Simple Logging with Levels

# LOG_LEVEL: 0=DEBUG, 1=INFO (default), 2=WARN, 3=ERROR
LOG_LEVEL=${LOG_LEVEL:-1}

# Use if-form guards — the && short-circuit form returns 1 when the
# level check fails, which triggers set -e at the call site.
log_debug() { if [[ ${LOG_LEVEL} -le 0 ]]; then echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; fi; }
log_info()  { if [[ ${LOG_LEVEL} -le 1 ]]; then echo "[INFO]  $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; fi; }
log_warn()  { if [[ ${LOG_LEVEL} -le 2 ]]; then echo "[WARN]  $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; fi; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }

File-based Logging

readonly LOG_FILE="${LOG_FILE:-/var/log/myscript.log}"

log_to_file() {
    local level="$1"
    shift
    echo "[${level}] $(date '+%Y-%m-%d %H:%M:%S') $*" >> "${LOG_FILE}"
}

log_info() {
    local msg="$*"
    echo "[INFO] ${msg}" >&2
    log_to_file "INFO" "${msg}"
}

Structured JSON Logging

log_json() {
    local level="$1"
    local message="$2"
    local timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ')

    cat <<EOF
{"timestamp":"${timestamp}","level":"${level}","message":"${message}","script":"${SCRIPT_NAME}"}
EOF
}

log_info() {
    log_json "INFO" "$*" >&2
}

Parallel Processing

Using xargs for Parallel Execution

# Process files in parallel
find . -name "*.txt" -print0 | xargs -0 -P 4 -I {} process_file {}

# With function export
process_file() {
    echo "Processing $1..."
    # ... processing logic
}
export -f process_file

find . -name "*.txt" | xargs -P 4 -I {} bash -c 'process_file "$@"' _ {}

Using GNU Parallel

# Requires: apt-get install parallel

# Simple parallel execution
parallel process_file ::: file1.txt file2.txt file3.txt

# From file list
cat files.txt | parallel process_file

# With progress bar
parallel --bar process_file ::: *.txt

# Control number of jobs
parallel -j 4 process_file ::: *.txt

Background Jobs Pattern

# Track background jobs
pids=()

# Start jobs
for file in *.txt; do
    process_file "${file}" &
    pids+=($!)
done

# Wait for all jobs
for pid in "${pids[@]}"; do
    if wait "${pid}"; then
        echo "Job ${pid} completed successfully"
    else
        echo "Job ${pid} failed" >&2
    fi
done

Lock Files

Simple Lock File

readonly LOCK_FILE="/var/lock/myscript.lock"

acquire_lock() {
    if [[ -f "${LOCK_FILE}" ]]; then
        echo "Another instance is running (lock file exists)" >&2
        exit 1
    fi

    echo $$ > "${LOCK_FILE}"
    trap 'rm -f "${LOCK_FILE}"' EXIT
}

acquire_lock

PID-based Lock with Stale Lock Detection

acquire_lock() {
    local lock_file="/var/lock/myscript.lock"

    if [[ -f "${lock_file}" ]]; then
        local old_pid=$(cat "${lock_file}")

        # Check if process is still running
        if kill -0 "${old_pid}" 2>/dev/null; then
            echo "Another instance (PID ${old_pid}) is running" >&2
            return 1
        else
            echo "Removing stale lock file" >&2
            rm -f "${lock_file}"
        fi
    fi

    echo $$ > "${lock_file}"
    trap 'rm -f "${lock_file}"' EXIT
}

Using flock for Atomic Locking

# Requires flock command

exec 200>/var/lock/myscript.lock
flock -n 200 || { echo "Another instance is running" >&2; exit 1; }

# Script runs exclusively
# Lock is released when script exits

Signal Handling

Cleanup on Exit

cleanup() {
    local exit_code=$?
    echo "Cleaning up..." >&2

    # Remove temp files
    [[ -n "${temp_dir:-}" ]] && rm -rf "${temp_dir}"

    # Release locks
    [[ -f "${lock_file:-}" ]] && rm -f "${lock_file}"

    exit "${exit_code}"
}

trap cleanup EXIT

Handling Multiple Signals

handle_sigint() {
    echo "Received SIGINT, cleaning up..." >&2
    cleanup
    exit 130  # Standard exit code for SIGINT
}

handle_sigterm() {
    echo "Received SIGTERM, cleaning up..." >&2
    cleanup
    exit 143  # Standard exit code for SIGTERM
}

trap handle_sigint INT
trap handle_sigterm TERM
trap cleanup EXIT ERR

Graceful Shutdown

SHUTDOWN=false

handle_signal() {
    echo "Shutdown signal received, finishing current work..." >&2
    SHUTDOWN=true
}

trap handle_signal INT TERM

# Main processing loop
while [[ "${SHUTDOWN}" == "false" ]]; do
    process_next_item || break
done

echo "Graceful shutdown complete" >&2

Retry Logic

Simple Retry with Backoff

retry() {
    local max_attempts=3
    local delay=1
    local attempt=1

    while [[ ${attempt} -le ${max_attempts} ]]; do
        if "$@"; then
            return 0
        else
            echo "Attempt ${attempt} failed, retrying in ${delay}s..." >&2
            sleep "${delay}"
            ((attempt++))
            ((delay*=2))  # Exponential backoff
        fi
    done

    echo "All ${max_attempts} attempts failed" >&2
    return 1
}

# Usage
retry curl -f https://api.example.com/data

Advanced Retry with Custom Parameters

retry_with_backoff() {
    local max_attempts="${1}"
    local delay="${2}"
    local max_delay="${3:-60}"
    shift 3
    local attempt=1

    while [[ ${attempt} -le ${max_attempts} ]]; do
        if "$@"; then
            return 0
        fi

        if [[ ${attempt} -lt ${max_attempts} ]]; then
            echo "Attempt ${attempt}/${max_attempts} failed" >&2
            echo "Retrying in ${delay}s..." >&2
            sleep "${delay}"

            # Exponential backoff with max cap
            delay=$((delay * 2))
            [[ ${delay} -gt ${max_delay} ]] && delay=${max_delay}
        fi

        ((attempt++))
    done

    echo "All ${max_attempts} attempts failed" >&2
    return 1
}

# Usage: retry_with_backoff MAX_ATTEMPTS INITIAL_DELAY MAX_DELAY command args...
retry_with_backoff 5 1 30 curl -f https://api.example.com/data

Retry with Jitter

retry_with_jitter() {
    local max_attempts="$1"
    local base_delay="$2"
    shift 2
    local attempt=1

    while [[ ${attempt} -le ${max_attempts} ]]; do
        if "$@"; then
            return 0
        fi

        if [[ ${attempt} -lt ${max_attempts} ]]; then
            # Add random jitter (0-100% of delay)
            local jitter=$((RANDOM % base_delay))
            local delay=$((base_delay + jitter))

            echo "Attempt ${attempt} failed, retrying in ${delay}s..." >&2
            sleep "${delay}"

            # Exponential backoff
            ((base_delay*=2))
        fi

        ((attempt++))
    done

    return 1
}

---

References

• Advanced Bash-Scripting Guide
• Bash Hackers Wiki
• Google Shell Style Guide

Text Processing Guide

Guide for choosing and using grep, awk, sed, and other text processing tools effectively in bash scripts.

Decision Tree: Which Tool to Use?

Is it a simple pattern match/filter?
├─ YES → Use grep
└─ NO
    ├─ Is it field/column-based data?
    │   └─ YES → Use awk
    └─ NO
        ├─ Is it find-and-replace or deletion?
        │   └─ YES → Use sed
        └─ NO
            └─ Complex processing → Use awk or consider Python/Perl

grep: Pattern Matching and Filtering

When to Use grep

• Searching for patterns in files
• Filtering lines by regex
• Simple line extraction
• Counting matches
• Finding files containing patterns

Common grep Patterns

# Basic search
grep "pattern" file.txt

# Case-insensitive
grep -i "error" log.txt

# Invert match (lines NOT containing pattern)
grep -v "DEBUG" log.txt

# Count matches
grep -c "ERROR" log.txt

# Show line numbers
grep -n "TODO" *.sh

# Extended regex (ERE)
grep -E "(error|fail|critical)" log.txt

# Recursive directory search
grep -r "function_name" src/

# Show filename only
grep -l "pattern" *.txt

# Show context (lines before/after)
grep -A 2 -B 2 "ERROR" log.txt  # 2 lines after and before

# Multiple patterns
grep -e "error" -e "fail" log.txt

# Read patterns from file
grep -f patterns.txt input.txt

# Whole word match
grep -w "test" file.txt  # Matches "test" but not "testing"

# Fixed string (not regex)
grep -F "a.b" file.txt  # Matches literal "a.b", not regex

# Binary file handling
grep -a "pattern" binary_file  # Treat binary as text

grep for Log Analysis

# Extract error messages from last hour
find /var/log -name "*.log" -mmin -60 -exec grep "ERROR" {} +

# Count errors by type
grep "ERROR" app.log | cut -d':' -f3 | sort | uniq -c | sort -rn

# Find errors excluding known issues
grep "ERROR" app.log | grep -v -f known_errors.txt

awk: Field-Based Text Processing

When to Use awk

• Processing structured data (CSV, logs, tables)
• Extracting specific fields
• Performing calculations
• Generating reports
• Complex conditional logic on fields

awk Basics

# Print specific fields (space-delimited by default)
awk '{print $1, $3}' file.txt

# Custom delimiter
awk -F',' '{print $1, $3}' data.csv
awk -F':' '{print $1}' /etc/passwd

# Multiple delimiters
awk -F'[,:]' '{print $1}' file.txt

# Print last field
awk '{print $NF}' file.txt

# Print all but first field
awk '{$1=""; print $0}' file.txt

awk Conditionals

# Print lines where field 3 > 100
awk '$3 > 100' data.txt

# Print lines where field matches pattern
awk '$2 ~ /error/' log.txt

# Print lines where field does NOT match
awk '$2 !~ /DEBUG/' log.txt

# Multiple conditions
awk '$3 > 100 && $4 < 500' data.txt

# If-else logic
awk '{if ($3 > 100) print "High:", $0; else print "Low:", $0}' data.txt

awk Calculations

# Sum values in column 3
awk '{sum += $3} END {print sum}' numbers.txt

# Average
awk '{sum += $3; count++} END {print sum/count}' numbers.txt

# Find max value
awk 'BEGIN {max=0} {if ($1 > max) max=$1} END {print max}' numbers.txt

# Count lines matching condition
awk '$3 > 100 {count++} END {print count}' data.txt

awk Formatted Output

# Printf-style formatting
awk '{printf "Name: %-20s Age: %3d\n", $1, $2}' people.txt

# Tab-separated output
awk 'BEGIN {OFS="\t"} {print $1, $2, $3}' file.txt

# Custom output formatting
awk '{printf "%s: %10.2f\n", $1, $2}' data.txt

awk Built-in Variables

NF      # Number of fields in current line
NR      # Current line number
FNR     # Line number in current file
FS      # Input field separator
OFS     # Output field separator
RS      # Input record separator
ORS     # Output record separator
FILENAME # Current filename

# Examples
awk '{print NR, NF, $0}' file.txt  # Line number, field count, full line
awk 'NR==10' file.txt              # Print line 10
awk 'NF > 5' file.txt              # Lines with more than 5 fields

awk for Log Analysis

# Apache/Nginx access log analysis
# Extract status codes and count
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Summarize traffic by IP
awk '{ip[$1]++} END {for (i in ip) print ip[i], i}' access.log | sort -rn

# Calculate average response time (field 11)
awk '{sum += $11; count++} END {print sum/count}' access.log

# Extract requests by hour
awk '{print substr($4, 2, 14)}' access.log | uniq -c

sed: Stream Editing

When to Use sed

• Find and replace operations
• Deleting specific lines
• In-place file editing
• Simple transformations

sed Substitution

# Basic substitution (first occurrence per line)
sed 's/old/new/' file.txt

# Global substitution (all occurrences)
sed 's/old/new/g' file.txt

# Case-insensitive substitution
sed 's/old/new/gi' file.txt

# In-place editing
sed -i 's/old/new/g' file.txt

# Backup before in-place edit
sed -i.bak 's/old/new/g' file.txt

# Replace only on specific line
sed '5s/old/new/' file.txt

# Replace on lines matching pattern
sed '/ERROR/s/old/new/g' file.txt

# Use different delimiter
sed 's|/usr/local|/opt|g' file.txt

# Backreferences
sed 's/\([0-9]*\)-\([0-9]*\)/\2-\1/' file.txt

# Multiple substitutions
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

sed Deletion

# Delete specific line
sed '5d' file.txt

# Delete range of lines
sed '5,10d' file.txt

# Delete lines matching pattern
sed '/pattern/d' file.txt

# Delete empty lines
sed '/^$/d' file.txt

# Delete lines NOT matching pattern
sed '/pattern/!d' file.txt

sed Line Operations

# Print specific line
sed -n '10p' file.txt

# Print range
sed -n '10,20p' file.txt

# Print lines matching pattern
sed -n '/ERROR/p' file.txt

# Insert line before match
sed '/pattern/i\New line before' file.txt

# Append line after match
sed '/pattern/a\New line after' file.txt

# Change entire line
sed '/pattern/c\Replacement line' file.txt

sed Advanced Patterns

# Remove comments
sed 's/#.*//' file.txt

# Remove leading whitespace
sed 's/^[ \t]*//' file.txt

# Remove trailing whitespace
sed 's/[ \t]*$//' file.txt

# Remove HTML tags
sed 's/<[^>]*>//g' file.html

# Extract text between delimiters
sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' file.html

Combining Tools: Pipeline Patterns

grep + awk

# Filter then extract fields
grep "ERROR" log.txt | awk '{print $1, $5}'

# Filter multiple patterns, process
grep -E "ERROR|WARN" log.txt | awk '{count[$2]++} END {for (i in count) print i, count[i]}'

sed + awk

# Clean then process
sed 's/[^[:print:]]//g' data.txt | awk '{sum += $2} END {print sum}'

# Remove comments, extract fields
sed 's/#.*//' config.txt | awk -F'=' '{print $1}'

Complete Pipeline Example

# Analyze web server logs
cat access.log \
    | grep "GET" \
    | grep -v "robot" \
    | sed 's/.*HTTP\/[0-9.]*" //' \
    | awk '$1 >= 200 && $1 < 300 {success++} $1 >= 400 {fail++} END {print "Success:", success, "Fail:", fail}'

Performance Tips

grep Performance

# Use -F for fixed strings (faster than regex)
grep -F "literal.string" large_file.txt

# Use -m to stop after N matches
grep -m 10 "pattern" large_file.txt

# Parallel grep for large files
parallel -j4 grep "pattern" ::: chunk1 chunk2 chunk3 chunk4

awk Performance

# Exit early if possible
awk '{if (condition) {print; exit}}' large_file.txt

# Process only needed lines
awk 'NR > 1000 {exit} {process}' large_file.txt

# Use built-in functions efficiently
awk '{count[$1]++} END {for (i in count) print i, count[i]}' file.txt

sed Performance

# Minimize patterns
sed -e 's/a/b/g' -e 's/c/d/g' file.txt  # Better than multiple sed calls

# Use in-place editing for large files
sed -i 's/old/new/g' large_file.txt  # Avoids loading entire file

Avoid Useless Use of cat

# Bad
cat file.txt | grep "pattern"
cat file.txt | awk '{print $1}'
cat file.txt | sed 's/old/new/g'

# Good
grep "pattern" file.txt
awk '{print $1}' file.txt
sed 's/old/new/g' file.txt

Real-World Examples

Example 1: CSV Processing

# Extract specific columns from CSV
awk -F',' '{print $1, $3, $5}' data.csv

# Filter rows by value
awk -F',' '$3 > 1000 {print $0}' data.csv

# Calculate sum per category
awk -F',' '{sum[$1] += $3} END {for (cat in sum) print cat, sum[cat]}' sales.csv

Example 2: Log Analysis

# Error rate over time
grep "ERROR" app.log \
    | awk '{print $1}' \
    | uniq -c \
    | awk '{print $2, $1}'

# Top 10 error messages
grep "ERROR" app.log \
    | sed 's/.*ERROR: //' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

Example 3: Configuration File Processing

# Extract non-comment, non-empty lines
sed -e 's/#.*//' -e '/^$/d' config.txt

# Convert KEY=VALUE to JSON
awk -F'=' 'BEGIN {print "{"} {printf "  \"%s\": \"%s\",\n", $1, $2} END {print "}"}' config.txt

---

References

• GNU grep Manual
• GNU awk Manual
• GNU sed Manual
• The Art of Command Line