Python Performance

Python Performance Optimization

Profiling and optimization patterns for Python code.

Table of Contents

1. Quick Start

Quick Start

# Basic timing
import timeit
time = timeit.timeit("sum(range(1000000))", number=100)
print(f"Average: {time/100:.6f}s")

Verification: Run the command with --help flag to verify availability.

When To Use

• Identifying performance bottlenecks
• Reducing application latency
• Optimizing CPU-intensive operations
• Reducing memory consumption
• Profiling production applications
• Improving database query performance

When NOT To Use

• Async concurrency - use python-async

instead

• CPU/GPU system monitoring - use conservation:cpu-gpu-performance
• Async concurrency - use python-async

instead

• CPU/GPU system monitoring - use conservation:cpu-gpu-performance

Modules

This skill is organized into focused modules for progressive loading:

profiling-tools

CPU profiling with cProfile, line profiling, memory profiling, and production profiling with py-spy. Essential for identifying where your code spends time and memory.

optimization-patterns

Ten proven optimization patterns including list comprehensions, generators, caching, string concatenation, data structures, NumPy, multiprocessing, and database operations.

memory-management

Memory optimization techniques including leak tracking with tracemalloc and weak references for caches. Depends on profiling-tools.

benchmarking-tools

Benchmarking tools including custom decorators and pytest-benchmark for verifying performance improvements.

best-practices

Best practices, common pitfalls, and exit criteria for performance optimization work. Synthesizes guidance from profiling-tools and optimization-patterns.

Exit Criteria

• Profiled code to identify bottlenecks
• Applied appropriate optimization patterns
• Verified improvements with benchmarks
• Memory usage acceptable
• No performance regressions

Benchmarking Tools

Tools and techniques for benchmarking Python code performance.

Custom Benchmark Decorator

import time
from functools import wraps

def benchmark(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"{func.__name__}: {elapsed:.6f}s")
        return result
    return wrapper

@benchmark
def my_function():
    # Code to benchmark
    pass

pytest-benchmark

# Install: pip install pytest-benchmark

def test_list_comprehension(benchmark):
    result = benchmark(lambda: [i**2 for i in range(10000)])
    assert len(result) == 10000

# Run: pytest --benchmark-compare

Best Practices

Guidelines and common pitfalls for Python performance optimization.

Best Practices

1. Profile before optimizing - Measure to find real bottlenecks 2. Focus on hot paths - Optimize frequently executed code 3. Use appropriate data structures - Dict for lookups, set for membership 4. Cache expensive computations - Use lru_cache 5. Batch I/O operations - Reduce system calls 6. Use generators for large datasets 7. Consider NumPy for numerical operations

Common Pitfalls

• Optimizing without profiling
• Using global variables unnecessarily
• Creating unnecessary copies of data
• Not using connection pooling
• Ignoring algorithmic complexity
• Over-optimizing rare code paths

Exit Criteria

• Profiled code to identify bottlenecks
• Applied appropriate optimization patterns
• Verified improvements with benchmarks
• Memory usage acceptable
• No performance regressions

Memory Management

Techniques for optimizing memory usage and preventing memory leaks.

Tracking Memory Leaks

import tracemalloc

tracemalloc.start()

# Your code here
result = memory_intensive_operation()

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("Top 10 memory allocations:")
for stat in top_stats[:10]:
    print(stat)

Weak References for Caches

import weakref

# Allows garbage collection when no strong references
weak_cache = weakref.WeakValueDictionary()

def get_resource(key):
    resource = weak_cache.get(key)
    if resource is None:
        resource = create_expensive_resource(key)
        weak_cache[key] = resource
    return resource

Optimization Patterns

Proven patterns for optimizing Python code performance.

Pattern 1: List Comprehensions vs Loops

# Slow
def slow_squares(n):
    result = []
    for i in range(n):
        result.append(i**2)
    return result

# Fast (~2x speedup)
def fast_squares(n):
    return [i**2 for i in range(n)]

Pattern 2: Generator Expressions for Memory

import sys

# Memory-intensive
list_data = [i**2 for i in range(1000000)]  # ~40MB

# Memory-efficient
gen_data = (i**2 for i in range(1000000))   # ~200 bytes

# Use generators when you only iterate once
total = sum(i**2 for i in range(1000000))

Pattern 3: String Concatenation

# Slow (O(n²))
def slow_concat(items):
    result = ""
    for item in items:
        result += str(item)
    return result

# Fast (O(n))
def fast_concat(items):
    return "".join(str(item) for item in items)

Pattern 4: Dictionary Lookups vs List Searches

# O(n) - slow for large datasets
def list_search(items, target):
    return target in items

# O(1) - constant time
def dict_search(lookup_dict, target):
    return target in lookup_dict

# Convert to set/dict for repeated lookups
lookup_set = set(items)

Pattern 5: Local Variable Access

# Global access is slower
GLOBAL_VALUE = 100

def use_global():
    total = 0
    for i in range(10000):
        total += GLOBAL_VALUE  # Slower
    return total

def use_local():
    local_value = 100  # Cache locally
    total = 0
    for i in range(10000):
        total += local_value  # Faster
    return total

Pattern 6: Caching with lru_cache

from functools import lru_cache

@lru_cache(maxsize=None)
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

# Without cache: O(2^n), With cache: O(n)

Pattern 7: NumPy for Numerical Operations

import numpy as np

# Pure Python
def python_multiply():
    a = list(range(100000))
    b = list(range(100000))
    return [x * y for x, y in zip(a, b)]

# NumPy (~100x faster)
def numpy_multiply():
    a = np.arange(100000)
    b = np.arange(100000)
    return a * b

Pattern 8: __slots__ for Memory

class RegularClass:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

class SlottedClass:
    __slots__ = ['x', 'y', 'z']

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

# SlottedClass uses ~40% less memory per instance

Pattern 9: Multiprocessing for CPU-Bound

import multiprocessing as mp

def cpu_intensive_task(n):
    return sum(i**2 for i in range(n))

def parallel_processing():
    with mp.Pool(processes=4) as pool:
        results = pool.map(cpu_intensive_task, [1000000] * 4)
    return results

Pattern 10: Batch Database Operations

# Slow: Individual commits
def slow_inserts(conn, data):
    cursor = conn.cursor()
    for item in data:
        cursor.execute("INSERT INTO items VALUES (?)", (item,))
        conn.commit()  # Commit each insert

# Fast: Batch with single commit
def fast_inserts(conn, data):
    cursor = conn.cursor()
    cursor.executemany("INSERT INTO items VALUES (?)", [(d,) for d in data])
    conn.commit()  # Single commit

Profiling Tools

detailed tools for profiling Python code performance and memory usage.

CPU Profiling with cProfile

import cProfile
import pstats
from pstats import SortKey

def main():
    # Your code here
    result = expensive_operation()
    return result

if __name__ == "__main__":
    profiler = cProfile.Profile()
    profiler.enable()

    main()

    profiler.disable()
    stats = pstats.Stats(profiler)
    stats.sort_stats(SortKey.CUMULATIVE)
    stats.print_stats(10)  # Top 10 functions

Command-line profiling:

python -m cProfile -o output.prof script.py
python -m pstats output.prof

Line Profiling

# Install: pip install line-profiler
from line_profiler import LineProfiler

def profile_function(func, *args, **kwargs):
    lp = LineProfiler()
    lp.add_function(func)
    lp_wrapper = lp(func)
    result = lp_wrapper(*args, **kwargs)
    lp.print_stats()
    return result

Memory Profiling

# Install: pip install memory-profiler
from memory_profiler import profile

@profile
def memory_intensive():
    big_list = [i for i in range(1000000)]
    big_dict = {i: i**2 for i in range(100000)}
    return sum(big_list)

# Run: python -m memory_profiler script.py

Production Profiling with py-spy

# Install: pip install py-spy

# Profile running process
py-spy top --pid 12345

# Generate flamegraph
py-spy record -o profile.svg --pid 12345

# Profile script
py-spy record -o profile.svg -- python script.py