Db Seeder

Database Seeder Skill

Seed any database with realistic fake data using ORM patterns and the Faker library. This skill provides scripts, references, and templates for efficiently populating databases with test data for development, testing, and staging environments.

When to Use This Skill

Use this skill when:

• Setting up local development databases with sample data
• Creating test fixtures for automated testing
• Populating staging environments with realistic production-like data
• Generating demo data for presentations or user onboarding
• Need to quickly create large volumes of realistic test data
• Migrating between database systems and need to populate new databases

Supported Databases

• PostgreSQL - Relational database (default for production)
• MySQL / MariaDB - Relational database
• SQLite - File-based database (testing, development)
• MongoDB - NoSQL document database
• Any ORM-supported database - Via SQLAlchemy, Django ORM, Prisma, etc.

The skill automatically detects database configuration from:

• Environment variables (DATABASE_URL, DB_TYPE, etc.)
• Configuration files (.env, settings.py, config.yaml)
• Alembic migrations (alembic.ini)
• Docker Compose files

Skill Workflow

Step 1: Detect Database Configuration

Before seeding, detect the database configuration automatically:

python scripts/detect_db_config.py

The detection script will: 1. Check environment variables (DATABASE_URL, DB_TYPE, etc.) 2. Search for configuration files (.env, settings.py, alembic.ini) 3. Analyze project structure (SQLite files, Docker Compose) 4. Output connection details and ready-to-use seeding commands

Step 1.5: Inspect Database Schema (Optional but Recommended)

NEW: Automatically inspect your database schema to generate factories and fixtures:

# Inspect schema and print summary
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb"

# Generate factory functions for all tables
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --generate-factories

# Generate JSON fixture templates
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --generate-fixtures \
  --fixture-count 5

# Both
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --generate-factories \
  --generate-fixtures

What it does:

• ✅ Detects all tables/collections in your database
• ✅ Analyzes column types (VARCHAR, INTEGER, DATE, etc.)
• ✅ Identifies foreign key relationships
• ✅ Generates appropriate Faker methods for each field
• ✅ Creates ready-to-use factory functions (generated_factories.py)
• ✅ Creates JSON fixture templates (generated_fixtures.json)

Works with ANY schema - No hardcoded assumptions!

Manual Configuration: If auto-detection fails, specify database details manually:

# PostgreSQL
python scripts/seed_database.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost:5432/mydb" \
  --count 100

# SQLite
python scripts/seed_database.py \
  --db sqlite \
  --connection "sqlite:///./test.db" \
  --count 50

# MongoDB
python scripts/seed_database.py \
  --db mongodb \
  --connection "mongodb://admin:pass@localhost:27017/mydb" \
  --count 100

Configuration Files Reference: For detailed database connection patterns and configuration examples, refer to: references/database-configs.md

Step 2: Choose Seeding Approach

Three primary approaches are available:

Approach A: Generate JSON Fixtures First (Recommended for Reusability)

Generate reusable JSON fixtures that can be version-controlled and shared:

# Generate using predefined templates
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/elios_data.json \
  --pretty

# Generate specific models
python scripts/generate_fixtures.py \
  --models User:100,Post:500,Candidate:50 \
  --output fixtures/test_data.json \
  --pretty

# Seed database from fixtures
python scripts/seed_database.py \
  --fixtures fixtures/test_data.json \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/db"

Benefits:

• Fixtures can be version-controlled
• Reusable across environments
• Consistent test data
• Easy to share with team

Approach B: Direct Database Seeding with Custom Factories

Create Python factory functions and seed directly:

# Create seeding script: scripts/seed_elios.py

from seed_database import DatabaseSeeder, create_seeder

def candidate_factory(fake, index):
    return {
        'full_name': fake.name(),
        'email': fake.email(),
        'years_of_experience': fake.random_int(min=0, max=15),
        'skills': fake.random_elements(
            ['Python', 'JavaScript', 'SQL', 'React'],
            length=fake.random_int(min=2, max=6),
            unique=True
        ),
        'status': fake.random_element(['pending', 'interviewed', 'hired']),
    }

# Run seeding
seeder = create_seeder('postgresql', 'postgresql://user:pass@localhost/db')
seeder.seed_model(Candidate, count=50, factory_func=candidate_factory)

Benefits:

• Full control over data generation
• Can use complex business logic
• Direct database insertion (faster)

Approach C: Configuration-Based Seeding

Use YAML configuration file for declarative seeding:

# Copy template
cp assets/seed-config-template.yaml seed-config.yaml

# Edit seed-config.yaml to define models and factories

# Run seeding
python scripts/seed_database.py --config seed-config.yaml

Benefits:

• Declarative configuration
• No code required
• Easy to modify counts and settings

Step 3: Execute Seeding

Based on chosen approach:

For Fixtures:

python scripts/seed_database.py \
  --fixtures fixtures/test_data.json \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/db"

For Custom Factories:

python scripts/seed_elios.py

For Configuration:

python scripts/seed_database.py --config seed-config.yaml

Step 4: Verify Seeding

After seeding completes, verify the data:

# For PostgreSQL/MySQL
psql -d mydb -c "SELECT COUNT(*) FROM users;"
psql -d mydb -c "SELECT COUNT(*) FROM candidates;"

# For SQLite
sqlite3 mydb.db "SELECT COUNT(*) FROM users;"

# For MongoDB
mongosh mydb --eval "db.users.countDocuments()"

Bundled Resources

Scripts (scripts/)

seed_database.py

Main seeding orchestrator with Faker integration.

Features:

• Auto-detects database type from connection string
• Supports SQLAlchemy-based databases (PostgreSQL, MySQL, SQLite)
• Supports MongoDB with PyMongo
• Batch insertion for performance
• Progress reporting
• Error handling and rollback

Usage:

# Seed from fixtures
python scripts/seed_database.py \
  --fixtures data.json \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/db"

# Seed from config
python scripts/seed_database.py --config seed-config.yaml

# Custom locale for international data
python scripts/seed_database.py \
  --fixtures data.json \
  --db sqlite \
  --connection "sqlite:///test.db" \
  --locale fr_FR

detect_db_config.py

Automatically detects database configuration from project.

Features:

• Scans environment variables
• Parses configuration files (.env, settings.py, config.yaml)
• Detects Alembic migrations configuration
• Finds SQLite database files
• Outputs connection strings with masked passwords

Usage:

# Auto-detect
python scripts/detect_db_config.py

# Specify config file
python scripts/detect_db_config.py --config-path src/infrastructure/config/settings.py

# Specify .env file
python scripts/detect_db_config.py --env-file .env.local

# Specify project root
python scripts/detect_db_config.py --project-root /path/to/project

generate_fixtures.py

Generates JSON fixtures with realistic fake data.

Features:

• Predefined templates (Elios interview system, blog, e-commerce)
• Custom model generation
• Configurable record counts
• Multiple Faker locales
• Pretty-printed JSON output

Usage:

# Generate from template
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures.json \
  --pretty

# Generate specific models
python scripts/generate_fixtures.py \
  --models User:100,Post:500 \
  --output test_data.json

# Use different locale (Vietnamese)
python scripts/generate_fixtures.py \
  --template elios-interview \
  --locale vi_VN \
  --output vietnamese_data.json \
  --pretty

# Japanese locale
python scripts/generate_fixtures.py \
  --template blog \
  --locale ja_JP \
  --output japanese_blog_data.json

Available Templates:

• elios-interview - Candidates, Questions, Interview Sessions (Elios-specific)
• blog - Users, Posts
• More templates can be added to the script

Supported Locales:

• en_US - English (United States) - Default
• vi_VN - Vietnamese (Vietnam) - Includes Vietnamese universities, degrees, majors
• ja_JP - Japanese (Japan)
• fr_FR - French (France)
• en_GB - English (United Kingdom)
• And 50+ more locales supported by Faker

inspect_schema.py ⭐ NEW

Automatically inspects database schema and generates seeding helpers.

Features:

• Discovers all tables/collections automatically
• Analyzes column types (VARCHAR, INTEGER, DATE, JSONB, etc.)
• Identifies foreign key relationships
• Generates appropriate Faker methods for each field type
• Creates ready-to-use factory functions
• Creates JSON fixture templates with sample data
• Works with ANY database schema - No hardcoded assumptions!

Usage:

# Print schema summary
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb"

# Generate factory functions for all tables
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --generate-factories
# Output: generated_factories.py

# Generate JSON fixture templates
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --generate-fixtures \
  --fixture-count 5
# Output: generated_fixtures.json

# Generate both
python scripts/inspect_schema.py \
  --db sqlite \
  --connection "sqlite:///./test.db" \
  --generate-factories \
  --generate-fixtures

# Save schema info as JSON
python scripts/inspect_schema.py \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb" \
  --output schema_info.json

Smart Field Detection: The script intelligently detects field types and generates appropriate Faker methods:

• email field → fake.email()
• phone field → fake.phone_number()
• address field → fake.address()
• first_name field → fake.first_name()
• description field → fake.paragraph()
• created_at (TIMESTAMP) → fake.date_time_between()
• Integer types → fake.random_int()
• Boolean types → fake.boolean()
• And many more patterns...

Example Output:

# generated_factories.py (auto-generated)

def candidate_factory(fake, index):
    """Factory for candidate model"""
    return {
        'full_name': fake.name(),
        'email': fake.email(),
        'phone': fake.phone_number(),
        'years_of_experience': fake.random_int(min=1, max=1000),
        'skills': {},  # JSON field
        'created_at': fake.date_time_between(start_date='-1y', end_date='now'),
        'status': fake.word(),
    }

References (references/)

database-configs.md

Comprehensive database connection patterns and configuration examples.

Contents:

• Connection string formats for all supported databases
• Environment variable patterns
• Configuration file examples (.env, settings.py, config.yaml)
• Docker Compose configurations
• Security best practices
• Common issues and solutions

Use when:

• Setting up database connections
• Troubleshooting connection errors
• Configuring different environments (dev, staging, prod)

faker-recipes.md

Common patterns and examples for generating realistic fake data with Faker.

Contents:

• Personal information (names, emails, addresses)
• Business data (companies, jobs)
• Technical data (URLs, IPs, UUIDs)
• Dates and times
• Text generation
• Numbers and sequences
• Localization
• Custom providers
• Database-specific factory patterns

Use when:

• Creating custom factory functions
• Need inspiration for data generation
• Understanding Faker capabilities
• Creating project-specific data generators

orm-patterns.md

Patterns and best practices for seeding databases using various ORMs.

Contents:

• SQLAlchemy - Setup, batch operations, relationships, error handling, factories
• Django ORM - Models, bulk operations, management commands
• Prisma (TypeScript) - Seeding scripts, relations
• MongoDB (PyMongo) - Document insertion, embedded docs, references
• Best practices (transactions, idempotency, constraints, progress reporting)

Use when:

• Implementing ORM-specific seeding
• Need examples for your ORM
• Understanding relationship seeding (one-to-many, many-to-many)
• Creating production-grade seeding scripts

Assets (assets/)

seed-config-template.yaml

Template for YAML-based seeding configuration.

Features:

• Database connection configuration
• Faker settings (locale, seed for reproducibility)
• Model definitions with factory functions
• Seeding options (batch size, clear existing, idempotency)
• Post-seeding hooks

Usage:

# Copy template
cp assets/seed-config-template.yaml seed-config.yaml

# Edit configuration
# ... customize models, counts, factories ...

# Run seeding
python scripts/seed_database.py --config seed-config.yaml

fixture-template.json

Template for JSON test fixtures.

Features:

• Example structure for common models (User, Post)
• Elios-specific models (Candidate, Question, InterviewSession)
• Proper JSON formatting with relationships
• Nested data examples (embedded documents, foreign keys)

Usage:

# Copy and customize
cp assets/fixture-template.json fixtures/my_data.json

# Edit JSON file with your data
# ...

# Seed database
python scripts/seed_database.py \
  --fixtures fixtures/my_data.json \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/db"

Common Workflows

Workflow 1: Quick Development Setup

Seed local database with sample data for immediate development:

# 1. Auto-detect database
python scripts/detect_db_config.py

# 2. Generate fixtures
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/dev_data.json

# 3. Seed database
python scripts/seed_database.py \
  --fixtures fixtures/dev_data.json \
  --db postgresql \
  --connection "postgresql://postgres:password@localhost/elios_dev"

# Verify
psql -d elios_dev -c "SELECT COUNT(*) FROM candidates;"

Workflow 2: Test Fixture Creation

Create fixtures for automated testing:

# Generate small, focused test fixtures
python scripts/generate_fixtures.py \
  --models Candidate:10,Question:20,InterviewSession:5 \
  --output tests/fixtures/test_data.json \
  --pretty

# Use in tests:
# - Version control fixtures/test_data.json
# - Load in test setup
# - Consistent test data across CI/CD

Workflow 3: Staging Environment Population

Populate staging with production-like data:

# 1. Generate large dataset
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/staging_data.json

# Manually increase counts in fixture file if needed

# 2. Detect staging database
python scripts/detect_db_config.py --env-file .env.staging

# 3. Seed staging
python scripts/seed_database.py \
  --fixtures fixtures/staging_data.json \
  --db postgresql \
  --connection "$STAGING_DATABASE_URL"

Workflow 4: Database Migration Testing

Test migrations with seeded data:

# 1. Seed old schema
python scripts/seed_database.py --fixtures fixtures/old_schema.json

# 2. Run migrations
alembic upgrade head

# 3. Verify data integrity
python scripts/verify_migration.py

# 4. Seed new fields (if needed)
python scripts/seed_database.py --fixtures fixtures/new_fields.json

Best Practices

1. Version Control Fixtures

Store fixtures in version control for consistency:

# Create fixtures directory
mkdir -p fixtures/{development,testing,staging}

# Generate and commit
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/development/base_data.json \
  --pretty

git add fixtures/
git commit -m "Add base development fixtures"

2. Use Reproducible Seeds

For consistent test data, use Faker seeds:

from faker import Faker

# Set seed for reproducibility
Faker.seed(12345)
fake = Faker()

# Always generates same data
fake.name()  # Always "John Smith" (example)

3. Separate Seeding Scripts by Environment

scripts/
├── seed_development.py    # Small datasets for local dev
├── seed_testing.py         # Controlled fixtures for tests
├── seed_staging.py         # Large production-like datasets
└── seed_demo.py            # Curated demo data

4. Idempotent Seeding

Ensure seeding can be run multiple times safely:

def seed_users(session, count=100):
    # Check if already seeded
    existing_count = session.query(User).count()
    if existing_count >= count:
        print(f"Already seeded with {existing_count} users, skipping...")
        return

    # Seed remaining
    remaining = count - existing_count
    # ... create users ...

5. Progress Reporting for Large Datasets

# Batch insertions with progress
batch_size = 1000
for i in range(0, count, batch_size):
    # ... create batch ...
    print(f"Progress: {i}/{count} ({(i/count)*100:.1f}%)")

Dependencies

Required

• Python 3.8+
• Faker (pip install faker)

Optional (based on database)

• SQLAlchemy (pip install sqlalchemy) - For PostgreSQL, MySQL, SQLite
• psycopg2 (pip install psycopg2-binary) - PostgreSQL driver
• pymysql (pip install pymysql) - MySQL driver
• pymongo (pip install pymongo) - MongoDB driver
• PyYAML (pip install pyyaml) - For YAML config support

Installation

# Install core dependencies
pip install faker pyyaml

# Install database drivers (choose based on your database)
pip install sqlalchemy psycopg2-binary  # PostgreSQL
pip install sqlalchemy pymysql          # MySQL
pip install pymongo                     # MongoDB

Troubleshooting

Issue: "Faker not installed"

Error: Faker library not installed

Solution:

pip install faker

Issue: Database connection refused

Error: connection refused

Solutions: 1. Check if database is running 2. Verify connection string (host, port, credentials) 3. Check firewall settings 4. Refer to references/database-configs.md for detailed troubleshooting

Issue: Unique constraint violation

Error: duplicate key value violates unique constraint

Solutions: 1. Clear database before seeding: session.query(Model).delete() 2. Use idempotent seeding (check existing records) 3. Generate unique values with Faker

Issue: Foreign key constraint violation

Error: foreign key constraint fails

Solutions: 1. Seed in correct order (parent models before children) 2. Use actual existing IDs for foreign keys 3. Store created records for reference: created_users = [...]; post.author_id = random.choice([u.id for u in created_users])

Issue: Out of memory for large datasets

Error: MemoryError

Solutions: 1. Use batch insertions with commits: session.flush() every N records 2. Reduce batch size 3. Stream data instead of loading all in memory

Advanced Usage

Custom Faker Provider

Create domain-specific providers:

from faker import Faker
from faker.providers import BaseProvider

class InterviewProvider(BaseProvider):
    def interview_status(self):
        return self.random_element(['pending', 'scheduled', 'completed', 'cancelled'])

    def skill_level(self):
        return self.random_element(['beginner', 'intermediate', 'advanced', 'expert'])

    def programming_language(self):
        return self.random_element(['Python', 'JavaScript', 'Java', 'C++', 'Go'])

fake = Faker()
fake.add_provider(InterviewProvider)

# Use custom methods
candidate = {
    'skill_level': fake.skill_level(),
    'primary_language': fake.programming_language(),
}

Multi-Locale Data Generation

Generate international test data:

from faker import Faker

# Create multiple locales
fake_us = Faker('en_US')
fake_jp = Faker('ja_JP')
fake_fr = Faker('fr_FR')

users = [
    {'name': fake_us.name(), 'address': fake_us.address()},
    {'name': fake_jp.name(), 'address': fake_jp.address()},
    {'name': fake_fr.name(), 'address': fake_fr.address()},
]

Seeding with Relationships

Handle complex relationships:

# One-to-Many
author = User(username=fake.user_name())
session.add(author)
session.flush()  # Get author.id

posts = [
    Post(title=fake.sentence(), author_id=author.id)
    for _ in range(10)
]
session.add_all(posts)

# Many-to-Many
skills = [Skill(name=name) for name in ['Python', 'JavaScript', 'SQL']]
session.add_all(skills)
session.flush()

candidate = Candidate(full_name=fake.name())
candidate.skills.extend(random.sample(skills, k=2))
session.add(candidate)

session.commit()

Examples

Example 1: Seed Elios Interview System

# Generate Elios-specific fixtures
python scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/elios_dev.json \
  --pretty

# Seed database
python scripts/seed_database.py \
  --fixtures fixtures/elios_dev.json \
  --db postgresql \
  --connection "postgresql://postgres:password@localhost/elios_dev"

# Verify
psql -d elios_dev -c "SELECT COUNT(*) FROM candidates;"
psql -d elios_dev -c "SELECT COUNT(*) FROM questions;"
psql -d elios_dev -c "SELECT COUNT(*) FROM interview_sessions;"

Example 2: Create Test Fixtures for CI/CD

# Generate small, controlled fixtures
python scripts/generate_fixtures.py \
  --models Candidate:5,Question:10 \
  --output tests/fixtures/ci_test_data.json \
  --pretty

# In CI pipeline:
python scripts/seed_database.py \
  --fixtures tests/fixtures/ci_test_data.json \
  --db sqlite \
  --connection "sqlite:///:memory:"

# Run tests with seeded data
pytest tests/

Example 3: Generate Vietnamese Test Data

# Generate Vietnamese fixtures for Elios
python scripts/generate_fixtures.py \
  --template elios-interview \
  --locale vi_VN \
  --output fixtures/elios_vietnamese.json \
  --pretty

# Seed database
python scripts/seed_database.py \
  --fixtures fixtures/elios_vietnamese.json \
  --db postgresql \
  --connection "postgresql://postgres:password@localhost/elios_dev"

# Verify Vietnamese data
psql -d elios_dev -c "SELECT full_name, education->>'university' as university FROM candidates LIMIT 5;"

Expected output:

        full_name        |          university
-------------------------+-------------------------------
 Nguyễn Văn Minh         | Đại học Bách Khoa Hà Nội
 Trần Thị Hương          | Đại học FPT
 Lê Minh Tuấn            | Đại học Quốc gia Hà Nội
 Phạm Thị Lan            | Đại học Công nghệ
 Hoàng Văn Nam           | Đại học Bách Khoa TP.HCM

Example 4: Custom Factory Script

# scripts/seed_custom.py

from seed_database import create_seeder
from faker import Faker

fake = Faker()

def advanced_candidate_factory(fake, index):
    """Generate realistic interview candidates"""
    skills_by_role = {
        'frontend': ['JavaScript', 'React', 'CSS', 'HTML', 'Vue'],
        'backend': ['Python', 'Django', 'FastAPI', 'SQL', 'Docker'],
        'fullstack': ['JavaScript', 'React', 'Python', 'SQL', 'AWS'],
        'data': ['Python', 'Pandas', 'SQL', 'Machine Learning', 'Statistics'],
    }

    role = fake.random_element(list(skills_by_role.keys()))
    skills = fake.random_elements(
        skills_by_role[role],
        length=fake.random_int(min=3, max=5),
        unique=True
    )

    return {
        'full_name': fake.name(),
        'email': fake.email(),
        'phone': fake.phone_number(),
        'years_of_experience': fake.random_int(min=0, max=15),
        'skills': list(skills),
        'desired_role': role,
        'expected_salary': fake.random_int(min=50000, max=200000),
        'created_at': fake.date_time_between(start_date='-6m', end_date='now'),
    }

# Run seeding
seeder = create_seeder('postgresql', 'postgresql://user:pass@localhost/db')
candidates = seeder.seed_model(Candidate, count=100, factory_func=advanced_candidate_factory)
print(f"✓ Created {len(candidates)} candidates")

Integration with Development Workflow

Docker Compose Integration

Add seeding to Docker Compose setup:

# docker-compose.yml

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: elios_dev
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"

  seeder:
    build: .
    depends_on:
      - postgres
    environment:
      DATABASE_URL: postgresql://postgres:password@postgres:5432/elios_dev
    command: >
      sh -c "
        sleep 5 &&
        python scripts/generate_fixtures.py --template elios-interview --output /tmp/fixtures.json &&
        python scripts/seed_database.py --fixtures /tmp/fixtures.json --db postgresql --connection $$DATABASE_URL
      "

Makefile Integration

# Makefile

.PHONY: seed seed-dev seed-test

seed-dev:
	python scripts/generate_fixtures.py --template elios-interview --output fixtures/dev.json
	python scripts/seed_database.py --fixtures fixtures/dev.json --db postgresql --connection $(DATABASE_URL)

seed-test:
	python scripts/generate_fixtures.py --models Candidate:10,Question:20 --output tests/fixtures/test.json
	python scripts/seed_database.py --fixtures tests/fixtures/test.json --db sqlite --connection "sqlite:///:memory:"

seed-staging:
	python scripts/detect_db_config.py --env-file .env.staging
	python scripts/seed_database.py --fixtures fixtures/staging.json --db postgresql --connection $(STAGING_DATABASE_URL)

Usage:

make seed-dev
make seed-test
make seed-staging

Summary

This skill provides a complete database seeding solution:

1. Auto-detection - Automatically finds database configuration 2. Multiple approaches - Fixtures, factories, or configuration-based 3. Comprehensive references - Database configs, Faker recipes, ORM patterns 4. Ready-to-use templates - JSON fixtures and YAML configs 5. Production-ready scripts - Batch operations, error handling, progress reporting

Use the skill whenever you need to populate databases with realistic test data efficiently.

Database Seeder Skill

A comprehensive Claude Code skill for seeding databases with realistic fake data.

Installation

Option 1: Extract Archive

# Extract the skill archive
tar -xzf db-seeder.tar.gz -C .claude/skills/

# Install dependencies
pip install faker pyyaml sqlalchemy psycopg2-binary

Option 2: Manual Copy

Copy the db-seeder directory to your project's .claude/skills/ folder.

Quick Start

1. Detect Your Database Configuration

python .claude/skills/db-seeder/scripts/detect_db_config.py

2. Generate Test Fixtures

# For Elios project
python .claude/skills/db-seeder/scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/elios_data.json \
  --pretty

# For custom models
python .claude/skills/db-seeder/scripts/generate_fixtures.py \
  --models User:100,Post:500 \
  --output fixtures/test_data.json

3. Seed Database

python .claude/skills/db-seeder/scripts/seed_database.py \
  --fixtures fixtures/elios_data.json \
  --db postgresql \
  --connection "postgresql://user:pass@localhost/mydb"

Features

• Auto-Detection: Automatically detects database configuration from environment variables, config files, and project structure
• Multiple Databases: PostgreSQL, MySQL, SQLite, MongoDB support
• ORM Integration: Works with SQLAlchemy, Django ORM, Prisma
• Faker Integration: Generate realistic fake data with 100+ data types
• Flexible Approaches: JSON fixtures, Python factories, or YAML configuration
• Production-Ready: Batch operations, error handling, progress reporting

Skill Contents

Scripts

• seed_database.py - Main seeding orchestrator
• detect_db_config.py - Auto-detect database configuration
• generate_fixtures.py - Generate JSON fixtures with Faker

References

• database-configs.md - Database connection patterns and troubleshooting
• faker-recipes.md - Common Faker patterns and examples
• orm-patterns.md - ORM-specific seeding patterns

Assets

• seed-config-template.yaml - Configuration template
• fixture-template.json - JSON fixture template

Use Cases

1. Development Setup - Seed local databases with sample data 2. Testing - Create consistent test fixtures for CI/CD 3. Staging - Populate staging environments with realistic data 4. Demo - Generate demo data for presentations

Example: Seed Elios Interview System

# Generate Elios-specific fixtures
python .claude/skills/db-seeder/scripts/generate_fixtures.py \
  --template elios-interview \
  --output fixtures/elios_dev.json \
  --pretty

# Seed database
python .claude/skills/db-seeder/scripts/seed_database.py \
  --fixtures fixtures/elios_dev.json \
  --db postgresql \
  --connection "postgresql://postgres:password@localhost/elios_dev"

# Verify
psql -d elios_dev -c "SELECT COUNT(*) FROM candidates;"
psql -d elios_dev -c "SELECT COUNT(*) FROM questions;"
psql -d elios_dev -c "SELECT COUNT(*) FROM interview_sessions;"

Documentation

See SKILL.md for complete documentation including:

• Detailed workflows
• Advanced usage
• Best practices
• Troubleshooting guide
• Integration examples

Dependencies

Required

• Python 3.8+
• Faker (pip install faker)

Optional (based on database)

• SQLAlchemy + psycopg2-binary (PostgreSQL)
• SQLAlchemy + pymysql (MySQL)
• pymongo (MongoDB)
• PyYAML (YAML config support)

Quick Reference

Common Commands

# Auto-detect database
python scripts/detect_db_config.py

# Generate fixtures
python scripts/generate_fixtures.py --template elios-interview --output fixtures.json

# Seed database
python scripts/seed_database.py --fixtures fixtures.json --db postgresql --connection "DB_URL"

# Custom locale
python scripts/generate_fixtures.py --template blog --locale ja_JP --output japanese_data.json

Supported Databases

Support

For issues or questions: 1. Check SKILL.md for detailed documentation 2. Review references/ for specific topics 3. Check assets/ for templates

License

Created for Elios AI Interview Service project.

Database Configuration Reference

This reference provides database connection patterns and configuration examples for various database systems.

PostgreSQL

Connection String Format

postgresql://username:password@host:port/database

Environment Variables

DATABASE_URL=postgresql://postgres:password@localhost:5432/elios_dev
# Or individual components
DB_TYPE=postgresql
DB_HOST=localhost
DB_PORT=5432
DB_NAME=elios_dev
DB_USER=postgres
DB_PASSWORD=password

SQLAlchemy Configuration

from sqlalchemy import create_engine

engine = create_engine('postgresql://postgres:password@localhost:5432/elios_dev')

Default Port

5432

MySQL / MariaDB

Connection String Format

mysql://username:password@host:port/database

Environment Variables

DATABASE_URL=mysql://root:password@localhost:3306/elios_dev
# Or individual components
DB_TYPE=mysql
DB_HOST=localhost
DB_PORT=3306
DB_NAME=elios_dev
DB_USER=root
DB_PASSWORD=password

SQLAlchemy Configuration

from sqlalchemy import create_engine

engine = create_engine('mysql://root:password@localhost:3306/elios_dev')

Default Port

3306

SQLite

Connection String Format

sqlite:///path/to/database.db
# Or for in-memory database
sqlite:///:memory:

Environment Variables

DATABASE_URL=sqlite:///./elios_dev.db
# Or
DB_TYPE=sqlite
DB_PATH=./elios_dev.db

SQLAlchemy Configuration

from sqlalchemy import create_engine

# File-based
engine = create_engine('sqlite:///./elios_dev.db')

# In-memory
engine = create_engine('sqlite:///:memory:')

Default Port

N/A (file-based database)

MongoDB

Connection String Format

mongodb://username:password@host:port/database
# With authentication database
mongodb://username:password@host:port/database?authSource=admin

Environment Variables

MONGODB_URI=mongodb://admin:password@localhost:27017/elios_dev
# Or individual components
DB_TYPE=mongodb
DB_HOST=localhost
DB_PORT=27017
DB_NAME=elios_dev
DB_USER=admin
DB_PASSWORD=password

PyMongo Configuration

from pymongo import MongoClient

client = MongoClient('mongodb://admin:password@localhost:27017/')
db = client['elios_dev']

Default Port

27017

Configuration File Patterns

.env File

# PostgreSQL
DATABASE_URL=postgresql://postgres:password@localhost:5432/elios_dev

# MySQL
DATABASE_URL=mysql://root:password@localhost:3306/elios_dev

# SQLite
DATABASE_URL=sqlite:///./elios_dev.db

# MongoDB
MONGODB_URI=mongodb://admin:password@localhost:27017/elios_dev

settings.py (Pydantic)

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    db_type: str = "postgresql"
    db_host: str = "localhost"
    db_port: int = 5432
    db_name: str = "elios_dev"
    db_user: str = "postgres"
    db_password: str = ""

    class Config:
        env_file = ".env"

config.yaml

database:
  type: postgresql
  host: localhost
  port: 5432
  name: elios_dev
  user: postgres
  password: password

alembic.ini (SQLAlchemy Migrations)

[alembic]
sqlalchemy.url = postgresql://postgres:password@localhost:5432/elios_dev

Docker Compose Configuration

PostgreSQL

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: elios_dev
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Connection string: postgresql://postgres:password@localhost:5432/elios_dev

MySQL

version: '3.8'
services:
  mysql:
    image: mysql:8
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: elios_dev
    ports:
      - "3306:3306"
    volumes:
      - mysql_data:/var/lib/mysql

volumes:
  mysql_data:

Connection string: mysql://root:password@localhost:3306/elios_dev

MongoDB

version: '3.8'
services:
  mongodb:
    image: mongo:6
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: password
      MONGO_INITDB_DATABASE: elios_dev
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db

volumes:
  mongodb_data:

Connection string: mongodb://admin:password@localhost:27017/elios_dev

Connection String Security

Best Practices

1. Never commit connection strings with credentials to version control

• Use .env files (add to .gitignore)
• Use environment variables
• Use secret management systems (AWS Secrets Manager, HashiCorp Vault)

2. Use connection string encoding for special characters

   from urllib.parse import quote_plus

   password = "p@ssw0rd!"
   encoded = quote_plus(password)  # "p%40ssw0rd%21"
   connection = f"postgresql://user:{encoded}@localhost/db"

3. Use SSL/TLS for remote connections

   # PostgreSQL with SSL
   postgresql://user:pass@host/db?sslmode=require

   # MySQL with SSL
   mysql://user:pass@host/db?ssl-mode=REQUIRED

   # MongoDB with SSL
   mongodb://user:pass@host/db?ssl=true

4. Rotate credentials regularly

5. Use least-privilege database users

• Create separate users for seeding vs. production
• Limit permissions to only what's needed

Common Issues and Solutions

PostgreSQL Connection Refused

Error: connection refused

Solutions:

• Check if PostgreSQL is running: pg_isready
• Verify port: 5432 (default)
• Check pg_hba.conf for connection permissions
• Ensure firewall allows connection

MySQL Access Denied

Error: Access denied for user 'root'@'localhost'

Solutions:

• Verify username/password
• Check MySQL user permissions: SHOW GRANTS FOR 'root'@'localhost';
• Reset password if needed

SQLite Database Locked

Error: database is locked

Solutions:

• Close other connections to the database
• Use sqlite3 CLI to check: .databases
• Restart the application

MongoDB Authentication Failed

Error: Authentication failed

Solutions:

• Verify authSource parameter (usually admin)
• Check user exists: db.getUsers() in mongo shell
• Ensure correct database in connection string

Detection Script Usage

The detect_db_config.py script automatically detects database configuration:

# Auto-detect from environment and config files
python scripts/detect_db_config.py

# Specify config file
python scripts/detect_db_config.py --config-path src/infrastructure/config/settings.py

# Specify .env file
python scripts/detect_db_config.py --env-file .env

# Specify project root
python scripts/detect_db_config.py --project-root /path/to/project

The script will output:

• Detected database type
• Connection details (with masked password)
• Ready-to-use command for seed_database.py

Faker Recipes and Patterns

Common patterns and examples for generating realistic fake data using the Faker library.

Basic Usage

from faker import Faker

fake = Faker()

# Generate single value
name = fake.name()
email = fake.email()

# Generate multiple values
names = [fake.name() for _ in range(10)]

Personal Information

Names

fake.name()                    # "John Smith"
fake.first_name()              # "John"
fake.last_name()               # "Smith"
fake.name_male()               # "Michael Johnson"
fake.name_female()             # "Sarah Williams"
fake.prefix()                  # "Dr."
fake.suffix()                  # "Jr."

Contact Information

fake.email()                   # "john.smith@example.com"
fake.safe_email()              # "john.smith@example.org" (safe domains)
fake.company_email()           # "john.smith@company.com"
fake.phone_number()            # "+1-555-123-4567"
fake.address()                 # "123 Main St, Springfield, IL 62701"
fake.street_address()          # "123 Main St"
fake.city()                    # "Springfield"
fake.state()                   # "Illinois"
fake.zipcode()                 # "62701"
fake.country()                 # "United States"

Business Data

Company Information

fake.company()                 # "Tech Corp Inc."
fake.company_suffix()          # "Inc."
fake.job()                     # "Software Engineer"
fake.bs()                      # "synergize innovative solutions" (business speak)
fake.catch_phrase()            # "Innovative solutions for tomorrow"

Technical Data

Internet and URLs

fake.url()                     # "https://example.com"
fake.uri()                     # "/path/to/resource"
fake.domain_name()             # "example.com"
fake.ipv4()                    # "192.168.1.1"
fake.ipv6()                    # "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
fake.mac_address()             # "00:1B:44:11:3A:B7"
fake.user_agent()              # "Mozilla/5.0 ..."
fake.slug()                    # "this-is-a-slug"

File and Storage

fake.file_name()               # "document.pdf"
fake.file_extension()          # "pdf"
fake.mime_type()               # "application/pdf"
fake.file_path()               # "/home/user/document.pdf"
fake.uuid4()                   # "550e8400-e29b-41d4-a716-446655440000"

Colors and Images

fake.color_name()              # "Red"
fake.hex_color()               # "#FF5733"
fake.rgb_color()               # "rgb(255, 87, 51)"
fake.image_url()               # "https://picsum.photos/640/480"

Dates and Times

from datetime import datetime, timedelta

# Random dates
fake.date()                                    # "2023-05-15"
fake.date_of_birth(minimum_age=18, maximum_age=70)  # Date object
fake.date_time()                               # datetime object
fake.date_between(start_date='-1y', end_date='today')  # Within last year
fake.date_time_between(start_date='-30d', end_date='now')  # Last 30 days

# Specific formats
fake.date_time_this_month()
fake.date_time_this_year()
fake.future_date()             # Future date
fake.past_date()               # Past date

# Time components
fake.time()                    # "14:30:45"
fake.timezone()                # "America/New_York"

Text Generation

# Words and sentences
fake.word()                    # "example"
fake.words(nb=5)               # ["word1", "word2", ...]
fake.sentence()                # "This is a sentence."
fake.sentence(nb_words=10)     # Sentence with 10 words
fake.sentences(nb=3)           # List of 3 sentences

# Paragraphs
fake.paragraph()               # Single paragraph
fake.paragraph(nb_sentences=5) # Paragraph with 5 sentences
fake.paragraphs(nb=3)          # List of 3 paragraphs
fake.text(max_nb_chars=200)    # Text up to 200 characters

# Lorem ipsum
fake.text()                    # Lorem ipsum text

Numbers and Data

# Random numbers
fake.random_int(min=0, max=100)          # Integer between 0-100
fake.random_number(digits=5)             # 5-digit number
fake.random_digit()                      # 0-9
fake.pyfloat(left_digits=3, right_digits=2)  # 123.45

# Boolean
fake.boolean()                           # True or False
fake.boolean(chance_of_getting_true=75)  # 75% chance of True

# Sequences
fake.random_element(['A', 'B', 'C'])     # Pick one
fake.random_elements(['A', 'B', 'C'], length=2, unique=True)  # Pick 2 unique

# Credit cards
fake.credit_card_number()
fake.credit_card_expire()
fake.credit_card_provider()

Localization

# Create Faker with specific locale
fake_us = Faker('en_US')
fake_uk = Faker('en_GB')
fake_fr = Faker('fr_FR')
fake_ja = Faker('ja_JP')
fake_vi = Faker('vi_VN')       # Vietnamese

# Examples
fake_us.phone_number()         # US format: +1-555-123-4567
fake_uk.phone_number()         # UK format: 01234 567890
fake_fr.name()                 # French name: Jean Dupont
fake_ja.address()              # Japanese address: 東京都...
fake_vi.name()                 # Vietnamese name: Nguyễn Văn An
fake_vi.phone_number()         # Vietnamese format: +84 912 345 678
fake_vi.address()              # Vietnamese address: 123 Nguyễn Huệ, Hà Nội

# Multiple locales
fake = Faker(['en_US', 'en_GB', 'fr_FR'])  # Random from all

Vietnamese Locale (vi_VN)

Faker supports Vietnamese data generation for creating localized test data:

from faker import Faker

fake_vi = Faker('vi_VN')

# Personal information
fake_vi.name()                 # "Nguyễn Văn An", "Trần Thị Bình"
fake_vi.first_name()           # "Văn", "Thị"
fake_vi.last_name()            # "Nguyễn", "Trần", "Lê"
fake_vi.phone_number()         # "+84 912 345 678", "0987654321"

# Addresses
fake_vi.address()              # "123 Nguyễn Huệ, Quận 1, Hồ Chí Minh"
fake_vi.city()                 # "Hà Nội", "Hồ Chí Minh", "Đà Nẵng"
fake_vi.street_address()       # "456 Lê Lợi"

# Company names (Vietnamese style)
fake_vi.company()              # "Công ty TNHH ABC"

Vietnamese-specific customizations for Elios:

from faker import Faker

fake_vi = Faker('vi_VN')

# Vietnamese universities
vietnamese_universities = [
    'Đại học Bách Khoa Hà Nội',
    'Đại học Quốc gia Hà Nội',
    'Đại học FPT',
    'Đại học Công nghệ',
    'Đại học Kinh tế Quốc dân',
    'Đại học Ngoại thương',
    'Đại học Bách Khoa TP.HCM',
    'Đại học Quốc gia TP.HCM',
    'Đại học Khoa học Tự nhiên',
    'Đại học Sư phạm Hà Nội',
]

# Vietnamese degrees
vietnamese_degrees = ['Cử nhân', 'Thạc sĩ', 'Tiến sĩ']

# Vietnamese majors
vietnamese_majors = [
    'Khoa học Máy tính',
    'Kỹ thuật Phần mềm',
    'Công nghệ Thông tin',
    'Khoa học Dữ liệu',
    'An toàn Thông tin',
    'Trí tuệ Nhân tạo',
]

# Interview statuses in Vietnamese
vietnamese_statuses = ['đang chờ', 'đã phỏng vấn', 'đã tuyển', 'đã từ chối']

# Generate Vietnamese candidate
def vietnamese_candidate_factory(fake, index):
    return {
        'full_name': fake.name(),
        'email': fake.email(),
        'phone': fake.phone_number(),
        'years_of_experience': fake.random_int(min=0, max=15),
        'skills': fake.random_elements(
            ['Python', 'JavaScript', 'SQL', 'React', 'Docker'],
            length=fake.random_int(min=2, max=5),
            unique=True
        ),
        'education': {
            'degree': fake.random_element(vietnamese_degrees),
            'major': fake.random_element(vietnamese_majors),
            'university': fake.random_element(vietnamese_universities),
            'graduation_year': fake.random_int(min=2015, max=2024),
        },
        'address': fake.address(),
        'city': fake.city(),
        'status': fake.random_element(vietnamese_statuses),
        'created_at': fake.date_time_between(start_date='-1y', end_date='now'),
    }

Custom Providers

from faker import Faker
from faker.providers import BaseProvider

class TechSkillProvider(BaseProvider):
    def programming_language(self):
        languages = ['Python', 'JavaScript', 'Java', 'C++', 'Go', 'Rust']
        return self.random_element(languages)

    def framework(self):
        frameworks = ['React', 'Vue', 'Angular', 'Django', 'FastAPI', 'Flask']
        return self.random_element(frameworks)

fake = Faker()
fake.add_provider(TechSkillProvider)

fake.programming_language()    # "Python"
fake.framework()               # "React"

Database Seeding Patterns

User Factory

def user_factory(fake, index):
    return {
        'id': index + 1,
        'username': fake.user_name(),
        'email': fake.email(),
        'first_name': fake.first_name(),
        'last_name': fake.last_name(),
        'password_hash': fake.sha256(),
        'created_at': fake.date_time_between(start_date='-2y', end_date='now'),
        'is_active': fake.boolean(chance_of_getting_true=80),
        'last_login': fake.date_time_between(start_date='-30d', end_date='now'),
    }

Product Factory

def product_factory(fake, index):
    return {
        'id': index + 1,
        'name': ' '.join(fake.words(nb=3)).title(),
        'slug': fake.slug(),
        'description': fake.paragraph(nb_sentences=5),
        'price': fake.pyfloat(left_digits=3, right_digits=2, min_value=10, max_value=1000),
        'stock': fake.random_int(min=0, max=500),
        'sku': fake.bothify(text='???-####', letters='ABCDEFGHIJKLMNOPQRSTUVWXYZ'),
        'category': fake.random_element(['Electronics', 'Clothing', 'Books', 'Home']),
        'created_at': fake.date_time_between(start_date='-1y', end_date='now'),
    }

Interview Candidate Factory (Elios-specific)

def candidate_factory(fake, index):
    skills_pool = [
        'Python', 'JavaScript', 'React', 'Node.js', 'SQL', 'MongoDB',
        'Docker', 'Kubernetes', 'AWS', 'Git', 'REST APIs', 'GraphQL',
    ]

    return {
        'id': index + 1,
        'full_name': fake.name(),
        'email': fake.email(),
        'phone': fake.phone_number(),
        'years_of_experience': fake.random_int(min=0, max=15),
        'skills': fake.random_elements(skills_pool, length=fake.random_int(min=2, max=6), unique=True),
        'education_degree': fake.random_element(['Bachelor', 'Master', 'PhD']),
        'education_major': 'Computer Science',
        'university': fake.company() + ' University',
        'graduation_year': fake.random_int(min=2010, max=2024),
        'cv_url': f"s3://bucket/cvs/candidate_{index+1}.pdf",
        'linkedin_url': f"https://linkedin.com/in/{fake.user_name()}",
        'created_at': fake.date_time_between(start_date='-6m', end_date='now'),
        'status': fake.random_element(['pending', 'interviewed', 'hired', 'rejected']),
    }

Question Factory (Elios-specific)

def question_factory(fake, index):
    categories = ['technical', 'behavioral', 'system-design', 'coding']
    difficulties = ['easy', 'medium', 'hard']

    return {
        'id': index + 1,
        'text': fake.sentence(nb_words=10) + '?',
        'category': fake.random_element(categories),
        'difficulty': fake.random_element(difficulties),
        'related_skill': fake.random_element(['Python', 'JavaScript', 'SQL', 'System Design']),
        'expected_keywords': [fake.word() for _ in range(fake.random_int(min=3, max=8))],
        'model_answer': fake.paragraph(nb_sentences=5),
        'time_limit_minutes': fake.random_element([5, 10, 15, 30]),
        'created_at': fake.date_time_between(start_date='-1y', end_date='now'),
    }

Performance Tips

Batch Generation

# Efficient: Generate all at once
fake = Faker()
names = [fake.name() for _ in range(1000)]

# Less efficient: Multiple Faker instances
names = [Faker().name() for _ in range(1000)]

Seeding for Reproducibility

from faker import Faker

# Set seed for reproducible data
Faker.seed(12345)
fake = Faker()

# Always generates same values
fake.name()  # Always "John Smith" (example)

Caching Common Values

# Cache commonly used values
fake = Faker()
user_ids = list(range(1, 101))  # 100 user IDs
cities = [fake.city() for _ in range(20)]  # 20 cities

# Reuse cached values
post = {
    'author_id': fake.random_element(user_ids),
    'city': fake.random_element(cities),
}

Common Pitfalls

1. Unique Constraint Violations

# Problem: May generate duplicate emails
users = [{'email': fake.email()} for _ in range(100)]

# Solution: Ensure uniqueness
emails = set()
users = []
while len(users) < 100:
    email = fake.email()
    if email not in emails:
        emails.add(email)
        users.append({'email': email})

2. Foreign Key References

# Problem: Random foreign keys may not exist
post = {
    'author_id': fake.random_int(min=1, max=1000),  # User may not exist
}

# Solution: Use actual existing IDs
existing_user_ids = [1, 2, 3, 4, 5]  # From created users
post = {
    'author_id': fake.random_element(existing_user_ids),
}

3. Data Type Mismatches

# Problem: String where integer expected
user = {
    'age': fake.random_int(min=18, max=80),  # ✓ Correct
    'age': str(fake.random_int(min=18, max=80)),  # ✗ Wrong if DB expects int
}

Resources

• Faker Documentation
• Available Locales
• Standard Providers

ORM Seeding Patterns

Patterns and best practices for seeding databases using various ORMs (Object-Relational Mappers).

SQLAlchemy

Basic Setup

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from faker import Faker

# Create engine and session
engine = create_engine('postgresql://user:pass@localhost/db')
Session = sessionmaker(bind=engine)
session = Session()

fake = Faker()

Simple Seeding

from domain.models import User

# Create single record
user = User(
    username=fake.user_name(),
    email=fake.email(),
    first_name=fake.first_name(),
    last_name=fake.last_name()
)
session.add(user)
session.commit()

# Create multiple records
users = []
for i in range(100):
    user = User(
        username=fake.user_name(),
        email=fake.email(),
        first_name=fake.first_name(),
        last_name=fake.last_name()
    )
    users.append(user)

session.add_all(users)
session.commit()

Batch Seeding for Performance

from domain.models import User

# Batch size for commits
BATCH_SIZE = 1000

users = []
for i in range(10000):
    user = User(
        username=fake.user_name(),
        email=fake.email()
    )
    users.append(user)

    # Commit in batches
    if (i + 1) % BATCH_SIZE == 0:
        session.add_all(users)
        session.commit()
        users = []
        print(f"Committed {i + 1} users")

# Commit remaining
if users:
    session.add_all(users)
    session.commit()

Handling Relationships

from domain.models import User, Post

# Create users first
users = []
for i in range(10):
    user = User(username=fake.user_name(), email=fake.email())
    users.append(user)

session.add_all(users)
session.commit()

# Create posts with foreign key references
posts = []
for i in range(50):
    post = Post(
        title=fake.sentence(),
        content=fake.paragraph(),
        author_id=fake.random_element([u.id for u in users])
    )
    posts.append(post)

session.add_all(posts)
session.commit()

Many-to-Many Relationships

from domain.models import User, Skill

# Create skills
skills = [
    Skill(name='Python'),
    Skill(name='JavaScript'),
    Skill(name='SQL'),
]
session.add_all(skills)
session.commit()

# Create users with skills (many-to-many)
for i in range(20):
    user = User(
        username=fake.user_name(),
        email=fake.email()
    )

    # Add random skills to user
    user_skills = fake.random_elements(
        skills,
        length=fake.random_int(min=1, max=len(skills)),
        unique=True
    )
    user.skills.extend(user_skills)

    session.add(user)

session.commit()

Error Handling

from sqlalchemy.exc import IntegrityError

for i in range(100):
    try:
        user = User(
            username=fake.user_name(),
            email=fake.email()
        )
        session.add(user)
        session.commit()
    except IntegrityError as e:
        # Handle duplicate key violations
        session.rollback()
        print(f"Error creating user {i}: {e}")

Using Factories Pattern

from faker import Faker
from domain.models import User

class UserFactory:
    def __init__(self, session, faker=None):
        self.session = session
        self.fake = faker or Faker()

    def create(self, **kwargs):
        """Create single user"""
        data = {
            'username': self.fake.user_name(),
            'email': self.fake.email(),
            'first_name': self.fake.first_name(),
            'last_name': self.fake.last_name(),
        }
        data.update(kwargs)

        user = User(**data)
        self.session.add(user)
        self.session.commit()
        return user

    def create_batch(self, count, **kwargs):
        """Create multiple users"""
        users = []
        for _ in range(count):
            data = {
                'username': self.fake.user_name(),
                'email': self.fake.email(),
                'first_name': self.fake.first_name(),
                'last_name': self.fake.last_name(),
            }
            data.update(kwargs)
            users.append(User(**data))

        self.session.add_all(users)
        self.session.commit()
        return users

# Usage
factory = UserFactory(session)
user = factory.create(username='john_doe')
users = factory.create_batch(100, is_active=True)

Django ORM

Basic Setup

import os
import django

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
django.setup()

from myapp.models import User
from faker import Faker

fake = Faker()

Simple Seeding

from myapp.models import User

# Create single record
user = User.objects.create(
    username=fake.user_name(),
    email=fake.email(),
    first_name=fake.first_name(),
    last_name=fake.last_name()
)

# Create multiple records
users = [
    User(
        username=fake.user_name(),
        email=fake.email(),
        first_name=fake.first_name(),
        last_name=fake.last_name()
    )
    for _ in range(100)
]
User.objects.bulk_create(users)

Bulk Operations

from myapp.models import User

# Bulk create (efficient for many records)
users = [User(username=fake.user_name(), email=fake.email()) for _ in range(1000)]
User.objects.bulk_create(users, batch_size=500)

# Get created objects with IDs
users = User.objects.bulk_create(users, batch_size=500)
# Note: In Django 4.0+, bulk_create returns objects with IDs

Relationships

from myapp.models import User, Post

# Create user
user = User.objects.create(username=fake.user_name(), email=fake.email())

# Create posts for user
posts = [
    Post(
        title=fake.sentence(),
        content=fake.paragraph(),
        author=user
    )
    for _ in range(10)
]
Post.objects.bulk_create(posts)

Management Command for Seeding

# myapp/management/commands/seed_database.py

from django.core.management.base import BaseCommand
from faker import Faker
from myapp.models import User

class Command(BaseCommand):
    help = 'Seed database with fake data'

    def add_arguments(self, parser):
        parser.add_argument('--users', type=int, default=10)

    def handle(self, *args, **options):
        fake = Faker()
        count = options['users']

        self.stdout.write(f'Creating {count} users...')

        users = [
            User(
                username=fake.user_name(),
                email=fake.email()
            )
            for _ in range(count)
        ]
        User.objects.bulk_create(users)

        self.stdout.write(self.style.SUCCESS(f'✓ Created {count} users'))

# Usage: python manage.py seed_database --users 100

Prisma (TypeScript/JavaScript)

Basic Setup

import { PrismaClient } from '@prisma/client';
import { faker } from '@faker-js/faker';

const prisma = new PrismaClient();

Simple Seeding

// Single record
const user = await prisma.user.create({
  data: {
    username: faker.internet.userName(),
    email: faker.internet.email(),
    firstName: faker.person.firstName(),
    lastName: faker.person.lastName(),
  },
});

// Multiple records
const users = await Promise.all(
  Array.from({ length: 100 }, () =>
    prisma.user.create({
      data: {
        username: faker.internet.userName(),
        email: faker.internet.email(),
        firstName: faker.person.firstName(),
        lastName: faker.person.lastName(),
      },
    })
  )
);

Batch Operations

// Create many (more efficient)
const users = await prisma.user.createMany({
  data: Array.from({ length: 100 }, () => ({
    username: faker.internet.userName(),
    email: faker.internet.email(),
    firstName: faker.person.firstName(),
    lastName: faker.person.lastName(),
  })),
});

Seeding Script

// prisma/seed.ts

import { PrismaClient } from '@prisma/client';
import { faker } from '@faker-js/faker';

const prisma = new PrismaClient();

async function main() {
  console.log('Seeding database...');

  // Create users
  const users = await prisma.user.createMany({
    data: Array.from({ length: 10 }, () => ({
      username: faker.internet.userName(),
      email: faker.internet.email(),
    })),
  });

  console.log(`✓ Created ${users.count} users`);
}

main()
  .catch((e) => {
    console.error(e);
    process.exit(1);
  })
  .finally(async () => {
    await prisma.$disconnect();
  });

// Add to package.json:
// "prisma": {
//   "seed": "ts-node prisma/seed.ts"
// }

MongoDB (PyMongo)

Basic Setup

from pymongo import MongoClient
from faker import Faker

client = MongoClient('mongodb://localhost:27017/')
db = client['mydb']
fake = Faker()

Simple Seeding

# Single document
user = {
    'username': fake.user_name(),
    'email': fake.email(),
    'created_at': fake.date_time()
}
result = db.users.insert_one(user)
print(f"Created user with ID: {result.inserted_id}")

# Multiple documents
users = [
    {
        'username': fake.user_name(),
        'email': fake.email(),
        'created_at': fake.date_time()
    }
    for _ in range(100)
]
result = db.users.insert_many(users)
print(f"Created {len(result.inserted_ids)} users")

With Relationships (Embedded Documents)

# User with embedded posts
user = {
    'username': fake.user_name(),
    'email': fake.email(),
    'posts': [
        {
            'title': fake.sentence(),
            'content': fake.paragraph(),
            'created_at': fake.date_time()
        }
        for _ in range(5)
    ]
}
db.users.insert_one(user)

With References

# Create users first
users = [
    {'username': fake.user_name(), 'email': fake.email()}
    for _ in range(10)
]
user_ids = db.users.insert_many(users).inserted_ids

# Create posts with user references
posts = [
    {
        'title': fake.sentence(),
        'content': fake.paragraph(),
        'author_id': fake.random_element(user_ids)
    }
    for _ in range(50)
]
db.posts.insert_many(posts)

Best Practices

1. Transaction Safety

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

engine = create_engine('postgresql://user:pass@localhost/db')
Session = sessionmaker(bind=engine)

def seed_database():
    session = Session()
    try:
        # Seeding operations
        users = [User(username=fake.user_name()) for _ in range(100)]
        session.add_all(users)
        session.commit()
        print("✓ Seeding successful")
    except Exception as e:
        session.rollback()
        print(f"✗ Seeding failed: {e}")
        raise
    finally:
        session.close()

2. Idempotent Seeding

def seed_users(session, count=100):
    # Check if already seeded
    existing_count = session.query(User).count()
    if existing_count >= count:
        print(f"Database already has {existing_count} users, skipping...")
        return

    # Seed remaining
    remaining = count - existing_count
    users = [User(username=fake.user_name()) for _ in range(remaining)]
    session.add_all(users)
    session.commit()
    print(f"✓ Created {remaining} users ({count} total)")

3. Clearing Before Seeding

def clear_and_seed(session):
    # Clear existing data
    session.query(Post).delete()
    session.query(User).delete()
    session.commit()

    # Seed fresh data
    users = [User(username=fake.user_name()) for _ in range(100)]
    session.add_all(users)
    session.commit()

4. Seeding with Constraints

def seed_unique_users(session, count=100):
    """Ensure unique usernames/emails"""
    existing_emails = {u.email for u in session.query(User.email).all()}

    users = []
    attempts = 0
    max_attempts = count * 10

    while len(users) < count and attempts < max_attempts:
        email = fake.email()
        if email not in existing_emails:
            user = User(username=fake.user_name(), email=email)
            users.append(user)
            existing_emails.add(email)
        attempts += 1

    session.add_all(users)
    session.commit()
    return len(users)

5. Progress Reporting

def seed_with_progress(session, count=10000):
    """Seed with progress reporting"""
    batch_size = 1000

    for i in range(0, count, batch_size):
        batch = [
            User(username=fake.user_name(), email=fake.email())
            for _ in range(min(batch_size, count - i))
        ]
        session.add_all(batch)
        session.commit()

        completed = min(i + batch_size, count)
        progress = (completed / count) * 100
        print(f"Progress: {completed}/{count} ({progress:.1f}%)")

Resources

• SQLAlchemy Documentation
• Django ORM Documentation
• Prisma Documentation
• PyMongo Documentation