Using Document Databases

Document Database Implementation

Guide NoSQL document database selection and implementation for flexible schema applications across Python, TypeScript, Rust, and Go.

When to Use This Skill

Use document databases when applications need:

• Flexible schemas - Data models evolve rapidly without migrations
• Nested structures - JSON-like hierarchical data
• Horizontal scaling - Built-in sharding and replication
• Developer velocity - Object-to-database mapping without ORM complexity

Database Selection

Quick Decision Framework

DEPLOYMENT ENVIRONMENT?
├── AWS-Native Application → DynamoDB
│   ✓ Serverless, auto-scaling, single-digit ms latency
│   ✗ Limited query flexibility
│
├── Firebase/GCP Ecosystem → Firestore
│   ✓ Real-time sync, offline support, mobile-first
│   ✗ More expensive for heavy reads
│
└── General-Purpose/Complex Queries → MongoDB
    ✓ Rich aggregation, full-text search, vector search
    ✓ ACID transactions, self-hosted or managed

Database Comparison

See references/mongodb.md for MongoDB details See references/dynamodb.md for DynamoDB single-table design See references/firestore.md for Firestore real-time patterns

Schema Design Patterns

Embedding vs Referencing

Use the decision matrix in `references/schema-design-patterns.md`

Quick guide:

Embedding Example (MongoDB)

// User with embedded addresses
{
  _id: ObjectId("..."),
  email: "user@example.com",
  name: "Jane Doe",
  addresses: [
    {
      type: "home",
      street: "123 Main St",
      city: "Boston",
      default: true
    }
  ],
  preferences: {
    theme: "dark",
    notifications: { email: true, sms: false }
  }
}

Referencing Example (E-commerce)

// Orders reference products
{
  _id: ObjectId("..."),
  userId: ObjectId("..."),
  items: [
    {
      productId: ObjectId("..."),      // Reference
      priceAtPurchase: 49.99,          // Denormalize (historical)
      quantity: 2
    }
  ],
  totalAmount: 99.98
}

When to denormalize:

• Frequently read together
• Historical snapshots (prices, names)
• Read-heavy workloads

Indexing Strategies

MongoDB Index Types

// 1. Single field (unique email)
db.users.createIndex({ email: 1 }, { unique: true })

// 2. Compound index (ORDER MATTERS!)
db.orders.createIndex({ status: 1, createdAt: -1 })

// 3. Partial index (index subset)
db.orders.createIndex(
  { userId: 1 },
  { partialFilterExpression: { status: { $eq: "pending" }}}
)

// 4. TTL index (auto-delete after 30 days)
db.sessions.createIndex(
  { createdAt: 1 },
  { expireAfterSeconds: 2592000 }
)

// 5. Text index (full-text search)
db.articles.createIndex({
  title: "text",
  content: "text"
})

Index Best Practices:

• Add indexes for all query filters
• Compound index order: Equality → Range → Sort
• Use covering indexes (query + projection in index)
• Use explain() to verify index usage
• Monitor with Performance Advisor (Atlas)

Validate indexes with the script:

python scripts/validate_indexes.py

See references/indexing-strategies.md for complete guide.

MongoDB Aggregation Pipelines

Key Operators: $match (filter), $group (aggregate), $lookup (join), $unwind (arrays), $project (reshape)

For complete pipeline patterns and examples, see: references/aggregation-patterns.md

DynamoDB Single-Table Design

Design for access patterns using PK/SK patterns. Store multiple entity types in one table with composite keys.

For complete single-table design patterns and GSI strategies, see: references/dynamodb.md

Firestore Real-Time Patterns

Use onSnapshot() for real-time listeners and Firestore security rules for access control.

For complete real-time patterns and security rules, see: references/firestore.md

Multi-Language Examples

Complete implementations available in `examples/` directory:

• examples/mongodb-fastapi/ - Python FastAPI + MongoDB
• examples/mongodb-nextjs/ - TypeScript Next.js + MongoDB
• examples/dynamodb-serverless/ - Python Lambda + DynamoDB
• examples/firestore-react/ - React + Firestore real-time

Frontend Skill Integration

• Media Skill - Use MongoDB GridFS for large file storage with metadata
• AI Chat Skill - MongoDB Atlas Vector Search for semantic conversation retrieval
• Feedback Skill - DynamoDB for high-throughput event logging with TTL

For integration examples, see: references/skill-integrations.md

Performance Optimization

Key practices:

• Always use indexes for query filters (verify with .explain())
• Use connection pooling (reuse clients across requests)
• Avoid collection scans in production

For complete optimization guide, see: references/performance.md

Common Patterns

Pagination: Use cursor-based pagination for large datasets (recommended over offset) Soft Deletes: Mark as deleted with timestamp instead of removing Audit Logs: Store version history within documents

For implementation details, see: references/common-patterns.md

Validation and Scripts

Validate Index Coverage

# Run validation script
python scripts/validate_indexes.py --db myapp --collection orders

# Output:
# ✓ Query { status: "pending" } covered by index status_1
# ✗ Query { userId: "..." } missing index - add: { userId: 1 }

Schema Analysis

# Analyze schema patterns
python scripts/analyze_schema.py --db myapp

# Output:
# Collection: users
# - Average document size: 2.4 KB
# - Embedding ratio: 87% (addresses, preferences)
# - Reference ratio: 13% (orderIds)
# Recommendation: Good balance

Anti-Patterns to Avoid

Unbounded Arrays: Limit embedded arrays (use references for large collections) Over-Indexing: Only index queried fields (indexes slow writes) DynamoDB Scans: Always use Query with partition key (avoid Scan)

For detailed anti-patterns, see: references/anti-patterns.md

Dependencies

Python

# MongoDB
pip install motor pymongo

# DynamoDB
pip install boto3

# Firestore
pip install firebase-admin

TypeScript

# MongoDB
npm install mongodb

# DynamoDB
npm install @aws-sdk/client-dynamodb @aws-sdk/util-dynamodb

# Firestore
npm install firebase firebase-admin

Rust

# MongoDB
mongodb = "2.8"

# DynamoDB
aws-sdk-dynamodb = "1.0"

Go

# MongoDB
go get go.mongodb.org/mongo-driver

# DynamoDB
go get github.com/aws/aws-sdk-go-v2/service/dynamodb

Additional Resources

Database-Specific Guides:

• references/mongodb.md - Complete MongoDB documentation
• references/dynamodb.md - DynamoDB single-table patterns
• references/firestore.md - Firestore real-time guide

Pattern Guides:

• references/schema-design-patterns.md - Embedding vs referencing decisions
• references/indexing-strategies.md - Index optimization
• references/aggregation-patterns.md - MongoDB pipeline cookbook
• references/common-patterns.md - Pagination, soft deletes, audit logs
• references/anti-patterns.md - Mistakes to avoid
• references/performance.md - Query optimization
• references/skill-integrations.md - Frontend skill integration

Examples: examples/mongodb-fastapi/, examples/mongodb-nextjs/, examples/dynamodb-serverless/, examples/firestore-react/

Firestore + React Real-Time Example

React application with Firestore real-time listeners, offline support, and security rules.

Stack

• React 18
• Firebase/Firestore
• Real-time listeners (onSnapshot)
• Offline persistence
• Firestore Security Rules
• TypeScript

Features

• Real-time data synchronization
• Offline support with local cache
• Optimistic updates
• Security rules enforcement
• Subcollection queries

Project Structure

firestore-react/
├── src/
│   ├── firebase.ts          # Firebase configuration
│   ├── hooks/
│   │   ├── useCollection.ts # Real-time collection hook
│   │   └── useDocument.ts   # Real-time document hook
│   ├── components/
│   │   ├── PostList.tsx
│   │   └── CreatePost.tsx
│   └── App.tsx
├── firestore.rules          # Security rules
└── package.json

Quick Start

# Install
npm install firebase

# Configure Firebase (create project at console.firebase.google.com)
# Add config to .env

# Run
npm run dev

Firebase Setup

// src/firebase.ts
import { initializeApp } from 'firebase/app';
import { getFirestore, enableIndexedDbPersistence } from 'firebase/firestore';

const firebaseConfig = {
  apiKey: process.env.REACT_APP_FIREBASE_API_KEY,
  authDomain: "myapp.firebaseapp.com",
  projectId: "myapp",
  storageBucket: "myapp.appspot.com",
};

const app = initializeApp(firebaseConfig);
export const db = getFirestore(app);

// Enable offline persistence
enableIndexedDbPersistence(db).catch((err) => {
  if (err.code === 'failed-precondition') {
    console.warn('Multiple tabs open, persistence can only be enabled in one tab');
  }
});

Real-Time Hooks

// hooks/useCollection.ts
import { useEffect, useState } from 'react';
import { collection, query, onSnapshot, QueryConstraint } from 'firebase/firestore';
import { db } from '../firebase';

export function useCollection<T>(
  collectionName: string,
  ...queryConstraints: QueryConstraint[]
) {
  const [data, setData] = useState<T[]>([]);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState<Error | null>(null);

  useEffect(() => {
    const q = query(collection(db, collectionName), ...queryConstraints);

    const unsubscribe = onSnapshot(
      q,
      (snapshot) => {
        const items = snapshot.docs.map((doc) => ({
          id: doc.id,
          ...doc.data(),
        })) as T[];

        setData(items);
        setLoading(false);
      },
      (err) => {
        setError(err);
        setLoading(false);
      }
    );

    return () => unsubscribe();
  }, [collectionName]);

  return { data, loading, error };
}

Real-Time Component

// components/PostList.tsx
import { useCollection } from '../hooks/useCollection';
import { orderBy, limit } from 'firebase/firestore';

interface Post {
  id: string;
  title: string;
  content: string;
  createdAt: Date;
}

export function PostList() {
  const { data: posts, loading } = useCollection<Post>(
    'posts',
    orderBy('createdAt', 'desc'),
    limit(20)
  );

  if (loading) return <div>Loading...</div>;

  return (
    <div>
      {posts.map((post) => (
        <div key={post.id}>
          <h2>{post.title}</h2>
          <p>{post.content}</p>
        </div>
      ))}
    </div>
  );
}

CRUD Operations

import {
  collection,
  addDoc,
  updateDoc,
  deleteDoc,
  doc,
  serverTimestamp,
} from 'firebase/firestore';

// Create
const createPost = async (title: string, content: string) => {
  await addDoc(collection(db, 'posts'), {
    title,
    content,
    createdAt: serverTimestamp(),
    userId: currentUser.uid,
  });
};

// Update
const updatePost = async (postId: string, updates: Partial<Post>) => {
  await updateDoc(doc(db, 'posts', postId), {
    ...updates,
    updatedAt: serverTimestamp(),
  });
};

// Delete
const deletePost = async (postId: string) => {
  await deleteDoc(doc(db, 'posts', postId));
};

Security Rules

// firestore.rules
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    // Posts: Read public, write authenticated
    match /posts/{postId} {
      allow read: if true;
      allow create: if request.auth != null
                    && request.resource.data.userId == request.auth.uid;
      allow update, delete: if request.auth != null
                             && resource.data.userId == request.auth.uid;
    }

    // Comments: Nested under posts
    match /posts/{postId}/comments/{commentId} {
      allow read: if true;
      allow create: if request.auth != null;
      allow update, delete: if request.auth.uid == resource.data.userId;
    }

    // User profiles: Read public, write owner only
    match /users/{userId} {
      allow read: if true;
      allow write: if request.auth.uid == userId;
    }
  }
}

Optimistic Updates

import { doc, updateDoc, writeBatch } from 'firebase/firestore';

function PostActions({ postId }: { postId: string }) {
  const [likes, setLikes] = useState(0);

  const handleLike = async () => {
    // Optimistic update
    setLikes((prev) => prev + 1);

    try {
      await updateDoc(doc(db, 'posts', postId), {
        likes: increment(1),
      });
    } catch (error) {
      // Rollback on error
      setLikes((prev) => prev - 1);
    }
  };

  return <button onClick={handleLike}>❤️ {likes}</button>;
}

Batch Operations

import { writeBatch, doc } from 'firebase/firestore';

const batch = writeBatch(db);

// Add operations to batch
batch.set(doc(db, 'users', 'user1'), { name: 'Alice' });
batch.update(doc(db, 'posts', 'post1'), { views: increment(1) });
batch.delete(doc(db, 'temp', 'temp1'));

// Commit atomically
await batch.commit();

Real-Time Presence

import { onDisconnect, ref, set } from 'firebase/database';
import { getDatabase } from 'firebase/database';

const rtdb = getDatabase();
const userStatusRef = ref(rtdb, `/status/${currentUser.uid}`);

// Set online
await set(userStatusRef, {
  state: 'online',
  lastSeen: serverTimestamp(),
});

// Set offline on disconnect
onDisconnect(userStatusRef).set({
  state: 'offline',
  lastSeen: serverTimestamp(),
});

Integration Summary

• Real-time updates - onSnapshot for live data
• Offline support - IndexedDB persistence
• Security - Firestore rules enforce access control
• Optimistic UI - Update UI before server confirms
• Batch writes - Atomic multi-document operations

Best Practices

1. Enable offline persistence - Better UX on poor connections 2. Security rules - Never trust client, validate server-side 3. Optimize queries - Create indexes for filtered/sorted fields 4. Limit listeners - Unsubscribe when component unmounts 5. Handle errors - Network failures, permission denied 6. Batch writes - Atomic multi-operation updates 7. Denormalize - Duplicate data for read performance

Resources

• Firestore Docs: https://firebase.google.com/docs/firestore
• React Fire: https://github.com/FirebaseExtended/reactfire
• Security Rules: https://firebase.google.com/docs/firestore/security/get-started

MongoDB + Next.js Example

Full-stack Next.js 14 application with MongoDB, demonstrating CRUD operations, server components, and real-time updates.

Stack

• Next.js 14 (App Router)
• MongoDB (with Motor/PyMongo or native Node driver)
• Server Components + API Routes
• TypeScript
• Tailwind CSS

Features

• CRUD operations (Create, Read, Update, Delete)
• Server-side data fetching
• API routes for database operations
• Optimistic updates
• Form validation
• Error handling

Project Structure

mongodb-nextjs/
├── app/
│   ├── page.tsx             # Home page (Server Component)
│   ├── posts/
│   │   ├── page.tsx         # Posts list
│   │   ├── [id]/page.tsx    # Post detail
│   │   └── new/page.tsx     # Create post
│   └── api/
│       └── posts/
│           ├── route.ts     # GET /api/posts, POST /api/posts
│           └── [id]/
│               └── route.ts # GET/PUT/DELETE /api/posts/:id
├── lib/
│   └── mongodb.ts           # MongoDB client singleton
├── models/
│   └── post.ts              # TypeScript types
└── package.json

Quick Start

# Install
npm install mongodb

# Configure
cp .env.example .env
# Set MONGODB_URI=mongodb://localhost:27017/myapp

# Run
npm run dev

MongoDB Connection

// lib/mongodb.ts
import { MongoClient } from 'mongodb';

if (!process.env.MONGODB_URI) {
  throw new Error('Please add MONGODB_URI to .env');
}

const uri = process.env.MONGODB_URI;
const options = {};

let client: MongoClient;
let clientPromise: Promise<MongoClient>;

if (process.env.NODE_ENV === 'development') {
  // Preserve client across hot reloads
  if (!(global as any)._mongoClientPromise) {
    client = new MongoClient(uri, options);
    (global as any)._mongoClientPromise = client.connect();
  }
  clientPromise = (global as any)._mongoClientPromise;
} else {
  client = new MongoClient(uri, options);
  clientPromise = client.connect();
}

export default clientPromise;

Server Component (Read)

// app/posts/page.tsx
import clientPromise from '@/lib/mongodb';

export default async function PostsPage() {
  const client = await clientPromise;
  const db = client.db('myapp');

  const posts = await db.collection('posts')
    .find({})
    .sort({ createdAt: -1 })
    .limit(20)
    .toArray();

  return (
    <div>
      <h1>Posts</h1>
      {posts.map((post) => (
        <PostCard key={post._id.toString()} post={post} />
      ))}
    </div>
  );
}

API Route (Create)

// app/api/posts/route.ts
import { NextRequest, NextResponse } from 'next/server';
import clientPromise from '@/lib/mongodb';

export async function POST(request: NextRequest) {
  const { title, content } = await request.json();

  const client = await clientPromise;
  const db = client.db('myapp');

  const result = await db.collection('posts').insertOne({
    title,
    content,
    createdAt: new Date(),
    updatedAt: new Date(),
  });

  return NextResponse.json({
    id: result.insertedId.toString(),
    title,
    content,
  }, { status: 201 });
}

export async function GET() {
  const client = await clientPromise;
  const db = client.db('myapp');

  const posts = await db.collection('posts')
    .find({})
    .sort({ createdAt: -1 })
    .toArray();

  return NextResponse.json({ posts });
}

Client Component (Optimistic Update)

'use client';

import { useMutation, useQueryClient } from '@tanstack/react-query';

function CreatePostForm() {
  const queryClient = useQueryClient();

  const mutation = useMutation({
    mutationFn: (newPost) => fetch('/api/posts', {
      method: 'POST',
      body: JSON.stringify(newPost),
    }).then(r => r.json()),

    onMutate: async (newPost) => {
      // Optimistic update
      await queryClient.cancelQueries({ queryKey: ['posts'] });

      const previous = queryClient.getQueryData(['posts']);

      queryClient.setQueryData(['posts'], (old: any) => [
        ...old,
        { ...newPost, _id: 'temp-id' },
      ]);

      return { previous };
    },

    onError: (err, newPost, context) => {
      // Rollback on error
      queryClient.setQueryData(['posts'], context?.previous);
    },

    onSuccess: () => {
      // Refetch to get server data
      queryClient.invalidateQueries({ queryKey: ['posts'] });
    },
  });

  return <form onSubmit={(e) => {
    e.preventDefault();
    mutation.mutate({ title: '...', content: '...' });
  }} />;
}

Integration Summary

Best Practices

1. Singleton connection - Reuse MongoDB client 2. Index query fields - All find() filters should have indexes 3. Cursor pagination - Not offset for large collections 4. Optimistic updates - Better UX for mutations 5. Error boundaries - Graceful error handling 6. TypeScript types - Type safety for documents 7. Validation - Zod/Pydantic on API layer 8. Connection pooling - Default is usually sufficient

Resources

• Next.js + MongoDB: https://github.com/vercel/next.js/tree/canary/examples/with-mongodb
• MongoDB Node Driver: https://www.mongodb.com/docs/drivers/node/

Document Database Implementation Skill

Production-ready Claude Skill for NoSQL document database selection and implementation.

Overview

This skill guides document database selection and implementation for flexible schema applications across Python, TypeScript, Rust, and Go.

Primary databases covered:

• MongoDB (general-purpose, rich queries, vector search)
• DynamoDB (AWS serverless, single-table design)
• Firestore (real-time sync, mobile-first)

Skill Structure

using-document-databases/
├── SKILL.md                          # Main skill file (<500 lines)
├── references/
│   ├── mongodb.md                    # MongoDB collections, indexes, aggregation
│   ├── dynamodb.md                   # DynamoDB single-table, GSI patterns
│   ├── firestore.md                  # Firestore real-time, security rules
│   └── schema-design-patterns.md     # Embedding vs referencing framework
├── examples/
│   ├── mongodb-fastapi/              # Python FastAPI + MongoDB (Motor)
│   │   ├── main.py
│   │   └── requirements.txt
│   └── dynamodb-serverless/          # Python Lambda + DynamoDB
│       ├── handler.py
│       └── serverless.yml
└── scripts/
    └── validate_indexes.py           # MongoDB index validation tool

Quick Start

Using the Skill

The skill automatically triggers when building applications with:

• Content management systems
• User profiles with flexible attributes
• Product catalogs
• Event logging systems
• Mobile apps requiring offline sync

Database Selection

Use MongoDB when:

• Complex aggregation queries needed
• Full-text or vector search required
• ACID multi-document transactions needed
• Self-hosted or multi-cloud deployment

Use DynamoDB when:

• AWS-native serverless architecture
• Predictable single-digit ms latency required
• Auto-scaling without capacity planning
• Event-driven workflows (Streams + Lambda)

Use Firestore when:

• Real-time sync across clients required
• Mobile-first with offline support
• Firebase ecosystem (Auth, Hosting, Analytics)
• Rapid prototyping with generous free tier

Key Features

Schema Design Patterns

Decision matrix for embedding vs referencing:

• One-to-Few (<10) → Embed
• One-to-Many (10-1000) → Hybrid
• One-to-Millions → Reference
• Many-to-Many → Reference

See references/schema-design-patterns.md

Indexing Strategies

MongoDB index types:

• Single field, compound, multikey
• Text (full-text search)
• Geospatial (2dsphere)
• TTL (auto-expiring documents)
• Wildcard (dynamic schemas)

Validate indexes:

python scripts/validate_indexes.py --db myapp --collection orders

Aggregation Pipelines

MongoDB's killer feature for complex transformations:

• $match, $project, $group, $lookup
• $unwind, $sort, $limit, $facet

See references/mongodb.md for aggregation cookbook.

DynamoDB Single-Table Design

Access pattern-driven modeling:

PK: USER#12345,  SK: METADATA       # User data
PK: USER#12345,  SK: ORDER#001      # User's orders
PK: ORDER#001,   SK: METADATA       # Order details
PK: ORDER#001,   SK: ITEM#001       # Order items

See references/dynamodb.md for complete patterns.

Examples

MongoDB + FastAPI (Python)

Production-ready async API with Motor:

cd examples/mongodb-fastapi
pip install -r requirements.txt
export MONGODB_URI="mongodb://localhost:27017/"
python main.py

Features:

• Async MongoDB with connection pooling
• CRUD operations with validation
• Aggregation pipeline analytics
• Soft deletes
• Health checks

DynamoDB + Lambda (Serverless)

AWS serverless API with single-table design:

cd examples/dynamodb-serverless
npm install -g serverless
serverless deploy

Features:

• Single-table design pattern
• Batch writes for efficiency
• GSI for status queries
• Lambda + API Gateway
• Auto-scaling with pay-per-request

Multi-Language Support

Python:

• pymongo (sync)
• motor (async with AsyncIO/FastAPI)
• boto3 (DynamoDB)

TypeScript:

• mongodb (native driver)
• @aws-sdk/client-dynamodb (DynamoDB SDK v3)
• firebase/firestore (Firestore)

Rust:

• mongodb crate
• aws-sdk-dynamodb

Go:

• mongo-go-driver
• aws-sdk-go-v2

Integration with Other Skills

media/ - File metadata storage (MongoDB GridFS) ai-chat/ - Conversation history + vector search (Atlas Vector Search) feedback/ - Event logging (DynamoDB high-throughput writes) forms/ - Dynamic form submissions (Firestore real-time validation) search-filter/ - Product catalogs (MongoDB Atlas Search)

Performance Best Practices

MongoDB

• Use indexes for all query filters
• Covering indexes (query + projection in index)
• Connection pooling (reuse client)
• Projection (fetch only needed fields)

DynamoDB

• Design for even partition distribution
• Batch operations (up to 100 items)
• GSI projections (KEYS_ONLY or INCLUDE)
• TTL for auto-expiring data

Firestore

• Denormalize frequently accessed data
• Use subcollections for large arrays
• Offline persistence for mobile
• Security rules for access control

Common Patterns

Pagination (MongoDB):

// Cursor-based (recommended)
db.products.find({ _id: { $gt: lastId }}).limit(20)

Soft Deletes:

// Mark as deleted instead of removing
{ deleted: true, deletedAt: ISODate("...") }

Audit Logs:

// Versioned documents
{ documentId: "doc123", version: 3, history: [...] }

Dependencies

Python:

pip install motor pymongo boto3 firebase-admin

TypeScript:

npm install mongodb @aws-sdk/client-dynamodb firebase

Anti-Patterns to Avoid

❌ Unbounded arrays - Use references instead ❌ Deep nesting - Flatten with references ❌ Over-indexing - Index only queried fields ❌ DynamoDB scans - Always use query with partition key ❌ Missing indexes - Validate with explain()

Testing

Run index validation:

python scripts/validate_indexes.py --db myapp --all

Expected output:

• ✓ Covered queries
• ✗ Missing indexes with suggestions
• 📈 Index usage statistics
• ⚠️ Unused indexes

Additional Resources

• MongoDB documentation: references/mongodb.md
• DynamoDB patterns: references/dynamodb.md
• Firestore guide: references/firestore.md
• Schema design: references/schema-design-patterns.md

Version

v0.1.0 - Initial release (December 2025)

---

Skill Author: Claude Code Skill Type: Database Implementation Languages: Python, TypeScript, Rust, Go Databases: MongoDB, DynamoDB, Firestore

MongoDB Aggregation Pipeline Patterns

Complete guide to MongoDB aggregation framework for complex queries, analytics, and data transformations.

Table of Contents

• Aggregation Pipeline Basics
• Core Stages
• $match (Filter)
• $group (Aggregate)
• $project (Reshape)
• $lookup (Join)
• $unwind (Flatten Arrays)
• Common Patterns
• Top N per Category
• Time-Based Aggregation
• Moving Average
• Pagination
• Advanced Patterns
• Faceted Search
• Full-Text Search + Aggregation
• Conditional Aggregation
• Performance Optimization
• Index Usage
• $match and $project Early
• Limit Result Size
• Pipeline Validation
• Best Practices
• Resources

Aggregation Pipeline Basics

Pipeline stages process documents sequentially:

db.collection.aggregate([
  { $match: { status: "active" } },      // Filter
  { $group: { _id: "$category" } },      // Group
  { $sort: { count: -1 } },              // Sort
  { $limit: 10 },                        // Limit
])

Core Stages

$match (Filter)

// Filter before expensive operations
db.orders.aggregate([
  { $match: {
      status: "completed",
      createdAt: { $gte: ISODate("2025-01-01") }
  }},
  // ... other stages
])

Best practice: Use $match early to reduce document count.

$group (Aggregate)

// Revenue by category
db.orders.aggregate([
  { $group: {
      _id: "$category",
      totalRevenue: { $sum: "$amount" },
      avgOrder: { $avg: "$amount" },
      count: { $sum: 1 },
      maxOrder: { $max: "$amount" },
      minOrder: { $min: "$amount" },
  }},
])

Accumulators: $sum, $avg, $max, $min, $first, $last, $push, $addToSet

$project (Reshape)

// Select and transform fields
db.users.aggregate([
  { $project: {
      fullName: { $concat: ["$firstName", " ", "$lastName"] },
      email: 1,  // Include field
      _id: 0,    // Exclude field
      ageGroup: {
        $cond: {
          if: { $gte: ["$age", 18] },
          then: "adult",
          else: "minor"
        }
      }
  }},
])

$lookup (Join)

// Join orders with users
db.orders.aggregate([
  { $lookup: {
      from: "users",
      localField: "userId",
      foreignField: "_id",
      as: "user"
  }},
  { $unwind: "$user" },  // Flatten array
])

$unwind (Flatten Arrays)

// Expand array elements into separate documents
db.posts.aggregate([
  { $unwind: "$tags" },  // Create one doc per tag
  { $group: {
      _id: "$tags",
      count: { $sum: 1 }
  }},
])

Common Patterns

Top N per Category

// Top 3 products per category
db.products.aggregate([
  { $sort: { category: 1, sales: -1 } },
  { $group: {
      _id: "$category",
      products: { $push: "$$ROOT" },
  }},
  { $project: {
      category: "$_id",
      topProducts: { $slice: ["$products", 3] }
  }},
])

Time-Based Aggregation

// Daily revenue
db.orders.aggregate([
  { $match: {
      createdAt: { $gte: ISODate("2025-11-01") }
  }},
  { $group: {
      _id: {
        $dateToString: { format: "%Y-%m-%d", date: "$createdAt" }
      },
      revenue: { $sum: "$amount" },
      orders: { $sum: 1 },
  }},
  { $sort: { _id: 1 } },
])

Moving Average

// 7-day moving average
db.metrics.aggregate([
  { $setWindowFields: {
      sortBy: { date: 1 },
      output: {
        movingAvg: {
          $avg: "$value",
          window: { documents: [-6, 0] }  // Current + 6 previous
        }
      }
  }},
])

Pagination

// Efficient pagination
db.products.aggregate([
  { $match: { category: "electronics" } },
  { $sort: { createdAt: -1 } },
  { $skip: 20 },   // Page 2 (skip 20)
  { $limit: 10 },  // 10 per page
])

// Total count for pagination
db.products.aggregate([
  { $match: { category: "electronics" } },
  { $facet: {
      items: [
        { $skip: 20 },
        { $limit: 10 },
      ],
      totalCount: [
        { $count: "count" },
      ],
  }},
])

Advanced Patterns

Faceted Search

// Multiple aggregations in one query
db.products.aggregate([
  { $match: { price: { $lte: 1000 } } },
  { $facet: {
      byCategory: [
        { $group: { _id: "$category", count: { $sum: 1 } } },
        { $sort: { count: -1 } },
      ],
      byBrand: [
        { $group: { _id: "$brand", count: { $sum: 1 } } },
        { $sort: { count: -1 } },
      ],
      priceRanges: [
        { $bucket: {
            groupBy: "$price",
            boundaries: [0, 100, 500, 1000],
            default: "Other",
            output: { count: { $sum: 1 } }
        }},
      ],
  }},
])

Full-Text Search + Aggregation

db.articles.aggregate([
  { $match: { $text: { $search: "mongodb aggregation" } } },
  { $addFields: {
      score: { $meta: "textScore" }
  }},
  { $sort: { score: -1 } },
  { $limit: 10 },
])

Conditional Aggregation

// Revenue by payment method
db.orders.aggregate([
  { $group: {
      _id: null,
      creditCardRevenue: {
        $sum: { $cond: [
          { $eq: ["$paymentMethod", "credit_card"] },
          "$amount",
          0
        ]}
      },
      paypalRevenue: {
        $sum: { $cond: [
          { $eq: ["$paymentMethod", "paypal"] },
          "$amount",
          0
        ]}
      },
  }},
])

Performance Optimization

Index Usage

// Use indexes for $match and $sort
db.orders.createIndex({ status: 1, createdAt: -1 });

db.orders.aggregate([
  { $match: { status: "active" } },  // Uses index
  { $sort: { createdAt: -1 } },      // Uses index
  // ... other stages
])

// Check index usage
db.orders.explain().aggregate([...])

$match and $project Early

// ✅ Good: Filter and project early
db.large_collection.aggregate([
  { $match: { active: true } },           // Reduce documents
  { $project: { needed_field: 1 } },      // Reduce field count
  { $lookup: ... },                       // Expensive operation on smaller dataset
])

// ❌ Bad: Expensive operations on full collection
db.large_collection.aggregate([
  { $lookup: ... },                       // Processes all documents
  { $match: { active: true } },           // Filter after expensive operation
])

Limit Result Size

// Limit intermediate results
db.products.aggregate([
  { $match: { inStock: true } },
  { $sort: { popularity: -1 } },
  { $limit: 100 },                  // Limit before expensive stages
  { $lookup: { /* join details */ } },
])

Pipeline Validation

// Use $merge for debugging
db.orders.aggregate([
  { $match: { status: "pending" } },
  { $merge: { into: "debug_stage1" } },  // Save intermediate result
  { $group: { _id: "$userId", total: { $sum: "$amount" } } },
  { $merge: { into: "debug_stage2" } },
])

Best Practices

1. $match early - Filter before expensive operations 2. Use indexes - Ensure $match and $sort use indexes 3. $project unwanted fields - Reduce memory usage 4. Limit results - Use $limit early when possible 5. Avoid $lookup on large collections - Index foreign keys 6. Test with explain() - Verify index usage 7. Use $facet sparingly - Multiple sub-pipelines are expensive 8. Consider denormalization - Avoid $lookup for frequent queries

Resources

• MongoDB Aggregation Docs: https://www.mongodb.com/docs/manual/aggregation/
• Aggregation Pipeline Operators: https://www.mongodb.com/docs/manual/reference/operator/aggregation/

MongoDB Anti-Patterns

Common mistakes and how to avoid them.

Table of Contents

• 1. Unbounded Arrays
• 2. Over-Embedding
• 3. Collection Scans
• 4. Large Documents (>1MB)
• 5. Inefficient $lookup
• 6. Massive Projections
• 7. Client-Side Joins
• 8. Unbounded $group
• 9. No Error Handling
• 10. Wrong Data Types
• Summary of Best Practices
• Resources

1. Unbounded Arrays

❌ Problem:

{
  _id: ObjectId("..."),
  userId: 123,
  events: [
    { type: "login", timestamp: "..." },
    { type: "click", timestamp: "..." },
    // ... 10,000 more events (document grows forever)
  ]
}

✅ Solution: Use separate collection with reference

// User document (bounded)
{ _id: ObjectId("..."), userId: 123, email: "..." }

// Events collection
{ _id: ObjectId("..."), userId: 123, type: "login", timestamp: "..." }

// Query recent events
db.events.find({ userId: 123 }).sort({ timestamp: -1 }).limit(100)

Rule: Arrays with potential for >100 elements should be separate collections.

2. Over-Embedding

❌ Problem:

{
  _id: ObjectId("..."),
  title: "Blog Post",
  author: {
    _id: ObjectId("..."),
    name: "John",
    email: "john@example.com",
    bio: "Long biography...",
    socialLinks: [...],  // Embedded author data duplicated in every post
  },
  comments: [
    {
      author: { /* full author embedded again */ },
      replies: [
        { author: { /* embedded again */ } }  // Deeply nested
      ]
    }
  ]
}

✅ Solution: Reference pattern

// Post (minimal author info)
{
  _id: ObjectId("..."),
  title: "Blog Post",
  authorId: ObjectId("author_id"),  // Reference only
  authorName: "John",  // Denormalize only display name
}

// Fetch author details when needed
const post = db.posts.findOne({ _id: postId })
const author = db.users.findOne({ _id: post.authorId })

3. Collection Scans

❌ Problem:

// No index on status
db.orders.find({ status: "pending" })  // COLLSCAN on 1M documents

✅ Solution: Create index

db.orders.createIndex({ status: 1, createdAt: -1 })
db.orders.find({ status: "pending" })  // IXSCAN

4. Large Documents (>1MB)

❌ Problem:

{
  _id: ObjectId("..."),
  productId: 123,
  largeImage: "<base64 encoded 5MB image>",  // 16MB doc limit approaching
}

✅ Solution: Use GridFS or external storage

// Reference S3/GridFS
{
  _id: ObjectId("..."),
  productId: 123,
  imageUrl: "https://cdn.example.com/products/123.jpg",
  thumbnailUrl: "https://cdn.example.com/products/123_thumb.jpg",
}

5. Inefficient $lookup

❌ Problem:

// $lookup without index on foreign key
db.orders.aggregate([
  { $lookup: {
      from: "users",  // No index on users._id
      localField: "userId",
      foreignField: "_id",
      as: "user"
  }}
])

✅ Solution: Index foreign keys

// Create index on lookup field
db.users.createIndex({ _id: 1 })  // Usually exists by default

// Or denormalize frequently accessed fields
{
  _id: ObjectId("..."),
  userId: 123,
  userName: "John",  // Denormalized for display
  userEmail: "john@example.com",
}

6. Massive Projections

❌ Problem:

// Selecting all fields when only need few
db.users.find({}, { password: 0 })  // Returns everything except password

✅ Solution: Explicit projection

// Select only needed fields
db.users.find({}, { email: 1, name: 1, _id: 0 })

// Covering index (no document fetch)
db.users.createIndex({ email: 1, name: 1 })
db.users.find({}, { email: 1, name: 1, _id: 0 })

7. Client-Side Joins

❌ Problem:

// N+1 queries
const posts = await db.posts.find({}).toArray();

for (const post of posts) {
  post.author = await db.users.findOne({ _id: post.authorId });  // N queries!
}

✅ Solution: Aggregation or denormalization

// Aggregation (server-side join)
db.posts.aggregate([
  { $lookup: {
      from: "users",
      localField: "authorId",
      foreignField: "_id",
      as: "author"
  }},
  { $unwind: "$author" },
])

// Or denormalize
{
  _id: ObjectId("..."),
  authorId: ObjectId("..."),
  authorName: "John",  // Cached for display
}

8. Unbounded $group

❌ Problem:

// Group by high-cardinality field
db.events.aggregate([
  { $group: {
      _id: "$userId",  // Millions of unique users
      events: { $push: "$$ROOT" }  // Massive memory usage
  }}
])

✅ Solution: Add $match and $limit

db.events.aggregate([
  { $match: { timestamp: { $gte: recentDate } } },  // Filter first
  { $group: { _id: "$userId", count: { $sum: 1 } } },  // Don't $push all docs
  { $limit: 1000 },  // Limit results
])

9. No Error Handling

❌ Problem:

const user = await db.users.insertOne({ email: "duplicate@example.com" });
// Throws on duplicate email if unique index exists

✅ Solution: Handle errors

try {
  const user = await db.users.insertOne({ email: "user@example.com" });
} catch (error) {
  if (error.code === 11000) {  // Duplicate key error
    throw new Error("Email already exists");
  }
  throw error;
}

10. Wrong Data Types

❌ Problem:

{
  createdAt: "2025-12-03T10:00:00Z",  // String, not Date
  price: "49.99",  // String, not Number
  isActive: "true",  // String, not Boolean
}

✅ Solution: Use proper types

{
  createdAt: ISODate("2025-12-03T10:00:00Z"),  // Date object
  price: 49.99,  // Number
  isActive: true,  // Boolean
}

Summary of Best Practices

✅ DO:

• Reference for one-to-many (>100 related docs)
• Index all query filters
• Use cursor-based pagination
• Implement soft deletes for important data
• Denormalize display fields
• Use proper data types
• Handle errors explicitly
• Limit array sizes (<100 elements)
• Use aggregation for complex queries
• Monitor slow queries

❌ DON'T:

• Embed unbounded arrays
• Do client-side joins (N+1 queries)
• Skip indexing query fields
• Store large binary data in documents
• Use offset pagination for large datasets
• Store strings when you need dates/numbers
• Ignore 16MB document limit
• Over-normalize (too many $lookups)
• Create 10+ indexes per collection
• Use $regex without index

Resources

• MongoDB Anti-Patterns: https://www.mongodb.com/developer/products/mongodb/schema-design-anti-pattern-summary/

Common MongoDB Patterns

Frequently-used patterns for pagination, soft deletes, audit logs, and data modeling.

Table of Contents

• Pagination Patterns
• Cursor-Based (Recommended)
• Offset-Based (Simple Cases)
• Range-Based (Time Series)
• Soft Deletes
• Pattern 1: Boolean Flag
• Pattern 2: Status Field
• Audit Logs
• Pattern 1: Embedded History
• Pattern 2: Separate Audit Collection
• Versioning Documents
• Pattern: Version Number + History
• Counter Pattern (Atomic Increments)
• Hierarchical Data
• Pattern 1: Parent Reference
• Pattern 2: Materialized Path
• Upsert Pattern
• Bulk Operations
• Caching Pattern
• Best Practices
• Resources

Pagination Patterns

Cursor-Based (Recommended)

Handles real-time changes, no skipped records:

// First page
db.posts.find({})
  .sort({ _id: -1 })
  .limit(20)

// Next page (using last _id as cursor)
db.posts.find({ _id: { $lt: lastSeenId } })
  .sort({ _id: -1 })
  .limit(20)

API response:

{
  "items": [...],
  "nextCursor": "507f1f77bcf86cd799439011",
  "hasMore": true
}

Offset-Based (Simple Cases)

Only for static datasets <10K records:

const page = 2;
const perPage = 20;

db.posts.find({})
  .sort({ createdAt: -1 })
  .skip(page * perPage)
  .limit(perPage)

Problem: Performance degrades with large skip values.

Range-Based (Time Series)

// Page by date range
db.events.find({
  createdAt: {
    $gte: ISODate("2025-12-01"),
    $lt: ISODate("2025-12-02")
  }
})

Soft Deletes

Pattern 1: Boolean Flag

// Schema
{
  _id: ObjectId("..."),
  email: "user@example.com",
  deletedAt: null,  // or ISODate("...")
  isDeleted: false,
}

// Soft delete
db.users.updateOne(
  { _id: userId },
  { $set: { isDeleted: true, deletedAt: new Date() } }
)

// Query (exclude deleted)
db.users.find({ isDeleted: { $ne: true } })

// Create index for efficient querying
db.users.createIndex({ isDeleted: 1, createdAt: -1 })

Pattern 2: Status Field

// Schema with status
{
  _id: ObjectId("..."),
  status: "active",  // active | archived | deleted
  statusChangedAt: ISODate("..."),
}

// Archive (soft delete)
db.posts.updateOne(
  { _id: postId },
  { $set: { status: "archived", statusChangedAt: new Date() } }
)

// Query active only
db.posts.find({ status: "active" })

// Index
db.posts.createIndex({ status: 1, createdAt: -1 })

Audit Logs

Pattern 1: Embedded History

{
  _id: ObjectId("..."),
  email: "user@example.com",
  name: "John Doe",
  history: [
    {
      action: "created",
      timestamp: ISODate("2025-01-01T00:00:00Z"),
      by: ObjectId("admin_id"),
    },
    {
      action: "updated",
      timestamp: ISODate("2025-02-01T00:00:00Z"),
      by: ObjectId("user_id"),
      changes: { name: { old: "John", new: "John Doe" } },
    },
  ],
}

Use when: <100 updates per document

Pattern 2: Separate Audit Collection

// Main collection
db.users.updateOne({ _id: userId }, { $set: { name: "New Name" } })

// Audit log collection
db.audit_log.insertOne({
  collection: "users",
  documentId: userId,
  action: "update",
  changes: { name: { old: "John", new: "New Name" } },
  userId: currentUserId,
  timestamp: new Date(),
  ip: "192.168.1.1",
})

Use when: Many updates, need queryable audit history

Versioning Documents

Pattern: Version Number + History

{
  _id: ObjectId("..."),
  version: 3,
  content: "Current content",
  versions: [
    { version: 1, content: "Original", createdAt: ISODate("...") },
    { version: 2, content: "Updated", createdAt: ISODate("...") },
    { version: 3, content: "Current content", createdAt: ISODate("...") },
  ],
}

// Update with versioning
db.documents.updateOne(
  { _id: docId },
  {
    $inc: { version: 1 },
    $set: { content: newContent },
    $push: {
      versions: {
        version: currentVersion + 1,
        content: newContent,
        createdAt: new Date(),
      }
    }
  }
)

Counter Pattern (Atomic Increments)

// Page view counter
db.posts.updateOne(
  { _id: postId },
  { $inc: { views: 1 } }
)

// Multiple counters
db.posts.updateOne(
  { _id: postId },
  {
    $inc: { views: 1, shares: 1 },
    $set: { lastViewed: new Date() }
  }
)

Hierarchical Data

Pattern 1: Parent Reference

// Category tree
{
  _id: ObjectId("..."),
  name: "Electronics",
  parentId: null,  // Root category
}

{
  _id: ObjectId("..."),
  name: "Laptops",
  parentId: ObjectId("electronics_id"),  // Child
}

// Find all children
db.categories.find({ parentId: electronicsId })

// Find path to root
function getPath(categoryId) {
  const path = [];
  let current = db.categories.findOne({ _id: categoryId });

  while (current) {
    path.unshift(current.name);
    current = current.parentId ?
      db.categories.findOne({ _id: current.parentId }) :
      null;
  }

  return path;
}

Pattern 2: Materialized Path

{
  _id: ObjectId("..."),
  name: "Laptops",
  path: "Electronics,Computers,Laptops",  // Full path
}

// Find all descendants
db.categories.find({ path: /^Electronics,Computers/ })

// Index for performance
db.categories.createIndex({ path: 1 })

Upsert Pattern

// Insert if not exists, update if exists
db.users.updateOne(
  { email: "user@example.com" },
  {
    $set: { name: "John", lastLogin: new Date() },
    $setOnInsert: { createdAt: new Date() },  // Only on insert
  },
  { upsert: true }
)

Bulk Operations

// Batch inserts (faster than individual)
db.users.insertMany([
  { email: "user1@example.com", name: "User 1" },
  { email: "user2@example.com", name: "User 2" },
  // ... thousands more
], { ordered: false })  // Continue on error

// Bulk write operations
db.users.bulkWrite([
  {
    insertOne: {
      document: { email: "new@example.com", name: "New User" }
    }
  },
  {
    updateOne: {
      filter: { _id: ObjectId("...") },
      update: { $set: { name: "Updated" } }
    }
  },
  {
    deleteOne: {
      filter: { _id: ObjectId("...") }
    }
  }
])

Caching Pattern

// Cache frequently accessed data
db.users.aggregate([
  { $match: { _id: userId } },
  { $lookup: {
      from: "posts",
      localField: "_id",
      foreignField: "userId",
      as: "recentPosts",
      pipeline: [
        { $sort: { createdAt: -1 } },
        { $limit: 5 }
      ]
  }},
  { $project: {
      email: 1,
      name: 1,
      postCount: { $size: "$recentPosts" },
      recentPosts: { $slice: ["$recentPosts", 3] }
  }},
  { $merge: {
      into: "user_cache",
      whenMatched: "replace",
      whenNotMatched: "insert"
  }}
])

Best Practices

1. Index all queries - Every filter should have supporting index 2. Compound index order - Equality, Range, Sort 3. Use partial indexes - Index only needed subset 4. Cursor pagination - Avoid skip for large offsets 5. Soft deletes - Preserve data, add isDeleted flag 6. Audit critical changes - Log who/what/when 7. Version important docs - Maintain history 8. Use upserts - Idempotent operations 9. Bulk operations - Batch for performance 10. Monitor slow queries - Enable profiling

Resources

• MongoDB Schema Design: https://www.mongodb.com/docs/manual/core/data-modeling-introduction/
• Index Best Practices: https://www.mongodb.com/docs/manual/applications/indexes/

DynamoDB Complete Guide

AWS DynamoDB single-table design, GSI patterns, and serverless best practices.

Table of Contents

• Core Concepts
• Single-Table Design
• Primary Keys
• Global Secondary Indexes
• Query Patterns
• DynamoDB Streams
• Pricing Optimization
• Performance Best Practices

---

Core Concepts

DynamoDB vs Traditional Databases

Key Terminology

• Partition Key (PK): Hash key for data distribution
• Sort Key (SK): Range key for sorting within partition
• GSI: Global Secondary Index (alternate query patterns)
• LSI: Local Secondary Index (different sort key, same PK)
• WCU: Write Capacity Unit (1 KB/sec)
• RCU: Read Capacity Unit (4 KB/sec for eventual, 4 KB/sec for strong)

---

Single-Table Design

Philosophy

Design for access patterns, not normalization. Store multiple entity types in one table using composite keys.

Entity Types in One Table

// Users table with multiple entity types

// User metadata
{
  PK: "USER#12345",
  SK: "METADATA",
  email: "user@example.com",
  name: "Jane Doe",
  createdAt: "2025-01-15T10:00:00Z"
}

// User's orders
{
  PK: "USER#12345",
  SK: "ORDER#2025-001234",
  orderNumber: "ORD-2025-001234",
  totalAmount: 249.97,
  status: "shipped",
  orderDate: "2025-11-25T14:30:00Z"
}

// Order detail (different access pattern)
{
  PK: "ORDER#2025-001234",
  SK: "METADATA",
  userId: "USER#12345",
  totalAmount: 249.97,
  status: "shipped"
}

// Order items
{
  PK: "ORDER#2025-001234",
  SK: "ITEM#001",
  productId: "PROD-456",
  quantity: 2,
  price: 49.99,
  name: "Widget Pro"
}

// Product catalog
{
  PK: "PRODUCT#456",
  SK: "METADATA",
  name: "Widget Pro",
  category: "widgets",
  price: 49.99,
  inventory: 245
}

Access Patterns Enabled

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('AppData')

# 1. Get user metadata
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345') & Key('SK').eq('METADATA')
)

# 2. Get all orders for user
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345') & Key('SK').begins_with('ORDER#')
)

# 3. Get order with items
response = table.query(
    KeyConditionExpression=Key('PK').eq('ORDER#2025-001234')
)

# 4. Get specific order
response = table.query(
    KeyConditionExpression=Key('PK').eq('ORDER#2025-001234') & Key('SK').eq('METADATA')
)

Composite Key Strategies

Pattern 1: Entity Type Prefix

PK: USER#12345
SK: METADATA
SK: ORDER#001
SK: ORDER#002

Pattern 2: Hierarchical

PK: TENANT#abc
SK: USER#12345
SK: USER#12345#ORDER#001
SK: USER#12345#ORDER#002

Pattern 3: Reverse Relationship

// Same item, two representations
PK: USER#12345,  SK: ORDER#001
PK: ORDER#001,   SK: USER#12345

---

Primary Keys

Partition Key Only

# Create table with partition key only
dynamodb = boto3.client('dynamodb')

dynamodb.create_table(
    TableName='Users',
    KeySchema=[
        {'AttributeName': 'userId', 'KeyType': 'HASH'}  # PK only
    ],
    AttributeDefinitions=[
        {'AttributeName': 'userId', 'AttributeType': 'S'}
    ],
    BillingMode='PAY_PER_REQUEST'
)

# Put item
table.put_item(Item={
    'userId': 'user123',
    'email': 'user@example.com',
    'name': 'Jane Doe'
})

# Get item
response = table.get_item(Key={'userId': 'user123'})

Partition Key + Sort Key

# Create table with composite key
dynamodb.create_table(
    TableName='AppData',
    KeySchema=[
        {'AttributeName': 'PK', 'KeyType': 'HASH'},   # Partition Key
        {'AttributeName': 'SK', 'KeyType': 'RANGE'}   # Sort Key
    ],
    AttributeDefinitions=[
        {'AttributeName': 'PK', 'AttributeType': 'S'},
        {'AttributeName': 'SK', 'AttributeType': 'S'}
    ],
    BillingMode='PAY_PER_REQUEST'
)

Partition Key Design

Good Partition Keys:

• High cardinality (many unique values)
• Even access distribution
• No hot partitions

# GOOD: User ID (unique per user)
PK: "USER#12345"

# GOOD: Composite (tenant + user)
PK: "TENANT#abc#USER#12345"

# BAD: Status (hot partition on "active")
PK: "STATUS#active"  # All active users in one partition!

# BAD: Date (hot partition for current date)
PK: "DATE#2025-11-28"  # All today's writes to one partition!

Fix Hot Partitions:

# Add random suffix to distribute writes
import random
suffix = random.randint(0, 9)
PK: f"DATE#2025-11-28#{suffix}"

---

Global Secondary Indexes (GSI)

When to Use GSI

• Query by attributes other than primary key
• Different sort order
• Sparse indexes (not all items have attribute)

GSI Example: Query Orders by Status

# Base table: PK = userId, SK = timestamp
# GSI: PK = status, SK = timestamp (query all pending orders)

dynamodb.create_table(
    TableName='Orders',
    KeySchema=[
        {'AttributeName': 'userId', 'KeyType': 'HASH'},
        {'AttributeName': 'timestamp', 'KeyType': 'RANGE'}
    ],
    AttributeDefinitions=[
        {'AttributeName': 'userId', 'AttributeType': 'S'},
        {'AttributeName': 'timestamp', 'AttributeType': 'S'},
        {'AttributeName': 'status', 'AttributeType': 'S'}
    ],
    GlobalSecondaryIndexes=[
        {
            'IndexName': 'StatusIndex',
            'KeySchema': [
                {'AttributeName': 'status', 'KeyType': 'HASH'},
                {'AttributeName': 'timestamp', 'KeyType': 'RANGE'}
            ],
            'Projection': {'ProjectionType': 'ALL'},
            'BillingMode': 'PAY_PER_REQUEST'
        }
    ],
    BillingMode='PAY_PER_REQUEST'
)

# Query GSI
table = dynamodb.Table('Orders')
response = table.query(
    IndexName='StatusIndex',
    KeyConditionExpression=Key('status').eq('pending')
)

GSI Projection Types

# ALL: Project all attributes
'Projection': {'ProjectionType': 'ALL'}

# KEYS_ONLY: Project only keys (smallest index)
'Projection': {'ProjectionType': 'KEYS_ONLY'}

# INCLUDE: Project specific attributes
'Projection': {
    'ProjectionType': 'INCLUDE',
    'NonKeyAttributes': ['email', 'name', 'totalAmount']
}

GSI Best Practices

1. Use sparse indexes (only items with attribute are indexed) 2. Projection: Use KEYS_ONLY or INCLUDE to reduce storage 3. Limit GSIs: Maximum 20 per table 4. GSI writes: Every item update may update multiple GSIs (cost!)

---

Query Patterns

Query vs Scan

# GOOD: Query (efficient, uses index)
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345')
)

# BAD: Scan (reads entire table, expensive!)
response = table.scan(
    FilterExpression=Attr('email').eq('user@example.com')
)

Query Operators

from boto3.dynamodb.conditions import Key, Attr

# Equals
Key('PK').eq('USER#12345')

# Less than, greater than
Key('timestamp').lt('2025-11-01')
Key('timestamp').gte('2025-11-01')

# Between
Key('timestamp').between('2025-11-01', '2025-11-30')

# Begins with (sort key only)
Key('SK').begins_with('ORDER#')

Filter Expressions (Post-Query Filtering)

# Query + filter (filter after query, doesn't save RCUs)
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345'),
    FilterExpression=Attr('status').eq('active') & Attr('age').gte(18)
)

Pagination

# Paginate through large result sets
last_evaluated_key = None

while True:
    if last_evaluated_key:
        response = table.query(
            KeyConditionExpression=Key('PK').eq('USER#12345'),
            ExclusiveStartKey=last_evaluated_key
        )
    else:
        response = table.query(
            KeyConditionExpression=Key('PK').eq('USER#12345')
        )

    items = response['Items']
    # Process items

    last_evaluated_key = response.get('LastEvaluatedKey')
    if not last_evaluated_key:
        break

---

Update Operations

Update Expressions

# Set attribute
table.update_item(
    Key={'userId': 'user123'},
    UpdateExpression='SET #name = :name',
    ExpressionAttributeNames={'#name': 'name'},
    ExpressionAttributeValues={':name': 'Jane Smith'}
)

# Increment counter
table.update_item(
    Key={'userId': 'user123'},
    UpdateExpression='SET loginCount = loginCount + :inc',
    ExpressionAttributeValues={':inc': 1}
)

# Add to list
table.update_item(
    Key={'userId': 'user123'},
    UpdateExpression='SET tags = list_append(tags, :tag)',
    ExpressionAttributeValues={':tag': ['new-tag']}
)

# Add to set
table.update_item(
    Key={'userId': 'user123'},
    UpdateExpression='ADD emailSet :email',
    ExpressionAttributeValues={':email': {'user@example.com'}}
)

# Remove attribute
table.update_item(
    Key={'userId': 'user123'},
    UpdateExpression='REMOVE deprecated_field'
)

Conditional Updates

# Update only if condition met
from botocore.exceptions import ClientError

try:
    table.update_item(
        Key={'userId': 'user123'},
        UpdateExpression='SET balance = balance - :amount',
        ConditionExpression='balance >= :amount',
        ExpressionAttributeValues={':amount': 100}
    )
except ClientError as e:
    if e.response['Error']['Code'] == 'ConditionalCheckFailedException':
        print("Insufficient balance")

Atomic Counters

# Increment view count (atomic)
response = table.update_item(
    Key={'postId': 'post123'},
    UpdateExpression='SET viewCount = if_not_exists(viewCount, :start) + :inc',
    ExpressionAttributeValues={':start': 0, ':inc': 1},
    ReturnValues='UPDATED_NEW'
)
print(f"New count: {response['Attributes']['viewCount']}")

---

DynamoDB Streams

Enable Change Data Capture

# Create table with streams enabled
dynamodb.create_table(
    TableName='Orders',
    # ... (key schema, attributes)
    StreamSpecification={
        'StreamEnabled': True,
        'StreamViewType': 'NEW_AND_OLD_IMAGES'  # Full document before/after
    }
)

Stream View Types

• KEYS_ONLY: Only key attributes
• NEW_IMAGE: Entire item after update
• OLD_IMAGE: Entire item before update
• NEW_AND_OLD_IMAGES: Both before and after

Process Streams with Lambda

# Lambda function triggered by DynamoDB Stream
def lambda_handler(event, context):
    for record in event['Records']:
        if record['eventName'] == 'INSERT':
            new_item = record['dynamodb']['NewImage']
            print(f"New order: {new_item}")

            # Send notification
            send_email(new_item['email']['S'], "Order Confirmed")

        elif record['eventName'] == 'MODIFY':
            old_item = record['dynamodb']['OldImage']
            new_item = record['dynamodb']['NewImage']

            # Check if status changed
            if old_item['status']['S'] != new_item['status']['S']:
                print(f"Status changed: {old_item['status']['S']} -> {new_item['status']['S']}")

        elif record['eventName'] == 'REMOVE':
            old_item = record['dynamodb']['OldImage']
            print(f"Order deleted: {old_item}")

---

Pricing Optimization

Billing Modes

Cost Optimization Strategies

# 1. Use BatchGetItem (up to 100 items)
response = dynamodb.batch_get_item(
    RequestItems={
        'Users': {
            'Keys': [
                {'userId': 'user1'},
                {'userId': 'user2'},
                {'userId': 'user3'}
            ]
        }
    }
)

# 2. Use BatchWriteItem (up to 25 items)
dynamodb.batch_write_item(
    RequestItems={
        'Users': [
            {'PutRequest': {'Item': {'userId': 'user1', 'name': 'User 1'}}},
            {'PutRequest': {'Item': {'userId': 'user2', 'name': 'User 2'}}}
        ]
    }
)

# 3. Use projection expressions (fetch only needed attributes)
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345'),
    ProjectionExpression='email, #name, createdAt',
    ExpressionAttributeNames={'#name': 'name'}
)

# 4. Use eventually consistent reads (half the RCUs)
response = table.get_item(
    Key={'userId': 'user123'},
    ConsistentRead=False  # Default
)

# 5. Use TTL for auto-expiring data (free deletes!)
table.meta.client.update_time_to_live(
    TableName='Sessions',
    TimeToLiveSpecification={
        'Enabled': True,
        'AttributeName': 'ttl'
    }
)

# Add TTL to item (Unix timestamp)
import time
ttl_value = int(time.time()) + 86400  # Expire in 24 hours
table.put_item(Item={
    'sessionId': 'session123',
    'ttl': ttl_value
})

---

Performance Best Practices

Design for Even Distribution

# GOOD: High cardinality partition key
PK: f"USER#{uuid.uuid4()}"

# GOOD: Composite key with multiple dimensions
PK: f"TENANT#{tenant_id}#USER#{user_id}"

# BAD: Low cardinality (hot partitions)
PK: f"STATUS#{status}"  # Only 3 values: active, pending, archived

Use Sparse Indexes

# Only index items with specific attribute (saves storage)
# GSI: PK = emailVerified, SK = timestamp
# Only items with emailVerified=true are indexed

table.put_item(Item={
    'userId': 'user123',
    'email': 'user@example.com',
    'emailVerified': True,  # This item appears in GSI
    'timestamp': '2025-11-28T10:00:00Z'
})

table.put_item(Item={
    'userId': 'user456',
    'email': 'other@example.com',
    # No emailVerified - this item NOT in GSI
    'timestamp': '2025-11-28T11:00:00Z'
})

Adjacent Item Pattern (Reduce Queries)

# Store related items with adjacent sort keys
# Query once, get all related data

# User metadata
PK: "USER#12345",  SK: "A#METADATA"

# User's addresses
PK: "USER#12345",  SK: "B#ADDRESS#001"
PK: "USER#12345",  SK: "B#ADDRESS#002"

# User's orders (most recent first)
PK: "USER#12345",  SK: "C#ORDER#2025-11-28#001"
PK: "USER#12345",  SK: "C#ORDER#2025-11-27#002"

# One query gets everything
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#12345')
)

Composite Sort Keys

# Hierarchical data in sort key
SK: "COUNTRY#USA#STATE#MA#CITY#Boston"

# Query all items in USA
Key('SK').begins_with('COUNTRY#USA')

# Query all items in Massachusetts
Key('SK').begins_with('COUNTRY#USA#STATE#MA')

# Query all items in Boston
Key('SK').begins_with('COUNTRY#USA#STATE#MA#CITY#Boston')

---

TypeScript Examples (AWS SDK v3)

import {
  DynamoDBClient,
  PutItemCommand,
  GetItemCommand,
  QueryCommand,
  UpdateItemCommand
} from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({ region: 'us-east-1' })

// Put item
await client.send(new PutItemCommand({
  TableName: 'Users',
  Item: marshall({
    userId: 'user123',
    email: 'user@example.com',
    name: 'Jane Doe',
    createdAt: new Date().toISOString()
  })
}))

// Get item
const response = await client.send(new GetItemCommand({
  TableName: 'Users',
  Key: marshall({ userId: 'user123' })
}))
const user = unmarshall(response.Item!)

// Query
const queryResponse = await client.send(new QueryCommand({
  TableName: 'Orders',
  KeyConditionExpression: 'PK = :pk',
  ExpressionAttributeValues: marshall({
    ':pk': 'USER#12345'
  })
}))
const items = queryResponse.Items!.map(item => unmarshall(item))

// Update
await client.send(new UpdateItemCommand({
  TableName: 'Users',
  Key: marshall({ userId: 'user123' }),
  UpdateExpression: 'SET #name = :name',
  ExpressionAttributeNames: { '#name': 'name' },
  ExpressionAttributeValues: marshall({ ':name': 'Jane Smith' })
}))

---

Common Patterns

Multi-Tenant Architecture

# Partition by tenant
PK: "TENANT#abc#USER#12345"
SK: "METADATA"

# Query all users in tenant
response = table.query(
    KeyConditionExpression=Key('PK').begins_with('TENANT#abc#USER#')
)

Time-Series Data

# Partition by entity, sort by timestamp
PK: "SENSOR#sensor123"
SK: "2025-11-28T10:30:45.123Z"

# Query range
response = table.query(
    KeyConditionExpression=Key('PK').eq('SENSOR#sensor123') &
                          Key('SK').between('2025-11-28T00:00:00Z', '2025-11-28T23:59:59Z')
)

# Use GSI for cross-sensor queries
# GSI: PK = date, SK = sensorId

Versioning

# Store versions with sort key
PK: "DOC#doc123"
SK: "VERSION#001"
SK: "VERSION#002"
SK: "VERSION#003"

# Get latest version
response = table.query(
    KeyConditionExpression=Key('PK').eq('DOC#doc123'),
    ScanIndexForward=False,  # Descending order
    Limit=1
)

---

This guide covers DynamoDB single-table design and AWS-specific patterns. For Python FastAPI + MongoDB examples, see ../examples/dynamodb-serverless/.

Firestore Complete Guide

Firebase/GCP Firestore real-time sync, security rules, and mobile-first patterns.

Table of Contents

• Core Concepts
• Data Model
• Real-Time Listeners
• Security Rules
• Queries
• Offline Support
• Performance Best Practices

---

Core Concepts

Firestore vs Realtime Database

Key Features

• Real-time sync: Live updates across all clients
• Offline-first: Local cache, auto-sync when online
• Security rules: Declarative access control
• Atomic operations: Batched writes, transactions
• Automatic indexing: Composite indexes for queries

---

Data Model

Collections and Documents

users (collection)
├── user123 (document)
│   ├── email: "user@example.com"
│   ├── name: "Jane Doe"
│   └── orders (subcollection)
│       ├── order001 (document)
│       └── order002 (document)
└── user456 (document)
    └── ...

Document Structure

// Document in users collection
{
  // Auto-generated ID
  id: "user123",

  // Document data
  email: "user@example.com",
  name: "Jane Doe",
  age: 32,
  address: {
    street: "123 Main St",
    city: "Boston",
    state: "MA"
  },
  tags: ["premium", "verified"],
  createdAt: Timestamp,
  metadata: {
    lastLogin: Timestamp,
    loginCount: 42
  }
}

Document Limits:

• Max size: 1 MB
• Max depth: 20 levels
• Max field name: 1,500 bytes

---

Real-Time Listeners

React Component with Real-Time Updates

import { collection, query, where, onSnapshot } from 'firebase/firestore'
import { useEffect, useState } from 'react'

function OrderList({ userId }: { userId: string }) {
  const [orders, setOrders] = useState([])
  const [loading, setLoading] = useState(true)

  useEffect(() => {
    const q = query(
      collection(db, 'orders'),
      where('userId', '==', userId),
      where('status', '==', 'pending')
    )

    // Real-time listener
    const unsubscribe = onSnapshot(q, (snapshot) => {
      const orderData = snapshot.docs.map(doc => ({
        id: doc.id,
        ...doc.data()
      }))
      setOrders(orderData)
      setLoading(false)
    }, (error) => {
      console.error('Error:', error)
    })

    // Cleanup on unmount
    return () => unsubscribe()
  }, [userId])

  if (loading) return <div>Loading...</div>

  return (
    <div>
      {orders.map(order => (
        <div key={order.id}>{order.orderNumber}</div>
      ))}
    </div>
  )
}

Listening to Document Changes

import { doc, onSnapshot } from 'firebase/firestore'

// Listen to single document
const unsubscribe = onSnapshot(doc(db, 'users', userId), (doc) => {
  if (doc.exists()) {
    console.log('User data:', doc.data())
  }
})

// Detect change type
onSnapshot(collection(db, 'orders'), (snapshot) => {
  snapshot.docChanges().forEach((change) => {
    if (change.type === 'added') {
      console.log('New order:', change.doc.data())
    }
    if (change.type === 'modified') {
      console.log('Modified order:', change.doc.data())
    }
    if (change.type === 'removed') {
      console.log('Removed order:', change.doc.data())
    }
  })
})

---

Security Rules

Basic Rules Structure

// firestore.rules
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    // Rules go here
  }
}

Common Patterns

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    // 1. Public read, authenticated write
    match /products/{productId} {
      allow read: if true;
      allow write: if request.auth != null;
    }

    // 2. User can only access their own data
    match /users/{userId} {
      allow read, write: if request.auth != null && request.auth.uid == userId;
    }

    // 3. Orders: users can only see their own
    match /orders/{orderId} {
      // Read: must be authenticated and own the order
      allow read: if request.auth != null &&
                     resource.data.userId == request.auth.uid;

      // Create: must be authenticated and set userId to their own ID
      allow create: if request.auth != null &&
                       request.resource.data.userId == request.auth.uid;

      // Update/Delete: must own the order
      allow update, delete: if request.auth != null &&
                               resource.data.userId == request.auth.uid;
    }

    // 4. Admin-only writes
    match /config/{document} {
      allow read: if true;
      allow write: if request.auth != null &&
                      request.auth.token.admin == true;
    }

    // 5. Validate data on write
    match /posts/{postId} {
      allow create: if request.auth != null &&
                       request.resource.data.title is string &&
                       request.resource.data.title.size() > 0 &&
                       request.resource.data.title.size() <= 100 &&
                       request.resource.data.userId == request.auth.uid;
    }

    // 6. Subcollection access
    match /users/{userId}/orders/{orderId} {
      allow read, write: if request.auth != null && request.auth.uid == userId;
    }

    // 7. Rate limiting (prevent abuse)
    match /posts/{postId} {
      allow create: if request.auth != null &&
                       request.time > resource.data.lastPost + duration.value(1, 'm');
    }
  }
}

Rule Functions

// Helper functions
function isSignedIn() {
  return request.auth != null;
}

function isOwner(userId) {
  return request.auth.uid == userId;
}

function isAdmin() {
  return isSignedIn() && request.auth.token.admin == true;
}

function validString(field, minLen, maxLen) {
  let value = request.resource.data[field];
  return value is string &&
         value.size() >= minLen &&
         value.size() <= maxLen;
}

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /posts/{postId} {
      allow read: if true;
      allow create: if isSignedIn() &&
                       isOwner(request.resource.data.userId) &&
                       validString('title', 1, 100);
    }
  }
}

---

Queries

Basic Queries

import { collection, query, where, orderBy, limit, getDocs } from 'firebase/firestore'

// Simple equality
const q = query(
  collection(db, 'orders'),
  where('userId', '==', 'user123')
)
const snapshot = await getDocs(q)
snapshot.forEach(doc => console.log(doc.data()))

// Multiple conditions (AND)
const q = query(
  collection(db, 'products'),
  where('category', '==', 'electronics'),
  where('price', '<=', 500),
  orderBy('price', 'asc')
)

// OR queries (requires composite index)
const q = query(
  collection(db, 'products'),
  or(
    where('category', '==', 'electronics'),
    where('category', '==', 'books')
  )
)

// In queries (up to 10 values)
const q = query(
  collection(db, 'products'),
  where('category', 'in', ['electronics', 'books', 'toys'])
)

// Array contains
const q = query(
  collection(db, 'users'),
  where('tags', 'array-contains', 'premium')
)

// Array contains any
const q = query(
  collection(db, 'users'),
  where('tags', 'array-contains-any', ['premium', 'verified'])
)

Pagination

// First page
const first = query(
  collection(db, 'products'),
  orderBy('price'),
  limit(20)
)
const snapshot = await getDocs(first)
const lastVisible = snapshot.docs[snapshot.docs.length - 1]

// Next page
const next = query(
  collection(db, 'products'),
  orderBy('price'),
  startAfter(lastVisible),
  limit(20)
)

Composite Indexes

Firestore automatically creates indexes for simple queries. Composite indexes required for:

• Multiple inequality filters
• Inequality + orderBy on different fields
• OR queries

// Requires composite index: category (asc), price (asc)
const q = query(
  collection(db, 'products'),
  where('category', '==', 'electronics'),
  orderBy('price', 'asc')
)

Create index via Firebase Console or CLI:

firebase deploy --only firestore:indexes

---

CRUD Operations

Create

import { collection, addDoc, setDoc, doc } from 'firebase/firestore'

// Auto-generate ID
const docRef = await addDoc(collection(db, 'users'), {
  email: 'user@example.com',
  name: 'Jane Doe',
  createdAt: new Date()
})
console.log('Created with ID:', docRef.id)

// Custom ID
await setDoc(doc(db, 'users', 'user123'), {
  email: 'user@example.com',
  name: 'Jane Doe'
})

// Merge (update if exists, create if not)
await setDoc(doc(db, 'users', 'user123'), {
  lastLogin: new Date()
}, { merge: true })

Read

import { doc, getDoc, collection, getDocs } from 'firebase/firestore'

// Get single document
const docSnap = await getDoc(doc(db, 'users', 'user123'))
if (docSnap.exists()) {
  console.log(docSnap.data())
}

// Get all documents in collection
const querySnapshot = await getDocs(collection(db, 'users'))
querySnapshot.forEach(doc => console.log(doc.id, doc.data()))

Update

import { doc, updateDoc, increment, arrayUnion, serverTimestamp } from 'firebase/firestore'

// Update fields
await updateDoc(doc(db, 'users', 'user123'), {
  name: 'Jane Smith',
  'address.city': 'Boston'  // Nested field
})

// Increment counter
await updateDoc(doc(db, 'posts', 'post123'), {
  views: increment(1)
})

// Add to array (no duplicates)
await updateDoc(doc(db, 'users', 'user123'), {
  tags: arrayUnion('premium')
})

// Remove from array
await updateDoc(doc(db, 'users', 'user123'), {
  tags: arrayRemove('trial')
})

// Server timestamp
await updateDoc(doc(db, 'users', 'user123'), {
  updatedAt: serverTimestamp()
})

Delete

import { doc, deleteDoc, deleteField } from 'firebase/firestore'

// Delete document
await deleteDoc(doc(db, 'users', 'user123'))

// Delete field
await updateDoc(doc(db, 'users', 'user123'), {
  phoneNumber: deleteField()
})

---

Transactions and Batches

Transactions (Atomic Reads and Writes)

import { runTransaction } from 'firebase/firestore'

// Transfer credits between users
await runTransaction(db, async (transaction) => {
  const fromRef = doc(db, 'users', 'user123')
  const toRef = doc(db, 'users', 'user456')

  const fromDoc = await transaction.get(fromRef)
  if (!fromDoc.exists()) {
    throw new Error('User not found')
  }

  const currentBalance = fromDoc.data().credits
  if (currentBalance < 100) {
    throw new Error('Insufficient credits')
  }

  transaction.update(fromRef, { credits: currentBalance - 100 })
  transaction.update(toRef, { credits: increment(100) })
})

Batched Writes

import { writeBatch } from 'firebase/firestore'

// Batch write (up to 500 operations)
const batch = writeBatch(db)

batch.set(doc(db, 'users', 'user1'), { name: 'User 1' })
batch.update(doc(db, 'users', 'user2'), { active: true })
batch.delete(doc(db, 'users', 'user3'))

await batch.commit()

---

Offline Support

Enable Offline Persistence

import { initializeFirestore, persistentLocalCache } from 'firebase/firestore'

const db = initializeFirestore(app, {
  localCache: persistentLocalCache()
})

Offline Behavior

import { onSnapshot } from 'firebase/firestore'

// Listener works offline
onSnapshot(collection(db, 'orders'), (snapshot) => {
  snapshot.forEach(doc => {
    // fromCache indicates if data is from local cache
    console.log(`${doc.id} (from cache: ${doc.metadata.fromCache})`)
  })
})

// Writes queued offline, synced when online
await addDoc(collection(db, 'orders'), {
  userId: 'user123',
  items: [...]
})
// If offline, write is queued and will sync when online

---

Performance Best Practices

Minimize Document Reads

// BAD: Read same document multiple times
const userDoc = await getDoc(doc(db, 'users', userId))
// ... later ...
const userDoc2 = await getDoc(doc(db, 'users', userId))  // Duplicate read!

// GOOD: Cache document in memory
const userDoc = await getDoc(doc(db, 'users', userId))
const userData = userDoc.data()
// Use userData throughout component

Use Subcollections for Large Arrays

// BAD: Store unbounded array in document
{
  userId: "user123",
  orders: [
    { orderId: "order1", ... },
    { orderId: "order2", ... },
    // ... 1000s of orders (exceeds 1MB limit!)
  ]
}

// GOOD: Use subcollection
// users/user123/orders/order1
// users/user123/orders/order2
const ordersRef = collection(db, 'users', userId, 'orders')

Denormalize for Read Performance

// Store frequently accessed data together
{
  postId: "post123",
  title: "My Post",
  content: "...",
  // Denormalize author info (instead of reference)
  author: {
    id: "user123",
    name: "Jane Doe",
    avatar: "/avatars/jane.jpg"
  },
  // Update pattern: when user updates profile, update all posts
}

Use Server Timestamps

import { serverTimestamp } from 'firebase/firestore'

// Better than client timestamp (avoids clock skew)
await addDoc(collection(db, 'posts'), {
  title: 'My Post',
  createdAt: serverTimestamp()
})

---

Mobile Integration (React Native)

import { initializeApp } from 'firebase/app'
import { getFirestore, collection, onSnapshot } from 'firebase/firestore'
import { useEffect, useState } from 'react'

const firebaseConfig = {
  apiKey: "...",
  authDomain: "...",
  projectId: "..."
}

const app = initializeApp(firebaseConfig)
const db = getFirestore(app)

function OrdersScreen({ userId }) {
  const [orders, setOrders] = useState([])

  useEffect(() => {
    const unsubscribe = onSnapshot(
      collection(db, 'orders'),
      where('userId', '==', userId),
      (snapshot) => {
        const orderData = snapshot.docs.map(doc => ({
          id: doc.id,
          ...doc.data()
        }))
        setOrders(orderData)
      }
    )

    return () => unsubscribe()
  }, [userId])

  return (
    <FlatList
      data={orders}
      keyExtractor={item => item.id}
      renderItem={({ item }) => <OrderItem order={item} />}
    />
  )
}

---

This guide covers Firestore real-time patterns and mobile-first architecture. For complete React implementation, see ../examples/firestore-react/.

MongoDB Indexing Strategies

Complete guide to index types, optimization, and best practices for document databases.

Table of Contents

• Index Types
• Single Field Index
• Compound Index
• Text Index (Full-Text Search)
• Geospatial Index
• Partial Index (Index Subset)
• TTL Index (Auto-Delete)
• Sparse Index
• Index Selection Rules
• Covering Indexes
• Performance Optimization
• Index Intersection
• Index Prefix Usage
• Monitoring and Analysis
• List Indexes
• Index Usage Stats
• Explain Query
• Slow Query Log
• Index Maintenance
• Rebuild Index
• Drop Unused Indexes
• Best Practices
• Anti-Patterns
• Resources

Index Types

Single Field Index

db.users.createIndex({ email: 1 }, { unique: true })  // Ascending
db.posts.createIndex({ createdAt: -1 })              // Descending

Compound Index

Order matters! Equality → Range → Sort

// Query: WHERE status = 'active' AND createdAt > date ORDER BY createdAt
db.orders.createIndex({ status: 1, createdAt: -1 })

// Query: WHERE userId = 123 AND status = 'active' ORDER BY createdAt
db.orders.createIndex({ userId: 1, status: 1, createdAt: -1 })

Text Index (Full-Text Search)

db.articles.createIndex({
  title: "text",
  content: "text",
  tags: "text"
}, {
  weights: { title: 3, content: 1, tags: 2 },  // Title 3x more important
  name: "article_text_index"
})

// Search
db.articles.find({ $text: { $search: "mongodb indexing" } })
  .sort({ score: { $meta: "textScore" } })

Geospatial Index

db.locations.createIndex({ location: "2dsphere" })

// Find nearby
db.locations.find({
  location: {
    $near: {
      $geometry: { type: "Point", coordinates: [-73.97, 40.77] },
      $maxDistance: 5000,  // 5km
    }
  }
})

Partial Index (Index Subset)

// Only index active users
db.users.createIndex(
  { email: 1 },
  { partialFilterExpression: { status: { $eq: "active" } } }
)

// Only index large orders
db.orders.createIndex(
  { userId: 1, createdAt: -1 },
  { partialFilterExpression: { amount: { $gte: 1000 } } }
)

TTL Index (Auto-Delete)

// Auto-delete sessions after 30 days
db.sessions.createIndex(
  { createdAt: 1 },
  { expireAfterSeconds: 2592000 }  // 30 days
)

Sparse Index

// Only index documents with field
db.users.createIndex({ phone: 1 }, { sparse: true })

Index Selection Rules

Compound index order: 1. Equality filters first 2. Range filters second 3. Sort fields last

// Query: status = X AND createdAt > Y ORDER BY createdAt
// Index: { status: 1, createdAt: -1 }  ✓ Correct order
// Index: { createdAt: -1, status: 1 }  ✗ Wrong order

Covering Indexes

Index includes all queried fields (no document fetch needed):

// Query needs: userId, createdAt, amount
db.orders.createIndex({ userId: 1, createdAt: -1, amount: 1 })

// Query
db.orders.find(
  { userId: 123 },
  { userId: 1, createdAt: 1, amount: 1, _id: 0 }  // Only indexed fields
).sort({ createdAt: -1 })

// COVERED - no document fetch!

Performance Optimization

Index Intersection

MongoDB can combine multiple indexes:

db.orders.createIndex({ userId: 1 })
db.orders.createIndex({ status: 1 })

// MongoDB automatically intersects indexes
db.orders.find({ userId: 123, status: "pending" })

Index Prefix Usage

Compound indexes can support prefix queries:

db.orders.createIndex({ userId: 1, status: 1, createdAt: -1 })

// Supports queries on:
// { userId }
// { userId, status }
// { userId, status, createdAt }

// Does NOT support:
// { status }
// { createdAt }

Monitoring and Analysis

List Indexes

db.collection.getIndexes()

Index Usage Stats

db.collection.aggregate([{ $indexStats: {} }])

Explain Query

db.orders.explain("executionStats").find({ userId: 123 })

// Check for:
// - "stage": "IXSCAN" (good - uses index)
// - "stage": "COLLSCAN" (bad - full collection scan)
// - "totalDocsExamined" should be close to "nReturned"

Slow Query Log

// Enable profiling
db.setProfilingLevel(1, { slowms: 100 })  // Log queries >100ms

// View slow queries
db.system.profile.find().sort({ ts: -1 }).limit(10)

Index Maintenance

Rebuild Index

db.collection.reIndex()  // Rebuilds all indexes

Drop Unused Indexes

// Find unused indexes
db.collection.aggregate([{ $indexStats: {} }])

// Drop index
db.collection.dropIndex("index_name")

Best Practices

1. Index queried fields - Every filter/sort should have index 2. Compound index order - Equality, Range, Sort 3. Use covering indexes - Avoid document fetches 4. Partial indexes - Index only needed subset 5. Monitor index usage - Drop unused indexes 6. Limit index count - Each index slows writes 7. Use explain() - Verify index usage 8. Index foreign keys - For $lookup performance 9. TTL indexes - Auto-cleanup for temp data 10. Text indexes - One per collection max

Anti-Patterns

❌ Over-indexing - 10+ indexes per collection slows writes ❌ Wrong compound order - Range before equality ❌ No index on filters - Collection scans on large data ❌ Index low-cardinality - Boolean fields rarely help ❌ Ignore index size - Indexes consume RAM

Resources

• MongoDB Indexes: https://www.mongodb.com/docs/manual/indexes/
• Index Performance: https://www.mongodb.com/docs/manual/core/index-performance/