Mongodb Schema Design

MongoDB Schema Design

Data modeling patterns and anti-patterns for MongoDB, maintained by MongoDB. Bad schema is the root cause of most MongoDB performance and cost issues—queries and indexes cannot fix a fundamentally wrong model.

When to Apply

Reference these guidelines when:

• Designing a new MongoDB schema from scratch
• Migrating from SQL/relational databases to MongoDB
• Reviewing existing data models for performance issues
• Troubleshooting slow queries or growing document sizes
• Deciding between embedding and referencing
• Modeling relationships (one-to-one, one-to-many, many-to-many)
• Implementing tree/hierarchical structures
• Seeing Atlas Schema Suggestions or Performance Advisor warnings
• Hitting the 16MB document limit
• Adding schema validation to existing collections

Quick Reference

1. Schema Anti-Patterns - 3 rules

• antipattern-unnecessary-collections - Splitting homogeneous data into multiple collections is often an anti-pattern; consult this reference to validate whether this is the case.
• antipattern-excessive-lookups - When encountering overly normalized collections that reference each other or frequent and possibly slow $lookup operations, consult this reference to validate whether this is problematic and how to fix it.
• antipattern-unnecessary-indexes - Consult this reference when indexes overlap or are not used by queries, to identify and remove unnecessary indexes that add overhead without benefit.

2. Schema Fundamentals - 4 rules

• fundamental-embed-vs-reference - Consult this reference for approaches to modeling different types of relationships (1:1, 1:few, 1:many, many:many, tree/hierarchical data) and how to decide between embedding and referencing based on access patterns.
• fundamental-document-model - Fundamentals of the document model. Consult this reference when migrating from SQL or other normalized data to a document database like MongoDB.
• fundamental-schema-validation - Consult this reference when creating new collections, or adding validation to existing collections, for example in response to finding inconsistent document structures or data quality issues.
• fundamental-document-size - Consult this reference when documents hit the hard 16MB limit, or when accesses are slower than expected as a result of large documents.

3. Design Patterns - 11 rules

• pattern-approximation - Use approximate values for high-frequency counters
• pattern-archive - Move historical data to separate/cold storage for performance
• pattern-attribute - Collapse many optional fields into key-value attributes
• pattern-bucket - Group time-series or IoT data into buckets
• pattern-computed - Pre-calculate expensive aggregations
• pattern-document-versioning - Track document changes to enable historical queries and audit trails
• pattern-extended-reference - Cache frequently-accessed data from related entities
• pattern-outlier - Handle collections in which a small subset of documents are much larger than the rest, to prevent outliers from dominating memory and index costs
• pattern-polymorphic - Store different types of entities in the same collection, often when they are different types of the same base entity (e.g. different types of users or different types of products)
• pattern-schema-versioning - Schema evolution, preventing drift, and safe online migrations. Consult when encountering inconsistent document structures, or when planning a schema change that cannot be applied atomically.
• pattern-time-series-collections - Use native time series collections for high-frequency time series data

Key Principle

"Data that is accessed together should be stored together."

This is MongoDB's core philosophy. Embedding related data eliminates joins, reduces round trips, and enables atomic updates. Reference only when you must.

A core way to implement this philosophy is the fact that MongoDB exposes flexible schemas. This means you can have different fields in different documents, and even different structures. This allows you to model data in the way that best fits your access patterns, without being constrained by a rigid schema. For example, if different documents have different sets of fields, that is perfectly fine as long as it serves your application's needs. You can also use schema validation to enforce certain rules while still allowing for flexibility.

Another implication of the key principle is that information about the expected read and write workload becomes very relevant to schema design. If pieces of information from different entities are often queried or updated together, that means that prioritizing co-location of that data in the same document can lead to significant performance benefits. On the other hand, if certain pieces of information are rarely accessed together, it may make sense to store them separately to avoid loading more data than necessary.

Schema Fundamentals Summary

• Embed vs Reference: Choose embedding or referencing based on access patterns: embed when data is always accessed together (1:1, 1:few, bounded arrays, atomic updates needed); reference when data is accessed independently, relationships are many-to-many, or arrays can grow without bound.
• Data accessed together stored together: MongoDB's core principle: design schemas around queries, not entities. Embed related data to eliminate cross-collection joins and reduce round trips. Identify your API endpoints/pages, list the data each returns, then shape documents to match those queries.
• Embrace the document model: Don't recreate SQL tables 1:1 as MongoDB collections. Instead, denormalize joined tables into rich documents for single-query reads and atomic updates. When migrating from SQL, identify tables that are always joined together and merge them into single documents.
• Schema validation: Use MongoDB's built-in $jsonSchema validator to catch invalid data at the database level (type checks, required fields, enum constraints, array size limits). Start with validationLevel: "moderate" and validationAction: "warn" on existing collections, then tighten to strict/error.
• 16MB document limit: MongoDB documents cannot exceed 16MB—this is a hard limit, not a guideline. Common causes: unbounded arrays, large embedded binaries, deeply nested objects. Mitigate by moving unbounded data to separate collections and monitoring document sizes with $bsonSize.

Embed/Reference Decision Framework

This is a rough guideline, and whether to embed or reference depends on your specific access patterns, data size, and read/write frequencies. Always verify with your actual workload.

How to Use

Each reference file listed above contains detailed explanations and code examples. Use the descriptions in the Quick Reference to identify which files are relevant to your current task.

Each reference file contains:

• Brief explanation of why it matters
• Incorrect code example with explanation
• Correct code example with explanation
• "When NOT to use" exceptions
• Performance impact and metrics
• Verification diagnostics

---

How These Rules Work

MongoDB MCP Integration

For automatic verification, connect the MongoDB MCP Server.

If the MCP server is running and connected, I can automatically run verification commands to check your actual schema, document sizes, array lengths, index usage, and more. This allows me to provide tailored recommendations based on your real data, not just code patterns.

⚠️ Security: Use --readOnly for safety. Remove only if you need write operations.

When connected, I can automatically:

• Infer schema via mcp__mongodb__collection-schema
• Measure document/array sizes via mcp__mongodb__aggregate
• Check collection statistics via mcp__mongodb__db-stats

⚠️ Action Policy

I will NEVER execute write operations without your explicit approval.

Before any write or destructive operation via MCP, I will: (1) summarize the exact operation (collection, index/validator, estimated number of docs affected), and (2) ask for explicit confirmation (yes/no). I will not proceed on partial or ambiguous approvals.

When I recommend schema changes or data modifications: 1. I'll explain what I want to do and why 2. I'll show you the exact command 3. I'll wait for your approval before executing 4. If you say "go ahead" or "yes", only then will I run it

Your database, your decision. I'm here to advise, not to act unilaterally.

Working Together

If you're not sure about a recommendation: 1. Run the verification commands I provide 2. Share the output with me 3. I'll adjust my recommendation based on your actual data

We're a team—let's get this right together.

Reduce Excessive $lookup Usage

Frequent $lookup operations on hot paths can indicate over-normalization. $lookup is useful, but repeated joins can be slower and more resource-intensive than querying a single collection, especially when supporting indexes or match selectivity are weak. If the same related fields are read together often, consider embedding or extended references.

Incorrect (constant $lookup for common operations):

// Every product page requires repeated joins across collections
db.products.aggregate([
  { $match: { _id: productId } },
  { $lookup: {
      from: "categories",          // Collection scan #2
      localField: "categoryId",
      foreignField: "_id",
      as: "category"
  }},
  { $lookup: {
      from: "brands",              // Collection scan #3
      localField: "brandId",
      foreignField: "_id",
      as: "brand"
  }},
  { $unwind: "$category" },
  { $unwind: "$brand" }
])
// Multiple join stages add planning/execution overhead on hot paths

Join cost depends on cardinality, stage order, index support, and result size. Measure before deciding to embed.

Correct (denormalize frequently-joined data):

Embed data that is always displayed alongside the product directly in the product document: include category fields (_id, name, path) and brand fields (_id, name, logo) as subdocuments. A single indexed query returns complete product data without $lookup. Listing queries (e.g. by category) also run against a single collection.

Managing denormalized data updates:

When category data changes (a rare event), use updateMany to update all products matching that category’s _id with the new field values. For frequently-changing data, keep both a reference ID (brandId) and a cache subdocument (brandCache) with a cachedAt timestamp; refresh the cache when it exceeds a staleness threshold.

When NOT to use this pattern:

• Data changes frequently and independently: If brand logos change daily, denormalization creates update overhead.
• Rarely-accessed data: Don't embed review details if only a small fraction of product views load reviews.
• Many-to-many with high cardinality: Avoid embedding large or fast-growing relationship sets.
• Analytics queries: Batch jobs can afford $lookup latency; real-time queries cannot.

Verify with

// Find pipelines with multiple $lookup stages
db.setProfilingLevel(1, { slowms: 50 }) // Disable afterwards
db.system.profile.find({
  "command.aggregate": { $exists: true },
  "command.pipeline.$lookup": {
    $exists: true
  }
}).sort({ millis: -1 })

// Check if $lookup foreign fields are indexed
db.reviews.aggregate([
  { $indexStats: {} }
])
// Look for index supporting the query in result

// Measure $lookup impact
db.products.aggregate([
  { $match: { category: "electronics" } },
  { $lookup: { from: "brands", localField: "brandId", foreignField: "_id", as: "brand" } }
]).explain("executionStats")
// Check totalDocsExamined in $lookup stage

Atlas Schema Suggestions flags: "Reduce $lookup operations"

Reference: Reduce Lookup Operations

Reduce Unnecessary Collections

Collection count alone is not the anti-pattern. The anti-pattern is using collections as a substitute for indexes — creating one collection per category, time period, or partition key instead of indexing a single collection. Every collection carries a default _id index that consumes storage and strains the replica set, and cross-collection queries require $lookup or $unionWith, adding complexity and overhead.

Incorrect (one collection per day as partitioning strategy):

Creating one collection per time period (e.g. temperatures_2024_05_10, temperatures_2024_05_11, …) means each collection carries its own default _id index (365 collections/year = 365 extra indexes), cross-day queries require $unionWith across many collections, schema validation / indexes / TTL must be duplicated on every collection, and application code must dynamically resolve the collection name for each query.

Correct (single collection with an index):

// All readings in one collection — the index does the partitioning work
{ _id: ObjectId(), timestamp: ISODate("2024-05-10T10:00:00Z"), temperature: 60 }
{ _id: ObjectId(), timestamp: ISODate("2024-05-10T11:00:00Z"), temperature: 61 }
{ _id: ObjectId(), timestamp: ISODate("2024-05-11T10:00:00Z"), temperature: 68 }

db.temperatures.createIndex({ timestamp: 1 })

// Efficient range query — one collection, one index
db.temperatures.find({
  timestamp: { $gte: ISODate("2024-05-10"), $lt: ISODate("2024-05-11") }
})

// Optional TTL for automatic expiry (e.g. 90 days)
db.temperatures.createIndex({ timestamp: 1 }, { expireAfterSeconds: 7776000 })

Even better (bucket pattern or time series collection):

For high-volume time-stamped data, group readings into buckets or use a native time series collection, which is optimized for this workload:

// Bucket pattern — one document per day
{
  _id: ISODate("2024-05-10T00:00:00Z"),
  readings: [
    { timestamp: ISODate("2024-05-10T10:00:00Z"), temperature: 60 },
    { timestamp: ISODate("2024-05-10T11:00:00Z"), temperature: 61 },
    { timestamp: ISODate("2024-05-10T12:00:00Z"), temperature: 64 }
  ]
}

// In this particular case, a native time series collection
// is also a good option to consider
db.createCollection("temperatures", {
  timeseries: { timeField: "timestamp", granularity: "hours" }
})

When to use separate collections:

When NOT to use this pattern:

• Data is genuinely independent: Products exist separately from orders; don't embed full product catalog in every order.
• Frequent independent updates: If customer email changes shouldn't update all historical orders (it shouldn't).
• Data is accessed in different contexts: Same address entity used for shipping, billing, user profile—keep it separate.
• Regulatory requirements: Some industries require normalized data for audit trails.

Verify with

// Count your collections
for (const d of db.adminCommand({ listDatabases: 1 }).databases) {
  const colls = db.getSiblingDB(d.name).getCollectionNames().length
  print(`${d.name}: ${colls} collections`)
}
// Count alone is not sufficient: combine with access and index/storage evidence

// Check if collections are always accessed together
// If orders always needs customer, items, addresses
// → they should be embedded
db.system.profile.aggregate([
  { $match: { op: "query" } },
  { $group: { _id: "$ns", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
])
// Collections with similar access patterns should be combined

Atlas Schema Suggestions flags: "Reduce number of collections"

Reference: Reduce the Number of Collections

Avoid Unnecessary Indexes

Every index has a write cost. On insert, update, and delete, MongoDB must update ALL indexes on the collection. Unused or redundant indexes slow down writes with no query benefit, and consume RAM in the WiredTiger cache competing with working set data. Atlas Performance Advisor specifically flags "Redundant Index" and "Unused Index".

Incorrect (indexes created "just in case"):

// Creating indexes speculatively without query evidence
db.orders.createIndex({ status: 1 })          // never queried by status alone
db.orders.createIndex({ status: 1, date: 1 }) // already have {status:1,date:1,amount:1}
db.orders.createIndex({ region: 1 })           // added during development, never used

// Problems:
// 1. Every insert/update/delete must update ALL indexes
// 2. Redundant {status: 1} is fully covered by {status: 1, date: 1, amount: 1}
// 3. Unused indexes waste RAM in WiredTiger cache
// 4. Atlas Performance Advisor flags these but they're never cleaned up

Correct (audit-driven index management):

// Only create indexes that serve real query patterns
// Audit before adding new indexes:
db.orders.aggregate([{ $indexStats: {} }])

// Review index list regularly
db.orders.getIndexes()

// A compound index {a: 1, b: 1} makes a single-field index {a: 1} redundant
// — the compound index serves all queries that {a: 1} alone serves
db.col.createIndex({ a: 1 })         // drop this — redundant
db.col.createIndex({ a: 1, b: 1 })   // keep this

// NOTE: Different leading field is NOT redundant
db.col.createIndex({ a: 1, b: 1 })
db.col.createIndex({ b: 1 })         // NOT covered by above — keep

Safe removal process (hide → monitor → drop):

// Never drop an index directly in production

// Step 1: Hide the index (invisible to query planner but stays on disk)
db.orders.hideIndex("status_1")

// Step 2: Monitor for a full workload cycle (days/weeks)
// If queries degrade, unhide immediately:
// db.orders.unhideIndex("status_1")

// Step 3: Once confident, permanently drop
db.orders.dropIndexes(["status_1"])

Atlas automatically flags:

• "Redundant Index" — a prefix of an existing compound index
• "Unused Index" — zero query usage in the observed window

When NOT to use this pattern:

• New collections with planned queries: If you know queries are coming, pre-creating indexes is fine.
• Indexes for rare but critical operations: A backup or compliance query that runs monthly may show low usage but is still needed.
• TTL indexes: These serve data lifecycle purposes even if not used for queries.

Verify with

// Find indexes with zero query usage since last restart
db.orders.aggregate([{ $indexStats: {} }])
// Look for: accesses.ops: 0
// Example output:
// { name: "status_1", accesses: { ops: 0, since: ISODate("...") }, ... }
// An accesses.ops: 0 after a representative workload period means the index
// is never used by any query

// Check total index count and sizes
db.orders.stats().indexSizes
// Large number of indexes or large total index size signals audit opportunity

// Find redundant indexes (prefix subsets)
for (const idx of db.orders.getIndexes()) {
  print(`${idx.name}: ${JSON.stringify(idx.key)}`)
}
// Compare index key prefixes — if {a:1} exists alongside {a:1,b:1}, the former is redundant

Reference: Remove Unnecessary Indexes

Embrace the Document Model

Don't recreate SQL tables one-to-one in MongoDB. The document model is designed to store related data together when it is read and updated together. Naively copying relational boundaries often increases application-side joins and coordination logic.

Incorrect (SQL patterns in MongoDB):

Mirroring a relational schema 1:1 — e.g. separate customers, addresses, phones, and preferences collections linked by customerId — requires four queries and four index lookups to load one customer profile, plus application-side joining. Updates may require cross-collection coordination or transactions.

Correct (rich document model):

// Customer document contains everything about the customer
// All data retrieved in single read, updated atomically
{
  _id: "cust123",
  name: "Alice Smith",
  email: "alice@example.com",
  addresses: [
    { type: "home", street: "123 Main", city: "Boston", zip: "02101" },
    { type: "work", street: "456 Oak", city: "Boston", zip: "02102" }
  ],
  phones: [
    { type: "mobile", number: "555-1234" },
    { type: "work", number: "555-5678" }
  ],
  preferences: {
    newsletter: true,
    theme: "dark",
    language: "en"
  },
  createdAt: ISODate("2024-01-01")
}

// Single query loads complete customer - 1 round-trip
db.customers.findOne({ _id: "cust123" })

// Atomic update - no transaction needed
db.customers.updateOne(
  { _id: "cust123" },
  { $push: { addresses: newAddress }, $set: { "preferences.theme": "light" } }
)

Common tradeoffs:

When migrating from SQL:

1. Don't convert tables 1:1 to collections 2. Identify which tables are always joined together 3. Denormalize those joins into single documents 4. Keep separate only what's accessed separately

When NOT to use this pattern:

• Genuinely independent data: If addresses are shared across users or accessed independently, keep them separate.
• Unbounded relationships: User with 10,000 orders should NOT embed all orders.
• Regulatory requirements: Some compliance rules require normalized audit trails.

Verify with

// Count your collections vs expected entities
for (const d of db.adminCommand({ listDatabases: 1 }).databases) {
  const colls = db.getSiblingDB(d.name).getCollectionNames().length
  print(`${d.name}: ${colls} collections`)
}
// Collection count alone is not enough evidence; inspect query/access patterns too

// Check for SQL-style foreign key patterns
db.addresses.aggregate([
  { $group: { _id: "$customerId", count: { $sum: 1 } } },
  { $match: { count: { $gt: 0 } } }
]).itcount()
// If addresses always belong to customers, they should be embedded

Reference: Schema Design Process

Keep Documents Small

MongoDB documents cannot exceed 16 megabytes. This is a hard BSON limit, not a guideline — writes fail once a document reaches it.

However, practical documents should be much smaller than 16MB. As a rule of thumb, aim for documents under 1MB. Smaller documents mean:

• Better working-set efficiency — more documents fit in the WiredTiger cache.
• Faster reads and writes — less data copied, serialized, and transferred per operation.
• Lower replication overhead — smaller oplog entries replicate faster.
• Room to grow — a document well under the limit won't surprise you after a year of appended data.

The 16MB ceiling is a safety net, not a design target.

How documents get too large

1. Unbounded arrays — e.g. an activityLog array receiving entries on every user action: 100,000 events × ~150 bytes ≈ 15MB, growing until writes are rejected. 2. Large bounded arrays — even a bounded comments array (5,000 items × ~500 bytes = 2.5MB) is expensive: each $push rewrites the growing document, and a multikey index fans out to one entry per element. 3. Bloated documents with cold fields — MongoDB reads full documents, even when queries only need a few fields. A product document carrying name and price (~18 bytes, frequently needed) alongside description (~5KB), full specs (~10KB), base64 images (~500KB), reviews (~100KB), and price history (~50KB) can reach ~665KB. Hot-path queries still load the entire document into cache, reducing working-set density. Even projecting a small field set (e.g. db.products.find({}, {name: 1, price: 1})) still reads the full document from storage. 4. Large embedded binary — a BinData PDF attachment of 10MB+; additional attachments push the document past the limit. 5. Deeply nested objects — a configuration document with 100+ nesting levels where metadata and keys alone approach 16MB.

Solution 1: move unbounded or large data to a separate collection

Keep the parent document small. Store children in their own collection with a reference field and a compound index for efficient queries.

// Parent stays lean
{ _id: "user123", name: "Alice", activityCount: 48210, lastActivity: ISODate("...") }

// Children in separate collection with efficient index
// Index: { userId: 1, ts: -1 }
{ userId: "user123", action: "login", ts: ISODate("...") }

For large binary blobs, use GridFS for in-database storage, or — often more efficient — store them in external object storage and keep only a reference in MongoDB.

Solution 2: split hot and cold fields (Subset Pattern)

Keep frequently-accessed (hot) data in the main document; store rarely-accessed (cold) data in a separate collection. This dramatically improves cache density for hot-path queries.

Incorrect (all data in one document): A movie document with all 10,000 reviews embedded (~1MB of cold data alongside ~1KB of hot data like title, rating, plot) means every page load pulls ~1MB into RAM. Most page views only need title + rating + plot, so this reduces how many movies fit in cache (e.g. 1GB RAM ≈ 1,000 movies instead of ~1,000,000 if only hot data were loaded).

Correct (subset pattern): The movie document (~2KB) contains only hot fields: title, year, rating, plot, reviewStats (count, avgRating, distribution), and a bounded featuredReviews array (top 5 only, ~500 bytes). Full reviews live in a separate reviews collection with movieId reference, loaded only when the user clicks "Show all reviews."

Similarly, a product document should keep only hot fields in the main document (~500 bytes): name, price, thumbnail URL, avgRating, reviewCount, inStock. Move cold data to separate collections — products_details (description, fullSpecs), products_images (images array), products_reviews (paginated reviews).

How to identify hot vs cold data:

Maintaining an embedded subset:

// When a new review is added:
// 1. Insert full review into reviews collection
db.reviews.insertOne({ movieId: "movie123", user: "newUser", rating: 5, text: "Amazing!", date: new Date(), helpful: 0 })

// 2. Update movie stats
db.movies.updateOne(
  { _id: "movie123" },
  { $inc: { "reviewStats.count": 1, "reviewStats.distribution.5": 1 } }
)

// 3. Periodically refresh featured reviews (background job)
const topReviews = db.reviews.find({ movieId: "movie123" }).sort({ helpful: -1 }).limit(5).toArray()
db.movies.updateOne({ _id: "movie123" }, { $set: { featuredReviews: topReviews } })

For arrays, atomic $slice keeps the embedded subset bounded without a background job:

db.posts.updateOne(
  { _id: "post123" },
  {
    $push: {
      recentComments: {
        $each: [newComment],
        $slice: -20,
        $sort: { ts: -1 }
      }
    },
    $inc: { commentCount: 1 }
  }
)
// Also insert into overflow comments collection
db.comments.insertOne({ postId: "post123", ...newComment })

Solution 3: projection (when you can't refactor)

// Only transfers ~500 bytes instead of 665KB over the network
db.products.find(
  { category: "electronics" },
  { name: 1, price: 1, thumbnail: 1 }
)

Projection reduces network transfer but still loads full documents into memory unless the query is fully covered by an index. For real working-set reduction, split hot and cold data into separate collections.

Prevention strategies

// 1. Schema validation with array limits
db.createCollection("users", {
  validator: {
    $jsonSchema: {
      properties: {
        addresses: { maxItems: 10 },
        tags: { maxItems: 100 }
      }
    }
  }
})
// (See fundamental-schema-validation.md for full validation guidance).

// 2. Application-level checks before write
const doc = await db.users.findOne({ _id: userId })
const currentSize = BSON.calculateObjectSize(doc)
if (currentSize > 200 * 1024) {  // 200KB warning — well before trouble
  logger.warn("Document size exceeding recommended threshold")
}

// 3. Use $slice to cap arrays
db.users.updateOne(
  { _id: userId },
  {
    $push: {
      activityLog: {
        $each: [newActivity],
        $slice: -1000  // Keep only last 1000
      }
    }
  }
)

Workload signals

When keeping data together is fine

• Small, bounded arrays — tags (max 20), roles (max 5), addresses (max 10) with a hard limit.
• Write-once arrays — built once and never modified; size still affects working set.
• Arrays of primitives — tags: ["a", "b", "c"] is much cheaper than arrays of objects.
• Small collections that fit in RAM — if your entire collection is <1GB, document size matters less.
• Always need all data — if every access pattern truly needs the full document, splitting adds overhead.

Verify with

// Find largest documents in collection
db.collection.aggregate([
  { $project: { size: { $bsonSize: "$$ROOT" } } },
  { $sort: { size: -1 } },
  { $limit: 10 }
])

// Check specific field sizes to find bloat
db.collection.aggregate([
  { $project: {
    total: { $bsonSize: "$$ROOT" },
    activitySize: { $bsonSize: { $ifNull: ["$activityLog", []] } },
    profileSize: { $bsonSize: { $ifNull: ["$profile", {}] } }
  }}
])

// Find documents with large arrays
db.collection.aggregate([
  { $project: {
    size: { $bsonSize: "$$ROOT" },
    arrayLen: { $size: { $ifNull: ["$myArray", []] } }
  }},
  { $match: { arrayLen: { $gt: 100 } } },
  { $sort: { arrayLen: -1 } },
  { $limit: 10 }
])

// Find documents with hot/cold imbalance
db.collection.aggregate([
  { $project: {
    totalSize: { $bsonSize: "$$ROOT" },
    coldSize: { $bsonSize: { $ifNull: ["$reviews", []] } },
    hotSize: { $subtract: [
      { $bsonSize: "$$ROOT" },
      { $bsonSize: { $ifNull: ["$reviews", []] } }
    ]}
  }},
  { $match: {
    $expr: { $gt: ["$coldSize", { $multiply: ["$hotSize", 10] }] }
  }},
  { $limit: 10 }
])

// Check working set vs RAM
db.serverStatus().wiredTiger.cache
// "bytes currently in the cache" vs "maximum bytes configured"

Atlas Schema Suggestions flags: "Array field may grow without bound", "Document size exceeds recommended limit"

References:

• BSON Document Size Limit
• Avoid Unbounded Arrays
• Reduce Bloated Documents

Embed vs Reference Decision Framework

This is one of the most important schema decisions you'll make. Choose embedding or referencing based on access patterns, not just entity relationships.

Embed when:

• Data is always accessed together (1:1 or 1:few relationships)
• Child data doesn't make sense without parent
• Updates to both happen atomically
• Child array is clearly bounded by product constraints

Reference when:

• Data is accessed independently
• Many-to-many relationships exist
• Child data is large relative to the parent or array growth is unbounded
• Different update frequencies

Decision Matrix:

---

One-to-One: embed in the parent document

Embed one-to-one related data directly in the parent when it is consistently co-accessed. Keeping it in one document eliminates a round-trip and guarantees atomicity.

Incorrect (separate collections for 1:1 data): Storing user accounts and profiles in separate collections when they are always accessed together requires two queries per lookup, two index lookups, and risks orphaned records.

Correct (embedded):

{
  _id: "user123",
  email: "alice@example.com",
  createdAt: ISODate("2024-01-01"),
  profile: {
    name: "Alice Smith",
    avatar: "https://cdn.example.com/alice.jpg",
    bio: "Developer building cool things"
  }
}

// Single query, atomic updates
db.users.updateOne(
  { _id: "user123" },
  { $set: { "profile.name": "Alice Johnson" } }
)

Use subdocuments to logically group related fields — e.g. auth (passwordHash, lastLogin), profile (name, avatar), settings (theme, notifications) — all 1:1 data, logically organized without separate collections.

Common 1:1 relationships to embed: User/Profile, Country/Capital, Building/Address, Order/ShippingAddress, Product/Dimensions.

When NOT to embed 1:1:

• Data accessed independently (profile page separate from auth operations)
• Different security requirements (auth vs profile)
• Extreme size difference (embedded doc >10KB, parent <1KB)
• Different update frequencies (profile hourly, auth rarely)

---

One-to-Few: embed bounded arrays

Embed bounded, small arrays directly in the parent document. When a parent has a limited number of children usually accessed together, embedding keeps data in one read path.

Incorrect (separate collection for few items):

// Addresses in separate collection — user typically has 1-3
{ userId: "user123", type: "home", street: "123 Main", city: "Boston" }
// Requires $lookup for ~2 addresses, orphan risk on user delete

Correct (embedded array):

{
  _id: "user123",
  name: "Alice Smith",
  addresses: [
    { type: "home", street: "123 Main St", city: "Boston", state: "MA", zip: "02101" },
    { type: "work", street: "456 Oak Ave", city: "Boston", state: "MA", zip: "02102" }
  ]
}

// Add address atomically
db.users.updateOne(
  { _id: "user123" },
  { $push: { addresses: { type: "vacation", street: "789 Beach", city: "Miami" } } }
)

// Update specific address
db.users.updateOne(
  { _id: "user123", "addresses.type": "home" },
  { $set: { "addresses.$.city": "Cambridge" } }
)

Common one-to-few: User/Addresses (1-5), User/PhoneNumbers (1-3), Product/Variants (3-10), Author/PenNames (1-3), Order/LineItems (1-50).

Enforce bounds with schema validation:

db.createCollection("users", {
  validator: {
    $jsonSchema: {
      properties: {
        addresses: {
          bsonType: "array",
          maxItems: 10,
          items: {
            bsonType: "object",
            required: ["city"],
            properties: {
              type: { enum: ["home", "work", "billing", "shipping"] },
              city: { bsonType: "string" }
            }
          }
        }
      }
    }
  }
})

(See fundamental-schema-validation.md for full validation guidance).

When NOT to embed arrays:

• Unbounded growth (comments, orders, events) — use separate collection
• Independent access (addresses queried without user context)
• Large child documents relative to parent
• Steadily growing array size approaching unbounded behavior

---

One-to-Many: reference in child documents

Use references when the "many" side is unbounded or frequently accessed independently. Store the parent's ID in each child document with an index on that field.

Incorrect (embedding unbounded arrays): Embedding all 10,000+ books inside a publisher document means adding one book rewrites the entire large document, eventually exceeding 16MB.

Correct (reference in children):

// Publisher stays small and fixed-size
{ _id: "oreilly", name: "O'Reilly Media", founded: 1978, bookCount: 3500 }

// Each book references publisher; index on { publisherId: 1 }
{ _id: "book001", title: "New MongoDB Book", publisherId: "oreilly" }

// Efficient indexed queries
db.books.find({ publisherId: "oreilly" })

// $lookup when you need details from both sides
db.books.aggregate([
  { $match: { publisherId: "oreilly" } },
  { $lookup: {
    from: "publishers",
    localField: "publisherId",
    foreignField: "_id",
    as: "publisher"
  }},
  { $unwind: "$publisher" }
])

Hybrid with subset: Embed a bounded subset (e.g. top 5 featured books with _id, title, isbn) in the publisher for display without $lookup. "View all books" queries the books collection.

Keep denormalized counts in sync:

db.books.insertOne({ title: "New Book", publisherId: "oreilly" })
db.publishers.updateOne({ _id: "oreilly" }, { $inc: { bookCount: 1 } })

When to reference: Unbounded children (Publisher→Books), large child documents (User→Orders), independent queries (Department→Employees), different lifecycles (Author→Articles).

When NOT to reference: Bounded small arrays (User's 3 addresses), always accessed together (Order→LineItems), never queried without parent.

---

Many-to-Many: choose a primary query direction

Many-to-many relationships require choosing a primary query direction. Unlike SQL's join tables, MongoDB favors denormalization toward your most common query pattern.

Incorrect (SQL-style junction table):

// 3 collections, always need joins
// students: { _id, name }  /  classes: { _id, name }  /  enrollments: { studentId, classId }
// Every query requires aggregation with $lookup

Correct (embed in primary query direction):

Embed references on the side you query most. If you primarily query "which classes is this student in," embed class summaries in the student. For the reverse, embed student summaries in the class.

Bidirectional embedding (when both directions are common):

// Book with author summaries
{
  _id: "book001",
  title: "Cell Biology",
  authors: [
    { authorId: "author124", name: "Ellie Smith" },
    { authorId: "author381", name: "John Palmer" }
  ]
}

// Author with book summaries
{
  _id: "author124",
  name: "Ellie Smith",
  books: [
    { bookId: "book001", title: "Cell Biology" },
    { bookId: "book042", title: "Molecular Biology" }
  ]
}
// Trade-off: data duplication, but fast queries in both directions

Reference-only (for large cardinality):

// Product stores category IDs (small array per product)
{ _id: "prod123", name: "Laptop", categoryIds: ["cat1", "cat2", "cat3"] }
// Category has no back-reference array (avoid huge arrays)
{ _id: "cat1", name: "Electronics" }
// Products in a category: db.products.find({ categoryIds: "cat1" })

Choosing strategy:

Maintaining bidirectional data — use transactions for atomicity:

const session = client.startSession()
session.withTransaction(async () => {
  await db.students.updateOne(
    { _id: "student1" },
    { $push: { classes: { classId: "class101", name: "Database Systems" } } },
    { session }
  )
  await db.classes.updateOne(
    { _id: "class101" },
    { $push: { students: { studentId: "student1", name: "Alice Smith" } } },
    { session }
  )
})

---

Tree and hierarchical data

Hierarchical data requires choosing a tree pattern based on your primary operations. MongoDB offers multiple patterns, each with different tradeoffs.

Common hierarchical data: Category trees, org charts, file/folder structures, comment threads, geographic hierarchies.

Pattern 1: Parent References

Best for: Finding parent, updating parent, simple child listing.

{ _id: "MongoDB", parent: "Databases" }
{ _id: "Databases", parent: "Programming" }
{ _id: "Programming", parent: null }

db.categories.createIndex({ parent: 1 })
db.categories.find({ parent: "Databases" })  // immediate children

Con: Finding all descendants requires recursive queries or $graphLookup.

Pattern 2: Child References

Best for: Finding children, graph-like structures.

{ _id: "Databases", children: ["MongoDB", "PostgreSQL", "MySQL"] }

Con: Finding ancestors requires recursion; array updates on every child add/remove.

Pattern 3: Array of Ancestors

Best for: Breadcrumb navigation, ancestor and descendant lookups.

{ _id: "MongoDB", parent: "Databases", ancestors: ["Programming", "Databases"] }
{ _id: "Atlas", parent: "MongoDB", ancestors: ["Programming", "Databases", "MongoDB"] }

db.categories.createIndex({ ancestors: 1 })
db.categories.find({ ancestors: "Databases" })  // all descendants

Including a parent field enables $graphLookup traversal without application-side recursion.

Pattern 4: Materialized Paths

Best for: Subtree queries, regex-based lookups, hierarchy sorting.

{ _id: "MongoDB", path: ",Programming,Databases,MongoDB," }
{ _id: "Atlas", path: ",Programming,Databases,MongoDB,Atlas," }

db.categories.createIndex({ path: 1 })
db.categories.find({ path: /^,Programming,Databases,MongoDB,/ })  // all descendants
db.categories.find({}).sort({ path: 1 })  // hierarchy display order

Tree pattern comparison

Recommended by use case: Category breadcrumbs → Array of Ancestors. File browser → Parent References. Org chart reporting → Materialized Paths. Comment threads → Parent References.

---

When NOT to embed (summary)

• Unbounded growth: Comments, logs, events — separate collection.
• Large child documents: If each child is large relative to the parent, references are usually safer.
• Independent access: If you ever query child without parent, reference.
• Different lifecycles: If child data is archived/deleted separately.
• Graph-like data: Multiple parents → use $graphLookup or a graph database.

Verify with

// Check document sizes for embedded collections
db.collection.aggregate([
  { $project: {
    size: { $bsonSize: "$$ROOT" },
    arrayLen: { $size: { $ifNull: ["$items", []] } }
  }},
  { $match: { size: { $gt: 1000000 } } }
])
// Large documents may indicate embedding that should be referencing

// Check embedded array sizes (one-to-few validation)
db.users.aggregate([
  { $project: { addressCount: { $size: { $ifNull: ["$addresses", []] } } } },
  { $group: { _id: null, avg: { $avg: "$addressCount" }, max: { $max: "$addressCount" } } }
])
// If max keeps growing, consider a separate collection

// Check for orphaned references (1:1 that should be embedded)
db.profiles.aggregate([
  { $lookup: { from: "users", localField: "userId", foreignField: "_id", as: "user" } },
  { $match: { user: { $size: 0 } } }
])
// Orphans suggest 1:1 data should be embedded

// Check for missing indexes on reference fields
db.books.getIndexes()
// Must have index on publisherId for efficient child lookups

// Verify bidirectional many-to-many consistency
db.students.aggregate([
  { $unwind: "$classes" },
  { $lookup: {
    from: "classes",
    let: { sid: "$_id", cid: "$classes.classId" },
    pipeline: [
      { $match: { $expr: { $eq: ["$_id", "$$cid"] } } },
      { $match: { $expr: { $in: ["$$sid", "$students.studentId"] } } }
    ],
    as: "match"
  }},
  { $match: { match: { $size: 0 } } }
])
// Mismatches indicate inconsistent bidirectional data

// Check tree consistency (no orphaned nodes)
db.categories.aggregate([
  { $match: { parent: { $ne: null } } },
  { $lookup: { from: "categories", localField: "parent", foreignField: "_id", as: "parentDoc" } },
  { $match: { parentDoc: { $size: 0 } } },
  { $count: "orphanedNodes" }
])

References:

• Embedding vs Referencing
• Model One-to-One Relationships
• Model One-to-Many Relationships with Embedded Documents
• Model One-to-Many Relationships with References
• Model Many-to-Many Relationships
• Model Tree Structures

Use Schema Validation

Enforce document structure with MongoDB's built-in JSON Schema validation. Catch invalid data before it corrupts your database, not after you've shipped 10,000 malformed documents to production. Schema validation is your last line of defense when application bugs slip through.

Incorrect (no validation):

Without validation, any document shape is accepted: an email field can contain a non-email string, an age field can hold a string instead of a number, and required fields like email can be omitted entirely. These invalid documents are discovered only when downstream consumers crash or return wrong data — often months later.

Correct (schema validation):

// Create collection with validation rules
db.createCollection("users", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["email", "name"],
      properties: {
        email: {
          bsonType: "string",
          pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$",
          description: "must be a valid email address"
        },
        name: {
          bsonType: "string",
          minLength: 1,
          maxLength: 100,
          description: "must be 1-100 characters"
        },
        age: {
          bsonType: "int",
          minimum: 0,
          maximum: 150,
          description: "must be integer 0-150"
        },
        status: {
          enum: ["active", "inactive", "pending"],
          description: "must be one of: active, inactive, pending"
        },
        addresses: {
          bsonType: "array",
          maxItems: 10,  // Prevent unbounded arrays
          items: {
            bsonType: "object",
            required: ["city"],
            properties: {
              street: { bsonType: "string" },
              city: { bsonType: "string" },
              zip: { bsonType: "string", pattern: "^[0-9]{5}$" }
            }
          }
        }
      }
    }
  },
  validationLevel: "strict",
  validationAction: "error"
})

// Invalid inserts now fail immediately with clear error
db.users.insertOne({ email: "not-an-email" })
// Error: Document failed validation:
// "email" does not match pattern, "name" is required

Validation levels and actions:

Add validation to existing collection:

// Start with moderate + warn to discover violations
db.runCommand({
  collMod: "users",
  validator: { $jsonSchema: {...} },
  validationLevel: "moderate",  // Don't break existing invalid docs
  validationAction: "warn"       // Log violations, don't block
})

// Check for violations using the actual validator shape
const info = db.getCollectionInfos({ name: "users" })[0]
const validator = info?.options?.validator
db.users.find({ $nor: [validator] })

// Then switch to strict + error
db.runCommand({
  collMod: "users",
  validationLevel: "strict",
  validationAction: "error"
})

When NOT to use this pattern:

• Rapid prototyping: Skip validation during early development, add before production.
• Schema-per-document designs: Some collections intentionally store varied document shapes.
• Log/event collections: High-write collections where validation overhead matters.

Verify with

// Read current validator and validation settings
const info = db.getCollectionInfos({ name: "users" })[0]
printjson({
  validationLevel: info?.options?.validationLevel,
  validationAction: info?.options?.validationAction,
  validator: info?.options?.validator
})

// Primary compliance check: find documents that do NOT match validator
const validator = info?.options?.validator
db.users.find({ $nor: [validator] })

Reference: Schema Validation

Approximation Pattern

Intentionally store approximate values to reduce write load when exact real-time counts are not required. High-frequency counters (page views, trending scores, social media counters) that increment by +1 per event can create expensive per-event writes. The approximation pattern batches these increments, trading staleness for dramatically lower write volume.

Incorrect (write to database on every event):

// Page view counter - writes to MongoDB on every single view
function recordPageView(articleId) {
  db.articles.updateOne(
    { _id: articleId },
    {
      $inc: { viewCount: 1 },
      $set: { lastViewedAt: new Date() }
    }
  )
}
// 1M page views/day = 1M database writes/day
// High write load for a counter that doesn't need real-time accuracy

Correct (batch writes with threshold):

The document stores an approximate count plus a sync timestamp. The application tracks counts in local memory (e.g. a Map keyed by article ID) and writes to the database only when the local counter crosses a threshold (e.g. every 100 views). At threshold=100 this yields ~100× fewer database writes.

The document includes viewCount (approximate — may lag by up to one threshold) and lastSyncedAt. When the local counter reaches the threshold, the application issues a single $inc by the threshold amount and updates lastSyncedAt. Unsynced local increments are lost on application restart.

Tradeoffs:

Difference from Computed Pattern:

• Computed Pattern: pre-computes expensive aggregations, stores exact results
• Approximation Pattern: intentionally stores inexact values to reduce write frequency

Use Approximation when staleness is acceptable. Use Computed when exact values are needed but recalculating each time is too expensive.

When NOT to use this pattern:

• Financial amounts, inventory counts: Exact values required — approximation is unacceptable.
• Low-frequency updates: If counter changes rarely, approximation adds complexity without benefit.
• Regulatory/audit requirements: When exact counts are mandated.

Verify with

// Check write frequency on counter fields
db.setProfilingLevel(1, { slowms: 0 })
db.system.profile.find({
  "command.update": "articles",
  "command.updates.u.$inc.viewCount": { $exists: true }
}).count()
// High count relative to read count suggests approximation would help

// Compare counter staleness
db.articles.aggregate([
  { $project: {
    title: 1,
    viewCount: 1,
    lastSyncedAt: 1,
    staleness: { $subtract: ["$$NOW", "$lastSyncedAt"] }
  }},
  { $sort: { staleness: -1 } },
  { $limit: 10 }
])
// Verify staleness is within acceptable bounds for your use case

Reference: Use the Approximation Pattern

Use Archive Pattern for Historical Data

Storing old data alongside recent data degrades performance. As collections grow with historical data that's rarely accessed, queries slow down, indexes bloat, and working set exceeds RAM. The archive pattern moves old data to separate storage while keeping your active collection fast.

Incorrect (all data in one collection):

A sales collection with 5 years of data (50M documents) where only the recent 6 months are actively queried suffers from: indexes covering the full 50M documents when only ~1M are relevant, working set including old data pages, backups including rarely-accessed history, and hot-tier storage costs for data that could be cold.

Correct (archive old data separately):

// Step 1: Define archive threshold (older than 6 months)
const sixMonthsAgo = new Date()
sixMonthsAgo.setMonth(sixMonthsAgo.getMonth() - 6)

// Step 2: Move old data to archive collection using $merge
db.sales.aggregate([
  { $match: { date: { $lt: sixMonthsAgo } } },
  { $merge: {
      into: "sales_archive",
      on: "_id",
      whenMatched: "keepExisting",  // Don't overwrite if re-run
      whenNotMatched: "insert"
    }
  }
])

// Step 3: Delete archived data from active collection
db.sales.deleteMany({ date: { $lt: sixMonthsAgo } })

// Result:
// - sales: Recent data, fast queries, small indexes
// - sales_archive: Historical data, rarely queried

Archive storage options (best to worst for cost/performance):

1. External file storage (S3, cloud object storage) — Best for compliance and long-term retention at lowest cost. Export to JSON/BSON, store in S3, query via Atlas Data Federation when needed. 2. Separate, cheaper cluster — Best for occasional historical queries. Replicate to a lower-tier Atlas cluster at reduced cost. 3. Separate collection on same cluster — Best for simple implementation with frequent historical access. As shown above with sales_archive, but still uses the same storage tier. 4. Atlas Online Archive (Atlas only) — MongoDB manages automatic movement to cloud object storage; query via Federated Database Instance.

Design tips for archivable schemas:

// TIP 1: Use embedded data model for archives
// Archived data must be self-contained

// BAD: References that may be deleted
{
  _id: "order123",
  customerId: "cust456",  // Customer may be deleted
  productIds: ["prod1", "prod2"]  // Products may change
}

// GOOD: Embedded snapshot of related data
{
  _id: "order123",
  customer: {
    _id: "cust456",
    name: "Jane Doe",
    email: "jane@example.com"
  },
  products: [
    { _id: "prod1", name: "Widget", price: 29.99 },
    { _id: "prod2", name: "Gadget", price: 49.99 }
  ],
  date: ISODate("2020-01-15")
}

// TIP 2: Store age in a single, indexable field
// Makes archive queries efficient
{
  date: ISODate("2020-01-15"),  // Single field for age
  // NOT: { year: 2020, month: 1, day: 15 }
}

// TIP 3: Handle "never expire" documents
{
  date: ISODate("2025-01-15"),
  retentionPolicy: "permanent"  // Or use far-future date
}

// Archive query excludes permanent records:
db.sales.aggregate([
  { $match: {
      date: { $lt: fiveYearsAgo },
      retentionPolicy: { $ne: "permanent" }
    }
  },
  { $merge: { into: "sales_archive" } }
])

Automated archival with scheduling:

Create a script (run via cron, Atlas Triggers, or an application scheduler) that:

1. Counts documents older than the cutoff date (excluding those with retentionPolicy: "permanent"). 2. Processes in batches (e.g. 10,000 IDs at a time) to avoid long-running operations: fetch a batch of _id values, pipe them through an aggregation with $match and $merge into the archive collection, then deleteMany the batch from the active collection. 3. Logs progress after each batch.

This reuses the same $merge-based archival shown above but throttles work to avoid overloading the cluster.

Atlas Online Archive (Atlas only):

Atlas Online Archive automatically tiers data to MongoDB-managed cloud object storage based on a date-field rule (e.g. archive after 365 days). Archived data is queried transparently via a Federated Database Instance — slightly slower but much cheaper. No application code changes are required.

When NOT to use archive pattern:

• Small datasets: If total data fits comfortably in RAM, archiving adds complexity without benefit.
• Uniform access patterns: If old and new data are queried equally.
• Compliance requires instant access: If regulations require sub-second queries on all historical data.
• Already using TTL: If data should be deleted, not archived, use TTL indexes.

Verify with

// Analyze archive candidates
const cutoff = new Date()
cutoff.setFullYear(cutoff.getFullYear() - 5)

db.sales.aggregate([
  { $facet: {
      total: [{ $count: "count" }],
      old: [
        { $match: { date: { $lt: cutoff } } },
        { $count: "count" }
      ]
    }
  }
])
// If old documents are >30% of total, archiving can improve performance

Reference: Archive Pattern

Use Attribute Pattern for Sparse or Variable Fields

If documents have many optional fields, move them into a key-value array. This avoids dozens of sparse indexes and lets you query across attributes with a single multikey index.

Incorrect (separate field and index per optional attribute):

// Many optional fields - most are missing on any given document
{
  _id: 1,
  name: "Bottle",
  color: "red",
  size: "M",
  material: "glass",
  // 20+ other optional fields, varying per document
}

// One partial index per optional field — correct use of partialFilterExpression,
// but you end up maintaining dozens of indexes as attributes grow
db.items.createIndex({ color: 1 }, { partialFilterExpression: { color: { $exists: true } } })
db.items.createIndex({ size: 1 }, { partialFilterExpression: { size: { $exists: true } } })
db.items.createIndex({ material: 1 }, { partialFilterExpression: { material: { $exists: true } } })
// … repeated for every new attribute

Correct (attribute pattern):

// Store optional fields as key-value pairs
{
  _id: 1,
  name: "Bottle",
  attributes: [
    { k: "color", v: "red" },
    { k: "size", v: "M" },
    { k: "material", v: "glass" }
  ]
}

// Single multikey index for all attributes

db.items.createIndex({ "attributes.k": 1, "attributes.v": 1 })

// Query for color = red

db.items.find({
  attributes: { $elemMatch: { k: "color", v: "red" } }
})

When NOT to use this pattern:

• Fixed schema: If fields are stable and always present.
• Type-specific validation: If each field needs strict schema rules.
• Single-field queries only: A normal field may be simpler and faster.
• Atlas Search workloads: The { k, v } key-value structure cannot be mapped as

named fields in Atlas Search indexes. If you need full-text search on attribute values by key name, use static named fields instead.

Verify with

// Ensure queries use the multikey index

db.items.find({
  attributes: { $elemMatch: { k: "material", v: "glass" } }
}).explain("executionStats")

Reference: Attribute Pattern

Use Bucket Pattern to Group Related Data

Group a series of related items into bounded arrays within a single document. The bucket pattern separates long series of data into distinct objects, reducing document count and aligning storage with how data is actually consumed. This is especially useful when an application accesses data in fixed-size groups (e.g. pages).

For time-series data, prefer Time Series Collections, which apply bucketing automatically with built-in compression and indexing optimizations.

Incorrect (one document per event):

Storing one document per stock trade (e.g. { ticker, customerId, type, quantity, date }) means the application pages through trades using skip/limit, which degrades as offset grows. Each trade is a separate document and index entry.

Correct (bucket pattern - group by customer, bounded per page):

Each document holds up to N trades for one customer (e.g. 10 trades = one page). The _id encodes customer ID and the first trade’s epoch seconds (e.g. "123_1698349623"), with a count field and a history array of trade objects. One bucket equals one page of data — a regex on _id uses the default _id index with no extra index needed, and document count drops by up to the bucket-size factor.

Insert with atomic upsert:

// Insert a new trade into the correct bucket
db.trades.findOneAndUpdate(
  {
    "_id": /^123_/,            // Match buckets for this customer
    "count": { $lt: 10 }       // Only if bucket isn't full
  },
  {
    $push: {
      history: {
        type: "buy",
        ticker: "MSFT",
        qty: 42,
        date: ISODate("2023-11-02T11:43:10Z")
      }
    },
    $inc: { count: 1 },
    $setOnInsert: {
      _id: "123_1698939791",   // New bucket ID if upsert fires
      customerId: 123
    }
  },
  { upsert: true, sort: { _id: -1 } }
)
// If a bucket with room exists, the trade is pushed into it
// Otherwise a new bucket document is created
// Array is bounded — never exceeds 10 elements

Query patterns:

// Page 1 of trades for customer 123
db.trades.find({ _id: /^123_/ }).sort({ _id: 1 }).limit(1)

// Page N (e.g. page 10)
db.trades.find({ _id: /^123_/ }).sort({ _id: 1 }).skip(9).limit(1)

// Each returned document IS a page — no per-trade skip/limit needed

Choosing bucket boundaries:

When NOT to use this pattern:

• Time-series workloads: Use Time Series Collections instead — they handle bucketing, compression, and indexing automatically.
• Random single-item access: If you frequently query individual items by their own ID, buckets add unnecessary indirection.
• Low volume: If the total series per entity is small, the added complexity isn't worth it.
• Highly variable item sizes: Bucketing works best when items are roughly uniform in size so bucket documents stay predictable.

Verify with

// Check that bucket size matches expectations
db.trades.aggregate([
  { $group: {
    _id: null,
    avgCount: { $avg: "$count" },
    maxCount: { $max: "$count" },
    totalBuckets: { $sum: 1 }
  }}
])
// avgCount should approach your target bucket size
// maxCount should not exceed it

// Check average document size
db.trades.aggregate([
  { $project: { size: { $bsonSize: "$$ROOT" } } },
  { $group: { _id: null, avgSize: { $avg: "$size" } } }
])

Reference: Group Data with the Bucket Pattern

Use Computed Pattern for Expensive Calculations

Pre-calculate and store frequently-accessed computed values. If you're running the same aggregation on every page load, you're wasting CPU cycles. Store the result in the document and update it on write or via background job—trades write complexity for read speed.

Incorrect (calculate on every read):

// Movie with all screenings in separate collection
{ _id: "movie1", title: "The Matrix" }

// Screenings collection - thousands of records
{ movieId: "movie1", date: ISODate("..."), viewers: 344, revenue: 3440 }
{ movieId: "movie1", date: ISODate("..."), viewers: 256, revenue: 2560 }
// ... 10,000 screenings

// Movie page aggregates every time
db.screenings.aggregate([
  { $match: { movieId: "movie1" } },
  { $group: {
    _id: "$movieId",
    totalViewers: { $sum: "$viewers" },
    totalRevenue: { $sum: "$revenue" },
    screeningCount: { $sum: 1 }
  }}
])
// Repeated scans can add substantial read latency and CPU overhead
// 1M page views/day = 1M expensive aggregations

Correct (pre-computed values):

Store computed stats directly in the movie document: stats.totalViewers, stats.totalRevenue, stats.screeningCount, stats.avgViewersPerScreening, and stats.computedAt. The movie page reads a single document with no aggregation needed on the hot path.

Update strategies:

// Strategy 1: Update on write (low write volume)
// When new screening is added
db.screenings.insertOne({
  movieId: "movie1",
  viewers: 400,
  revenue: 4000
})

// Immediately update computed values
db.movies.updateOne(
  { _id: "movie1" },
  {
    $inc: {
      "stats.totalViewers": 400,
      "stats.totalRevenue": 4000,
      "stats.screeningCount": 1
    },
    $set: { "stats.computedAt": new Date() }
  }
)

// Strategy 2: Background job (high write volume)
// Run hourly/daily aggregation job
db.screenings.aggregate([
  { $group: {
    _id: "$movieId",
    totalViewers: { $sum: "$viewers" },
    totalRevenue: { $sum: "$revenue" },
    count: { $sum: 1 }
  }},
  { $merge: {
    into: "movies",
    on: "_id",
    whenMatched: [{
      $set: {
        "stats.totalViewers": "$$new.totalViewers",
        "stats.totalRevenue": "$$new.totalRevenue",
        "stats.screeningCount": "$$new.count",
        "stats.computedAt": "$$NOW"
      }
    }]
  }}
])

Common computed values:

Handling staleness:

Include a computedAt timestamp alongside the stats. Application code compares this timestamp against a freshness threshold (e.g. one hour) and triggers a refresh if the values are stale. Alternatively, surface the timestamp to users (e.g. “1,840,000 viewers — updated 1 hour ago”).

Windowed computations:

// Compute for time windows (rolling 30 days)
{
  _id: "movie1",
  stats: {
    allTime: { viewers: 1840000, revenue: 25880000 },
    last30Days: { viewers: 45000, revenue: 630000 },
    last7Days: { viewers: 12000, revenue: 168000 }
  }
}

// Background job updates rolling windows
db.screenings.aggregate([
  { $match: {
    movieId: "movie1",
    date: { $gte: thirtyDaysAgo }
  }},
  { $group: {
    _id: null,
    viewers: { $sum: "$viewers" },
    revenue: { $sum: "$revenue" }
  }}
])
// Then update movie.stats.last30Days

Consider on-demand materialized views:

When the computed results are best stored in a separate collection rather than embedded in the source documents, MongoDB's on-demand materialized views formalize this approach. An on-demand materialized view is an aggregation pipeline whose output is written to a separate collection using $merge or $out—the same mechanism shown in Strategy 2 above. The difference is conceptual: instead of updating a field on existing documents, you maintain a dedicated read-optimized collection that can be independently indexed. This is especially useful when:

• The computed data has a different shape or granularity than the source (e.g. monthly summaries from daily records).
• Multiple consumers need the pre-aggregated data, and a shared collection is cleaner than duplicating fields across documents.
• You want to index the computed results independently of the source collection.

On-demand materialized views are not automatically refreshed—you control when to re-run the pipeline, which gives you the same staleness trade-offs described above.

When NOT to use this pattern:

• Rarely accessed calculations: If stat is viewed once/day, compute on demand.
• High write frequency: If source data changes every second, update overhead may exceed read savings.
• Complex multi-collection joins: Some computations are too complex to maintain incrementally.
• Strong consistency required: Computed values may be slightly stale.

Verify with

// Find expensive aggregations that should be pre-computed
db.setProfilingLevel(1, { slowms: 100 }) // Disable afterwards
db.system.profile.find({
  "command.aggregate": { $exists: true },
  millis: { $gt: 100 }
}).sort({ millis: -1 })

// Check if same aggregation runs repeatedly
db.system.profile.aggregate([
  { $match: { "command.aggregate": { $exists: true } } },
  { $group: {
    _id: "$command.pipeline",
    count: { $sum: 1 },
    avgMs: { $avg: "$millis" }
  }},
  { $match: { count: { $gt: 100 } } }  // Repeated 100+ times
])
// High count + high avgMs = candidate for computed pattern

Reference: Computed Schema Pattern

Document Versioning Pattern

Store full document history in a separate `revisions` collection to enable reproducing historical state. This is different from schema versioning (which handles field migration)—document versioning stores complete snapshots of each change. Use it for insurance policies, legal documents, compliance audit trails, and any data where you must reproduce exact historical state.

Incorrect (overwrite history with no trail):

// Policy document — only current state exists
{
  _id: "POL-001",
  holder: "Jane Smith",
  premium: 450,
  coverage: "comprehensive",
  updatedAt: ISODate("2024-06-01")
}

// When premium changes, old value is lost forever
db.policies.updateOne(
  { _id: "POL-001" },
  { $set: { premium: 475, updatedAt: new Date() } }
)
// Previous premium of 450 is gone — no audit trail
// Cannot reproduce what the policy looked like on 2024-03-15
// Compliance audit fails: "show me the policy as of Q1"

Correct (current collection + revisions collection):

// currentPolicies collection — current state only (fast reads)
{
  _id: "POL-001",
  holder: "Jane Smith",
  premium: 450,
  coverage: "comprehensive",
  v: 3,
  updatedAt: ISODate("2024-06-01")
}

// policyRevisions collection — full history snapshots
{
  policyId: "POL-001",
  v: 2,
  snapshot: {
    holder: "Jane Smith",
    premium: 425,
    coverage: "basic",
    v: 2
  },
  changedAt: ISODate("2024-03-15")
}

Implementation:

async function updatePolicy(policyId, newData, session) {
  const current = await db.currentPolicies.findOne({ _id: policyId }, { session })

  await db.policyRevisions.insertOne({
    policyId: current._id,
    v: current.v,
    snapshot: { ...current },
    changedAt: new Date()
  }, { session })

  await db.currentPolicies.updateOne(
    { _id: policyId },
    { $set: { ...newData, v: current.v + 1, updatedAt: new Date() } },
    { session }
  )
}

async function getPolicyAtVersion(policyId, version) {
  if (version === 'current') {
    return db.currentPolicies.findOne({ _id: policyId })
  }
  const rev = await db.policyRevisions.findOne({ policyId, v: version })
  return rev?.snapshot
}

Using Transactions for Atomicity:

The updatePolicy function writes to two collections (inserting a revision and updating the current document). It may or may not be prudent to wrap the call in a multi-document transaction to guarantee both writes succeed or fail together, depending on the use case:

const session = client.startSession()
try {
  await session.withTransaction(async () => {
    await updatePolicy("POL-001", { premium: 475, coverage: "premium" }, session)
  })
} finally {
  await session.endSession()
}

Indexes:

db.policyRevisions.createIndex({ policyId: 1, v: -1 })
// Optional TTL for retention (e.g., 7 years)
db.policyRevisions.createIndex({ changedAt: 1 }, { expireAfterSeconds: 220752000 })

Difference from Schema Versioning:

When NOT to use this pattern:

• High-frequency updates: If documents change many times per second, use event sourcing instead.
• Approximate history is sufficient: If you only need to know "what changed" but not reproduce exact state.
• Unbounded revision growth without retention: Ensure you have a TTL or archival policy for the revisions collection.

Verify with

// Check revision collection growth
db.policyRevisions.aggregate([
  { $group: {
    _id: "$policyId",
    revisionCount: { $sum: 1 },
    oldestRevision: { $min: "$changedAt" },
    newestRevision: { $max: "$changedAt" }
  }},
  { $sort: { revisionCount: -1 } },
  { $limit: 10 }
])
// Monitor for documents with unexpectedly high revision counts

// Verify current docs have version field
db.currentPolicies.countDocuments({ v: { $exists: false } })
// Should be 0 — all documents need version tracking

// Check that revisions are consistent with current version
db.currentPolicies.aggregate([
  { $lookup: {
    from: "policyRevisions",
    localField: "_id",
    foreignField: "policyId",
    as: "revisions"
  }},
  { $project: {
    currentVersion: "$v",
    revisionCount: { $size: "$revisions" },
    maxRevisionVersion: { $max: "$revisions.v" }
  }},
  { $match: {
    $expr: { $ne: [{ $subtract: ["$currentVersion", 1] }, "$maxRevisionVersion"] }
  }}
])
// Finds documents where revision history has gaps

Reference: Keep a History of Document Versions

Use Extended Reference Pattern

Copy frequently-accessed fields from referenced documents into the parent. If you always display author name with articles, embed it. This eliminates $lookup for common queries while keeping the full data normalized—best of both worlds.

Incorrect (always $lookup for display data):

// Order references customer by ID only
{
  _id: "order123",
  customerId: "cust456",  // Customer reference by ID only
  items: [...],
  total: 299.99
}

// Every order list/display requires $lookup
db.orders.aggregate([
  { $match: { status: "pending" } },
  { $lookup: {
    from: "customers",
    localField: "customerId",
    foreignField: "_id",
    as: "customer"
  }},
  { $unwind: "$customer" }
])
// Repeated joins add avoidable work for a common list view

Correct (extended reference):

Embed frequently-needed customer fields directly in the order document: include a customer subdocument with _id (kept as a reference for full lookups), name, and email. The order list query returns customer display data without $lookup. Full customer data is still available via a targeted read to the customers collection when needed.

Keeping cached data in sync:

When the source field changes (e.g. customer name), update the source collection first, then update cached copies in the orders collection using updateMany on the embedded reference _id. This can be done synchronously or asynchronously via Change Streams / background jobs. For data that changes more often, add a cachedAt timestamp to the embedded subdocument so the application can refresh on read when the cache exceeds a staleness threshold.

What to cache (extend):

Alternative: Hybrid pattern with cache expiry:

Keep both a bare reference (customerId) and an optional cache subdocument (customerCache) with name, email, and cachedAt. On read, if the cache is missing or older than a threshold (e.g. one day), refresh it from the customers collection and write the updated cache back to the order.

When NOT to use this pattern:

• Frequently-changing data: If customer name changes daily, update overhead exceeds $lookup cost.
• Large cached payloads: Don't embed 50KB of author bio in every article.
• Sensitive data segregation: Don't copy PII into collections with different access controls.
• Writes >> Reads: If writes greatly outnumber reads, caching adds overhead.

Verify with

// Find $lookup-heavy aggregations in profile
db.setProfilingLevel(1, { slowms: 20 }) // Disable afterwards
db.system.profile.find({
  "command.aggregate": { $exists: true },
  "command.pipeline.$lookup": {
    $exists: true
  }
}).sort({ millis: -1 }).limit(10)

// Check how often lookups hit same collections
db.system.profile.aggregate([
  { $match: { "command.pipeline.$lookup": { $exists: true } } },
  { $project: { pipeline: "$command.pipeline" } },
  { $unwind: "$pipeline" },
  { $project: { lookup: { $getField: { field: { $literal: '$lookup' }, input: '$pipeline' } } } },
  { $match: { "lookup": { $exists: true } } },
  { $group: { _id: "$lookup.from", count: { $sum: 1 } } }
])
// High count = candidate for extended reference

Reference: Reduce $lookup Operations

Use Outlier Pattern for Exceptional Documents

Isolate atypical documents with large arrays to prevent them from degrading performance for typical queries. When a small subset of documents is much larger than the rest, those outliers can dominate memory, index, and query costs. Split overflow data into a separate collection and flag the document.

Problem scenario:

A typical book might have 50 customers in an embedded array, while a bestseller like Harry Potter accumulates 50,000 (~2.5MB). Queries return the full document, so the outlier dominates memory and network cost. A multikey index on that array produces 50,000 entries for a single document.

Correct (outlier pattern):

Typical documents keep their full embedded array and set hasExtras: false. Outlier documents cap the embedded array at a threshold (e.g. 50), set hasExtras: true, store a denormalized customerCount, and overflow remaining items into a separate collection in batched documents (e.g. { bookId, customers: [...], batch: 1, count: 950 }). Application code checks the hasExtras flag to decide whether to load overflow batches.

Implementation with threshold (example; tune per workload):

const CUSTOMER_THRESHOLD = 50

async function addCustomer(bookId, customerId) {
  // Try the normal case first: atomically add to the embedded array only if
  // the current customerCount is below the threshold (treat missing/null as 0).
  const result = await db.books.updateOne(
    {
      _id: bookId,
      $or: [
        { customerCount: { $lt: CUSTOMER_THRESHOLD } },
        { customerCount: { $exists: false } },
        { customerCount: null }
      ]
    },
    {
      $push: { customers: customerId },
      $inc: { customerCount: 1 }
    }
  )

  if (result.matchedCount > 0) {
    // Normal case succeeded - customer added to embedded array
    return
  }

  // Outlier case - add to overflow collection
  const lastBatchDoc = await db.book_customers_extra
    .find({ bookId: bookId })
    .sort({ batch: -1 })
    .limit(1)
    .next()

  const nextBatch = lastBatchDoc ? lastBatchDoc.batch + 1 : 1
  const targetBatch =
    lastBatchDoc && lastBatchDoc.count < 1000
      ? lastBatchDoc.batch
      : nextBatch

  // First, try to append to the intended batch, enforcing the 1000-item cap under concurrency.
  const overflowFilter = { bookId: bookId, batch: targetBatch }
  if (targetBatch !== nextBatch) {
    // Only enforce the count cap when targeting an existing batch.
    overflowFilter.count = { $lt: 1000 }
  }

  const overflowResult = await db.book_customers_extra.updateOne(
    overflowFilter, // Write to the intended batch, respecting the count cap when reusing a batch
    {
      $push: { customers: customerId },
      $inc: { count: 1 },
      $setOnInsert: { bookId: bookId, batch: targetBatch }
    },
    { upsert: targetBatch === nextBatch }
  )

  // If we failed to match when trying to reuse the previous batch (it filled concurrently),
  // fall back to writing into the next batch.
  if (overflowResult.matchedCount === 0 && targetBatch !== nextBatch) {
    await db.book_customers_extra.updateOne(
      { bookId: bookId, batch: nextBatch },
      {
        $push: { customers: customerId },
        $inc: { count: 1 },
        $setOnInsert: { bookId: bookId, batch: nextBatch }
      },
      { upsert: true }
    )
  }

  await db.books.updateOne(
    { _id: bookId },
    {
      $set: { hasExtras: true },
      $inc: { customerCount: 1 }
    }
  )
}

Index strategy:

// Index on main collection - only 50 entries per outlier doc
db.books.createIndex({ "customers": 1 })

// Index on overflow collection
db.book_customers_extra.createIndex({ bookId: 1 })
db.book_customers_extra.createIndex({ customers: 1 })

When to use outlier pattern:

When NOT to use this pattern:

• Uniform distribution: If all documents have similar array sizes, no outliers to isolate.
• Always need full data: If you always display all 50,000 customers, pattern doesn't help.
• Write-heavy outliers: Complex update logic may not be worth the read optimization.
• Small outliers: If outliers are 200 vs typical 50, just use larger threshold.

Verify with

// Find outlier documents
db.books.aggregate([
  { $project: {
    title: 1,
    customerCount: { $size: { $ifNull: ["$customers", []] } }
  }},
  { $sort: { customerCount: -1 } },
  { $limit: 20 }
])

// Calculate distribution
db.books.aggregate([
  { $project: { count: { $size: { $ifNull: ["$customers", []] } } } },
  { $bucket: {
    groupBy: "$count",
    boundaries: [0, 50, 100, 500, 1000, 10000, 100000],
    default: "100000+",
    output: { count: { $sum: 1 } }
  }}
])
// Look for a long-tail distribution where a small subset is far above median/p95

// Check index sizes
db.books.stats().indexSizes
// Large multikey index suggests outliers are bloating it

Reference: Outlier Pattern

Use Polymorphic Pattern for Heterogeneous Documents

Store related but different document shapes in one collection with a type discriminator. This keeps shared queries and indexes simple while allowing type-specific fields. Common use cases: product catalogs with different product types, content management systems, event stores, and any domain with inheritance.

Incorrect (separate collections per subtype):

Using a separate collection per product type (e.g. products_books, products_electronics, products_clothing) means querying across all products requires multiple calls or $unionWith, shared indexes must be duplicated, adding new types requires new collections, and application code must branch on collection names.

Correct (single collection using optional fields):

Store all product types in one products collection. All documents share common fields (name, price, inStock); each type adds its own specific fields (books: author, isbn, pages; electronics: brand, wattage, batteryHours, warranty; clothing: size, color, material). If the categories are always fully disjoint, use a type discriminator field (e.g. "book", "electronics", "clothing"). Cross-type queries use shared fields; type-specific queries filter by type plus type-specific fields. If there is potential overlap (e.g. between different categories of users), you can omit this field and rely entirely on optional fields.

Index strategies for polymorphic collections:

// Strategy 1: Compound index with type first
// Best for: Queries that always filter by type
db.products.createIndex({ type: 1, price: 1 })
db.products.createIndex({ type: 1, name: 1 })

// Query uses index efficiently:
db.products.find({ type: "book", price: { $lt: 50 } })

// Strategy 2: Compound index with type second
// Best for: Queries that rarely filter by type
db.products.createIndex({ price: 1, type: 1 })

// Query across all types uses index:
db.products.find({ price: { $lt: 50 } })

// Strategy 3: Partial indexes for type-specific fields
// Best for: Fields that only exist on some types
db.products.createIndex(
  { author: 1 },
  { partialFilterExpression: { type: "book" } }
)

db.products.createIndex(
  { brand: 1, wattage: 1 },
  { partialFilterExpression: { type: "electronics" } }
)

// Strategy 4: Wildcard index for varying fields
// Best for: Many type-specific fields, ad-hoc queries
db.products.createIndex({ "specs.$**": 1 })

// Documents store type-specific data in specs:
{ type: "book", specs: { author: "...", isbn: "..." } }
{ type: "electronics", specs: { brand: "...", wattage: 20 } }

Query patterns across types:

// Pattern 1: Query all types with shared fields
db.products.find({ price: { $lt: 100 }, inStock: true })
  .sort({ price: 1 })

// Pattern 2: Query specific type with type-specific fields
db.products.find({
  type: "book",
  pages: { $gt: 300 },
  author: /bradshaw/i
})

// Pattern 3: Aggregation across types with type-specific handling
db.products.aggregate([
  { $match: { inStock: true } },
  { $group: {
      _id: "$type",
      count: { $sum: 1 },
      avgPrice: { $avg: "$price" }
    }
  }
])

// Pattern 4: Faceted search with type breakdown
db.products.aggregate([
  { $match: { price: { $lt: 100 } } },
  { $facet: {
      byType: [{ $group: { _id: "$type", count: { $sum: 1 } } }],
      priceRanges: [
        { $bucket: {
            groupBy: "$price",
            boundaries: [0, 25, 50, 100],
            default: "100+"
          }
        }
      ]
    }
  }
])

Validation per type:

// Use JSON Schema with discriminator-based validation
db.runCommand({
  collMod: "products",
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["type", "name", "price"],
      properties: {
        type: { enum: ["book", "electronics", "clothing"] },
        name: { bsonType: "string" },
        price: { bsonType: "number", minimum: 0 }
      },
      oneOf: [
        {
          properties: { type: { enum: ["book"] } },
          required: ["author", "isbn"]
        },
        {
          properties: { type: { enum: ["electronics"] } },
          required: ["brand"]
        },
        {
          properties: { type: { enum: ["clothing"] } },
          required: ["size", "color"]
        }
      ]
    }
  },
  validationLevel: "moderate"
})

Adding new types:

The polymorphic pattern makes adding types straightforward — no schema migration needed. Insert documents with the new type value and any type-specific fields. Add partial indexes for type-specific queries as needed, and update schema validation to include the new type if using strict validation.

When NOT to use polymorphic pattern:

• Completely different access patterns: If each type is queried independently with no cross-type queries, separate collections may be cleaner.
• Conflicting index requirements: If types need many different indexes, the index overhead may outweigh benefits.
• Strict type separation required: Regulatory or security requirements may mandate separate collections.
• Vastly different document sizes: If one type has 100-byte docs and another has 100KB docs, working set suffers.
• Type-specific sharding needs: Different types may need different shard keys.

Verify with

// Get type distribution
db.products.aggregate([
  { $group: {
      _id: "$type",
      count: { $sum: 1 },
      avgSize: { $avg: { $bsonSize: "$$ROOT" } }
    }
  },
  { $sort: { count: -1 } }
])

// Check for missing type field
db.products.countDocuments({ type: { $exists: false } })

Reference: Polymorphic Schema Pattern

Schema Evolution and Preventing Drift

Schema changes are inevitable, but uncontrolled changes cause schema drift — documents in the same collection with inconsistent structures, leading to application errors and query failures. Use schemaVersion fields for safe migration and schema validation to prevent unexpected drift.

The problem: schema drift

MongoDB's flexibility is a feature, but undisciplined field additions lead to code that must handle many document shapes.

Incorrect (uncontrolled drift over time):

// Over time, different versions of "user" documents accumulate
{ _id: 1, name: "Alice", email: "alice@ex.com" }                  // 2021
{ _id: 3, firstName: "Carol", lastName: "Smith", email: "carol@ex.com" }  // 2022 - restructured name
{ _id: 4, firstName: "Dave", lastName: "Jones", emails: ["dave@ex.com"] } // 2023 - email → emails

// Application code becomes defensive nightmare
function getUserEmail(user) {
  if (user.email) return user.email
  if (user.emails) return user.emails[0]
  throw new Error("No email found")
}

// Queries fail silently
db.users.find({ email: "test@ex.com" })  // Misses users with emails[] array

Solution: versioned documents with migration path

Add a schemaVersion field to every document. Application code checks version and handles both formats. This allows old and new documents to coexist, new code to deploy before data migration, gradual migration during low-traffic periods, and easy rollback.

Correct (versioned with validation):

// Define and enforce consistent schema
db.createCollection("users", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["emails", "profile", "schemaVersion"],
      properties: {
        emails: {
          bsonType: "array",
          items: {
            bsonType: "string",
            pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
          }
        },
        profile: {
          bsonType: "object",
          required: ["firstName", "lastName"],
          properties: {
            firstName: { bsonType: "string", minLength: 1 },
            lastName: { bsonType: "string", minLength: 1 }
          }
        },
        schemaVersion: {
          bsonType: "int",
          enum: [1, 2]  // Accept both during migration
        }
      }
    }
  },
  validationLevel: "strict",
  validationAction: "error"
})

Online migration strategies

// Strategy 1: Background batch migration
// Best for: Large collections, can tolerate mixed versions temporarily

function migrateToV2(batchSize = 1000) {
  let migrated = 0
  let cursor = db.users.find({ schemaVersion: { $lt: 2 } }).limit(batchSize)

  for (const doc of cursor) {
    const [firstName, ...rest] = (doc.name || "").split(" ")
    const lastName = rest.join(" ") || "Unknown"

    db.users.updateOne(
      { _id: doc._id, schemaVersion: { $lt: 2 } },  // Prevent double-migration
      {
        $set: {
          schemaVersion: 2,
          profile: { firstName, lastName },
          emails: doc.emails || (doc.email ? [doc.email] : []),
        },
        $unset: { name: "", email: "" }
      }
    )
    migrated++
  }
  return migrated
}

// Run in batches during off-peak hours
while (migrateToV2(1000) > 0) {
  sleep(100)  // Throttle to reduce load
}


// Strategy 2: Aggregation pipeline update (MongoDB 4.2+)
// Best for: Simple transformations, moderate collection sizes

db.users.updateMany(
  { schemaVersion: { $lt: 2 } },
  [
    {
      $set: {
        schemaVersion: 2,
        profile: {
          $cond: {
            if: { $eq: [{ $type: "$name" }, "string"] },
            then: {
              firstName: { $arrayElemAt: [{ $split: ["$name", " "] }, 0] },
              lastName: { $ifNull: [
                { $arrayElemAt: [{ $split: ["$name", " "] }, 1] },
                "Unknown"
              ]}
            },
            else: "$profile"
          }
        },
        emails: {
          $cond: {
            if: { $eq: [{ $type: "$email" }, "string"] },
            then: ["$email"],
            else: { $ifNull: ["$emails", []] }
          }
        },
      }
    },
    { $unset: ["name", "email"] }
  ]
)


// Strategy 3: Read-time migration (lazy migration)
// Best for: Low-traffic documents, immediate consistency needed

function getUser(userId) {
  const user = db.users.findOne({ _id: userId })

  if (user && user.schemaVersion < 2) {
    const migrated = migrateUserToV2(user)
    db.users.replaceOne({ _id: userId }, migrated)
    return migrated
  }

  return user
}

Handling multiple version jumps

// v1 → v2 → v3: define transformation functions for each step
const migrations = {
  1: (doc) => ({
    ...doc,
    schemaVersion: 2,
    profile: {
      firstName: doc.name.split(" ")[0],
      lastName: doc.name.split(" ").slice(1).join(" ") || "Unknown"
    },
    emails: doc.email ? [doc.email] : []
  }),
  2: (doc) => ({
    ...doc,
    schemaVersion: 3,
    profile: {
      ...doc.profile,
      displayName: `${doc.profile.firstName} ${doc.profile.lastName}`
    }
  })
}

function migrateToLatest(doc, targetVersion = 3) {
  let current = doc
  while (current.schemaVersion < targetVersion) {
    const migrator = migrations[current.schemaVersion]
    if (!migrator) throw new Error(`No migration from v${current.schemaVersion}`)
    current = migrator(current)
  }
  return current
}

When a version bump is (and isn't) needed

No version bump needed (backward-compatible):

• Adding new optional fields (old code ignores them)
• Adding new indexes (transparent to application)
• Relaxing validation (making a required field optional)

Version bump required (breaking):

• Renaming fields (address → shippingAddress)
• Changing field types (price: "19.99" → price: 19.99)
• Restructuring (flat firstName/lastName → nested name: { first, last })
• Removing fields that old code reads

Detecting existing schema drift

// Find all unique field combinations
db.users.aggregate([
  { $project: { fields: { $objectToArray: "$$ROOT" } } },
  { $project: { keys: "$fields.k" } },
  { $group: { _id: "$keys", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
])
// Multiple distinct key-sets = schema drift exists

// Find documents missing required fields
db.users.find({
  $or: [
    { emails: { $exists: false } },
    { profile: { $exists: false } },
    { "profile.firstName": { $exists: false } }
  ]
})

// Find documents with wrong field types
db.users.find({
  emails: { $not: { $type: "array" } }
})

When NOT to strictly enforce schema or use versioning

• Truly polymorphic data: Event logs with different event types may need flexible schemas — use pattern-polymorphic instead.
• Early prototyping: Skip validation during exploration, add before production.
• User-defined fields: Some applications allow custom metadata fields.
• Small datasets with downtime window: If you can migrate all data in minutes during maintenance.
• Additive-only changes: If you only add optional fields, versioning is overkill.

Verify with

// Track version distribution
db.users.aggregate([
  { $group: { _id: "$schemaVersion", count: { $sum: 1 } } },
  { $sort: { _id: 1 } }
])

// Check for missing version field (implicit v1 documents)
db.users.countDocuments({ schemaVersion: { $exists: false } })

// Check if validation exists on the collection
const collInfo = db.getCollectionInfos({ name: "users" })[0]
const validator = collInfo?.options?.validator
// Missing validator = higher schema drift risk

// Find documents that don't match current validator
if (validator) {
  db.users.find({ $nor: [validator] }).limit(20)
  db.users.countDocuments({ $nor: [validator] })
}

References:

• Schema Versioning Pattern
• Schema Validation

Use Time Series Collections for Time Series Data

Time series collections are purpose-built for append-only measurements. MongoDB automatically buckets, compresses, and indexes time series data so you get high ingest rates with far less storage and index overhead than a standard collection. Use them for IoT sensor data, application metrics, financial data, and event logs.

MongoDB 8.0 Performance: Block processing introduced in MongoDB 8.0 can significantly improve eligible analytical pipelines (for example, $match + $sort on the time field + $group). In some cases, throughput improves by more than 200%. This is automatic for eligible queries.

Incorrect (regular collection for measurements):

// Regular collection: one document per reading
// Creates huge collections and indexes at scale
{
  sensorId: "temp-01",
  ts: ISODate("2025-01-15T10:00:00Z"),
  value: 22.5
}

// Problems:
// 1. Each measurement is a separate document
// 2. Index overhead per document
// 3. No automatic compression
// 4. Working set grows linearly

// Standard index (large and grows fast)
db.sensor_data.createIndex({ sensorId: 1, ts: 1 })

Correct (time series collection with optimized settings):

// Create time series collection with careful configuration
db.createCollection("sensor_data", {
  timeseries: {
    timeField: "ts",           // Required: timestamp field
    metaField: "metadata",     // Recommended: grouping field
    granularity: "minutes"     // Match your data rate
  },
  expireAfterSeconds: 60 * 60 * 24 * 90  // 90-day retention
})

// Insert documents - MongoDB buckets automatically
db.sensor_data.insertOne({
  metadata: { sensorId: "temp-01", location: "building-A" },
  ts: new Date(),
  value: 22.5,
  unit: "celsius"
})

// Benefits:
// - Automatic bucketing (many measurements per internal doc)
// - Column compression (40-60% disk reduction)
// - MongoDB 6.3+: auto-created compound index on metaField + timeField for new collections
// - Optimized for time-range queries

Choose the right metaField:

// metaField groups measurements into buckets
// Choose fields that:
// 1. Are queried together with time ranges
// 2. Have moderate cardinality (not too unique, not too few)
// 3. Don't change for a given time series

// GOOD: Sensor/device identifier as metaField
{
  metadata: { sensorId: "temp-01", region: "us-east" },
  ts: new Date(),
  value: 22.5
}
// Queries like: "All readings from temp-01 in last hour"

// BAD: High-cardinality field as metaField
{
  metadata: { requestId: "uuid-123..." },  // Unique per doc!
  ts: new Date()
}
// Creates one bucket per requestId - no compression benefit

// BAD: Frequently changing field in metaField
{
  metadata: { sensorId: "temp-01", currentValue: 22.5 },  // Changes!
  ts: new Date()
}
// metaField should be static for the time series

Select appropriate granularity:

// Granularity determines bucket time span
// Match it to your data ingestion rate

// "seconds" - DEFAULT. High-frequency ingestion. Bucket spans ~1 hour.
db.createCollection("high_freq_metrics", {
  timeseries: { timeField: "ts", metaField: "host", granularity: "seconds" }
})

// "minutes" - Data every few seconds to minutes. Bucket spans ~24 hours.
db.createCollection("app_metrics", {
  timeseries: { timeField: "ts", metaField: "service", granularity: "minutes" }
})

// "hours"   - Data every few hours. Bucket spans ~30 days.
db.createCollection("daily_reports", {
  timeseries: { timeField: "ts", metaField: "reportType", granularity: "hours" }
})

// Custom bucketing (MongoDB 6.3+) for precise control
db.createCollection("custom_metrics", {
  timeseries: {
    timeField: "ts",
    metaField: "device",
    bucketMaxSpanSeconds: 3600,      // Max 1 hour per bucket
    bucketRoundingSeconds: 3600      // Align to hour boundaries
  }
})

Optimize insert performance:

// Batch inserts with insertMany
// Group documents with same metaField value together
const batch = [
  { metadata: { sensorId: "temp-01" }, ts: new Date(), value: 22.5 },
  { metadata: { sensorId: "temp-01" }, ts: new Date(), value: 22.6 },
  { metadata: { sensorId: "temp-02" }, ts: new Date(), value: 19.2 },
]

db.sensor_data.insertMany(batch, { ordered: false })
// ordered: false allows parallel processing
// Use consistent field order and omit empty values for better compression

Secondary indexes on time series:

// MongoDB 6.3+: time series auto-creates index on { metaField, timeField } for new collections
// Add secondary indexes for other query patterns

// Index on measurement values for threshold queries
db.sensor_data.createIndex({ "value": 1 })
// Query: "All readings where value > 100"

// Compound index for filtered time queries
db.sensor_data.createIndex({ "metadata.location": 1, "ts": 1 })
// Query: "Readings from building-A in last hour"

// Partial index for specific conditions
db.sensor_data.createIndex(
  { "metadata.alertLevel": 1 },
  { partialFilterExpression: { "metadata.alertLevel": { $exists: true } } }
)

When NOT to use time series collections:

• Not time-based data: Primary access isn't time range queries.
• Frequent updates/deletes: Time series optimized for append-only; updates to old data are slow.
• Very low volume: A few hundred events don't benefit from bucketing.
• Need transactional writes: Time series collections don't support writes in transactions (reads are supported).
• Complex queries on measurements: If you mostly query by non-time fields, regular collections may be better.

Verify with

// Get collection info
const info = db.getCollectionInfos({ name: "sensor_data" })[0]
const ts = info?.options?.timeseries
// Check timeField, metaField, granularity, expireAfterSeconds

// Check bucket efficiency (via system.buckets)
const bucketColl = `system.buckets.sensor_data`
const bucketCount = db.getCollection(bucketColl).countDocuments({})
const stats = db.sensor_data.stats()
if (bucketCount > 0 && stats.count) {
  const docsPerBucket = stats.count / bucketCount
  // Low docs/bucket suggests adjusting granularity or metaField
}

Reference: Time Series Collections