Mongodb Query Optimizer

MongoDB Query Optimizer

When this skill is invoked

Invoke only when the user wants:

• Query/index optimization or performance help
• Why a query is slow or how to speed it up
• Slow queries on their cluster and/or how to optimize them

Do not invoke for routine query authoring unless the user has requested help with optimization, slow queries, or indexing.

High Level Workflow

General Performance Help

If the user wants to examine slow queries, or is looking for general performance suggestions (not regarding any particular query):

• Use MongoDB MCP server atlas-get-performance-advisor tool to fetch slow query logs and performance advisor output
• Make suggestions based on this information

If Atlas MCP Server for Atlas is not configured or you don’t have enough information to run atlas-get-performance-advisor against the correct cluster, tell the user that general performance analysis requires Atlas MCP Server configuration with API credentials, and suggest they configure it or ask about a specific query instead.

Help with a Specific Query

If the user is asking about a particular query:

• Use collection-indexes, explain, and find MCP tools to get existing indexes on the collection, explain() output for the query, and a sample document from the collection
• Use atlas-get-performance-advisor MCP tool to fetch slow query logs and performance advisor output

Then make an optimization suggestion based on collected information and MongoDB best practices and examples from reference files. Prefer creating an index that fully covers the query if possible. If you cannot use MongoDB MCP Server then still try to make a suggestion.

MCP: available tools

How to invoke. Call the MongoDB MCP server with the exact tool name as toolName and a single arguments object as arguments. Do not pass the tool name as an option, query param, or nested key; pass it as the MCP tool name and the parameters as the arguments object. Full MCP Server tool reference: MongoDB MCP Server Tools.

Database tools (when the MCP cluster connection works):

Atlas tools (when Atlas API credentials are configured):

For a user question, try to fetch information from both the connection string and Atlas API related to the query you are optimizing.

1\. DB connection string works for MongoDB MCP

Typical flow: call collection-indexes → explain → find (sample doc).

• `collection-indexes` — Use the result's classicIndexes (each has name, key) to see if the query can already use an existing index.
• `explain` — Run in "queryPlanner" mode first to check for COLLSCAN. If the query uses an index or the collection is very small, run again with "executionStats" (10-second timeout) to get docs scanned vs. returned.

2\. Atlas API access works for MongoDB MCP

If you need a project ID, call atlas-list-projects first. Then call atlas-get-performance-advisor with only the operations you need:

Do not pass the MCP tool name as an operations value—operations is a separate argument listing what data to fetch.

Example workflow 1 (help with specific query)

User: "Why is this query slow? db.orders.find({status: 'shipped', region: 'US'}).sort({date: -1})"

If MCP db connection is configured and the database + collection names are known, run steps 1–3. Otherwise skip to step 4.

1. Check existing collection indexes:

• Call collection-indexes with database=store, collection=orders
• Result shows: {_id: 1}, {status: 1}, {date: -1}

2. Run explain:

• Call explain with method=find, filter={status: 'shipped', region: 'US'}, sort={date: -1}, verbosity=queryPlanner and executionStats
• Result: Uses {status: 1} index, then in-memory SORT, totalKeysExamined: 50000, nReturned: 100

3. Run find:

• Call find with limit=1 to fetch a sample document to impute the schema.

If MCP Atlas connection is configured, run step 4. Otherwise skip to step 5.

4. Run atlas-get-performance-advisor:

• Try to get the cluster name from the MCP connection string, or ask the user for projectId/clusterName
• Use slowQueryLogs to fetch slow query logs from database=store, collection=orders in the past 24 hours
• Use suggestedIndexes to check for index suggestions for the query

5. Diagnose: Based on explain output and slow query logs, this query targets 100 docs but scans 50K index entries (poor selectivity: 0.002). In-memory sort adds overhead. Index doesn't support both filter fields or sort.

6. Recommend: Create compound index {status: 1, region: 1, date: -1} following ESR (two equality fields, then sort). This eliminates in-memory sort and improves selectivity by filtering on both status and region.

If the MongoDB MCP server is not set up, follow best indexing practices.

Example workflow 2 (general database performance help)

User: "Can you help with optimizing slow queries on my cluster?”

1. Run atlas-get-performance-advisor:

• Try to get the cluster name from the connection string and deduce the project name you need in atlas-list-projects; if you are not sure, then ask the user for cluster name and project id.
• Use slowQueryLogs to fetch slow query logs from the past 24 hours
• Use suggestedIndexes
• Use dropIndexSuggestions
• Use schemaSuggestions

2. Diagnose and Recommend: Based on slow query logs and performance advisor advice, you can create the compound index {status: 1, region: 1, date: -1} on the db.orders collection to optimize queries such as find({status: 'shipped', region: 'US'}).sort({date: -1})

Examine all performance advisor output as well as slow query logs. Provide information on what is being improved and why, and focus on suggestions that have the potential for greatest impact (e.g., indexes that affect the most queries, or queries that have the worst performance).

Load references

Before beginning diagnosis and recommendation, load reference files.

Always load:

• references/core-indexing-principles.md
• references/antipattern-examples.md

Conditionally load these files:

• If diagnosing aggregation pipelines → references/aggregation-optimization.md
• If diagnosing queries that change docs such as replaceOne, findOneAndUpdate, etc. → references/update-query-examples.md for oplog-efficient updates and common update anti-patterns

Output

• Keep answers short and clear: a few sentences on index and optimization suggestions, and reasoning behind them (e.g. general indexing principles, observing slow query logs in the cluster, or seeing advice in Performance Advisor)
• Focus on highest impact indexes or optimizations - if you've omitted some optimizations let the user know and present them if asked.
• Do not use strong language, such as saying “You should create these indexes and they will definitely improve application performance” \- Explain they are suggestions for certain queries, and give the reasoning behind them.
• Consider how many indexes already exist on the collection (if known) \- there shouldn’t generally be more than 20
• Suggest removing indexes only if the suggestion comes from Atlas Performance Advisor
• Do not create indexes directly via MCP unless the user gives approval

Principles

Aggregation pipelines process documents through sequential stages. Focus on:

• Reducing documents early in the pipeline
• Minimizing data moved between stages
• Leveraging indexes where possible
• Managing memory usage

Memory limits and disk spilling

Blocking stages (such as in-memory $sort and $group) have a 100MB memory limit per stage. Default behavior when this limit is exceeded is to spill to disk automatically (allowDiskUse defaults to true).

Better solutions:

• Filter more aggressively early in pipeline
• Add indexes to enable $sort to use index order
• Use $limit with $sort to reduce the amount of data the sort must process in memory for unindexed sorts
• Consider materialized views for repeated aggregations

Optimization Examples

These examples are not exhaustive but representative of some common optimization patterns.

Unindexed $lookup vs. Indexed $lookup

Bad — No index on the foreign collection's join field:

db.orders.aggregate([
  { $lookup: {
      from: "products",
      localField: "productId",
      foreignField: "sku",   // no index on products.sku!
      as: "product"
  }}
])

Good — Index on foreignField in the foreign collection:

db.products.createIndex({ sku: 1 })

db.orders.aggregate([
  { $lookup: {
      from: "products",
      localField: "productId",
      foreignField: "sku",
      as: "product"
  }}
])

Why: Each $lookup executes a find on the from collection. Without an index on foreignField, every join does a full collection scan. This is the single most critical $lookup optimization.

Early $project Defeating Optimization vs. Late $project

Bad — Early $project prevents the optimizer from pruning unused fields, forgets to exclude _id which is unneeded, and includes name which is not used:

db.collection.aggregate([
  { $project: { name: 1, status: 1, amount: 1 } },
  { $match: { status: "active" } },
  { $group: { _id: "$status", total: { $sum: "$amount" } } }
])

Good — Let the optimizer handle field pruning; use $project only at the end for reshaping:

db.collection.aggregate([
  { $match: { status: "active" } },
  { $group: { _id: "$status", total: { $sum: "$amount" } } },
  { $project: { _id: 0, status: "$_id", total: 1 } }  // reshape at the end
])

Why: MongoDB's pipeline optimizer automatically analyzes which fields are used and avoids fetching unused ones. An early $project defeats this optimization, and can inadvertently request the wrong fields.

$facet for Divergent Processing vs. $unionWith

Bad — $facet sends all documents to every branch, even if branches need very different subsets:

db.collection.aggregate([
  { $facet: {
      "top10": [{ $sort: { score: -1 } }, { $limit: 10 }],
      "totalCount": [{ $count: "n" }]  // gets ALL docs even though it's just counting
  }}
])

Good — Separate pipelines via $unionWith let each branch optimize independently:

db.collection.aggregate([
  { $sort: { score: -1 } }, { $limit: 10 },
  { $unionWith: {
      coll: "collection",
      pipeline: [{ $count: "n" }]
  }}
])

Why: $facet funnels every document into every branch. $unionWith runs independent pipelines that each benefit from their own index usage and optimization.

$sort \+ $limit as Separate Concerns vs. Top-N Sort

Bad — Large sort, then limit (MongoDB may sort entire dataset):

db.collection.aggregate([
  { $group: { _id: "$category", total: { $sum: "$amount" } } },
  { $sort: { total: -1 } },
  // ... many stages later ...
  { $limit: 10 }
])

Good — Place $limit immediately after $sort:

db.collection.aggregate([
  { $group: { _id: "$category", total: { $sum: "$amount" } } },
  { $sort: { total: -1 } },
  { $limit: 10 }
])

Why: When $sort is immediately followed by $limit, MongoDB performs a top-N sort — it only tracks the top N values instead of sorting the full dataset. Far less memory.

$unwind Best Practices

When $unwind is needed, filter before unwinding so that the $match stage allows index usage:

[
  { $match: { "items.category": "electronics" } },  // Reduce documents first
  { $unwind: "$items" },  // Then unwind
  { $match: { "items.category": "electronics" } }  // Filter unwound elements
]

Never $unwind to re-group by `_id`: If you are using $unwind followed by $group with _id: you can replace it with an array operator like $filter, $map or $reduce to match or transform array elements without unwinding.

Optimize $lookup operations

$lookup performs collection joins and can be expensive. Strategies to improve performance:

1. Filter before lookup to reduce left-side documents 2. Use indexed fields in the lookup localField/foreignField 3. Add $match in the lookup pipeline to reduce right-side documents early 4. Add $project last in the lookup pipeline to keep only the fields you need 5. $unwind immediately after lookup when you need as result flattened

[
  { $match: { active: true } },  // Reduce left side
  { $lookup: {
      from: "inventory",
      localField: "product_id",
      foreignField: "_id",  // _id is always indexed
      pipeline: [
        { $match: { inStock: true } },  // Reduce right side
        { $project: { _id: 0, name: 1, price: 1 } }
      ],
      as: "product"
  }},
  { $unwind: "$product" }
]

Schema consideration: Excessive $lookup usage may indicate over-normalization. Consider embedding frequently-joined data.

$group efficiency

Group operations require accumulating result documents in memory. Keys to efficiency:

1. Include only needed fields within the $group stage \- reference only the fields you need in accumulators 2. Be mindful of unbounded accumulators \- $push and $addToSet grow as group size increases and can cause memory issues

Bad \- do not add $project before $group to "reduce fields":

[
  { $match: { date: { $gte: ISODate("2024-01-01") } } },
  { $project: { category: 1, amount: 1 } },
  { $group: {
      _id: "$category",
      total: { $sum: "$amount" },
      count: { $sum: 1 }
  }}
]

Good \- reference only needed fields directly in $group:

[
  { $match: { date: { $gte: ISODate("2024-01-01") } } },
  { $group: {
      _id: "$category",
      total: { $sum: "$amount" },
      count: { $sum: 1 }
  }}
]

Why: The $group stage only processes the fields referenced in its expressions. Adding a $project before it does not save memory.

$exists on Regular Index vs. Sparse Index

Bad — $exists: true on a regular index still requires a document fetch:

db.collection.createIndex({ a: 1 })
db.collection.find({ a: { $exists: true } })
// Cannot efficiently answer — null semantics require checking each document

Good — Use a sparse index, which only contains entries where the field exists:

db.collection.createIndex({ a: 1 }, { sparse: true })
db.collection.find({ a: { $exists: true } })
// Answered directly from the index — no document fetch needed

Why: Regular indexes store null for both missing and existing fields that are set to null, so $exists can't be answered from the index alone. Sparse indexes only store entries for documents where the field exists.

Unanchored $regex vs. Anchored $regex

Bad — Unanchored case insensitive regex cannot use the index efficiently:

db.collection.find({ name: { $regex: /smith/i } })
// Full index or collection scan — case-insensitive, not anchored

Good — Anchored, case-sensitive regex uses the index as a range query:

db.collection.find({ name: { $regex: /^Smith/ } })
// Efficient index range scan on the "Smith" prefix

Why: Indexes store values in sorted order. Only a left-anchored, case-sensitive $regex can be converted into an efficient index range scan. For case-insensitive matching, use a case-insensitive collation index instead.

$where / JavaScript vs. Native MQL Operators

Bad — Server-side JavaScript execution:

db.collection.find({
  $where: "this.price * this.quantity > 1000"
})

Good — Native aggregation expression:

db.collection.find({
  $expr: { $gt: [{ $multiply: ["$price", "$quantity"] }, 1000] }
})

Why: JavaScript executed on the server is always slower than native MQL, cannot use indexes. It's also a security risk and is deprecated. Use $expr with aggregation operators instead.

In-Memory Sort vs. Index-Supported Sort

Bad — Sort on an unindexed field triggers in-memory sort:

db.orders.find({ status: "processing" }).sort({ createdAt: -1 })
// Index: { status: 1 } — sort is done in memory

Good — Compound index supports both filter and sort:

db.orders.createIndex({ status: 1, createdAt: -1 })
db.orders.find({ status: "processing" }).sort({ createdAt: -1 })
// No SORT stage in the plan — results come pre-sorted from the index

Core Index Principles

Compound Index Guidelines

The first field of the index should be in the query's filter or sort condition.

Equality → Sort → Range order is most often preferred:

• Equality fields first (e.g. {field: value}, {$in: [...]} with \<= 200 elements, {field: {$eq: value}})
• Sort fields next
• Range fields last (e.g. $gt, $lt, $gte, $lte, {$in: [...]} with \> 200 elements in the array, $ne, anchored case-sensitive $regex)

If equality is not very selective and range is, then ERS may perform better than ESR.

Sort direction

Index {a:1, b:1} supports sort({a:1, b:1}) and reverse sort({a:-1, b:-1}), but NOT mixed directions like sort({a:1, b:-1}). For mixed sorts, create index matching exact pattern.

Collation Match

Before — Query collation differs from index collation, so the index cannot be used:

db.users.createIndex({ name: 1 })
db.users.find({ name: "José" }).collation({ locale: "es", strength: 2 })
// Index cannot be used for query

After — Create the index with the same collation the query uses:

db.users.createIndex({ name: 1 }, { collation: { locale: "es", strength: 2 } })
db.users.find({ name: "José" }).collation({ locale: "es", strength: 2 })
// Index can be used for query

Why: Collation must match between index and query.

Covered Queries

A covered query retrieves data directly from the index, never accessing the actual documents. This is extremely fast and preferable when possible.

Requirements

1. All query fields are in the index 2. All returned fields are in the index (includes sort fields) 3. Inclusion projection required \- you must use an inclusion projection (e.g., { field: 1 }) that requests only indexed fields, plus _id: 0 if _id is not in the index. Exclusion projections cannot produce covered queries. 4. No `$exists` or null equality checks \- queries using $exists or querying for null/missing values cannot usually be covered by an index 5. Multikey index constraints \- multikey indexes can cover queries under certain conditions, such as when the array field itself is not included in the projection and operators like $elemMatch are not used. If the array field must be projected, covering is not possible.

Building a covered query

Step 1: Identify your query pattern

db.products.find(
  { category: "electronics", inStock: true },
  { category: 1, inStock: 1, price: 1, _id: 0 }
).sort({ price: 1 })

Step 2: Create index with all accessed fields

Following ESR (Equality-Sort-Range):

db.products.createIndex({
  category: 1,    // Equality
  inStock: 1,     // Equality
  price: 1        // Sort
})

Step 3: Project only indexed fields

• Include indexed fields in projection
• Exclude \_id unless \_id is in the index (use _id: 0)
• Don't request fields not in the index

Common mistakes

Forgetting to explicitly exclude \_id

// NOT COVERED - _id not in index but included in result
db.products.find(
  { category: "electronics" },
  { category: 1, price: 1 }  // _id included by default!
)

Fix: Explicitly exclude \_id

db.products.find(
  { category: "electronics" },
  { category: 1, price: 1, _id: 0 }  // Now covered
)

Requesting non-indexed fields

// NOT COVERED - description not in index
db.products.find(
  { category: "electronics" },
  { category: 1, price: 1, description: 1, _id: 0 }
)

Fix: Only project indexed fields, or add description to index

Array fields (multikey indexes)

// NOT COVERED - tags is an array field and is included in projection
db.products.createIndex({ tags: 1, price: 1 })
db.products.find(
  { tags: "sale" },
  { tags: 1, price: 1, _id: 0 }
)

Fix: If the array field is not needed in the result, remove it from the projection:

// COVERED - array field (tags) used in query but not projected
db.products.find(
  { tags: "sale" },
  { price: 1, _id: 0 }
)

Multikey indexes can cover queries when the array field itself is not projected and operators like $elemMatch are not used. If you must return the array field, the query cannot be covered.

Update Query Examples

replaceOne vs. updateOne with $replaceWith

Bad — Full document replacement generates a large oplog entry:

db.coll.replaceOne({ _id: X }, entireNewDocument)

Good — Use aggregation-based update to generate smaller oplog deltas:

db.coll.updateOne({ _id: X }, [{ $replaceWith: { $literal: entireNewDocument } }])

Why: replaceOne writes the full document to the oplog. The aggregation update syntax lets MongoDB compute deltas, resulting in smaller oplog entries when only a few fields are changed.

findOneAndUpdate Misuse vs. updateOne

Bad — Using findOneAndUpdate when you don't need the document returned:

db.coll.findOneAndUpdate(
  { _id: X },
  { $set: { status: "processed" } }
)

Good — Use updateOne when you don't need the result document:

db.coll.updateOne(
  { _id: X },
  { $set: { status: "processed" } }
)

Why: findOneAndUpdate writes a copy of the pre-change document to a side collection for retryable writes. This overhead is unnecessary if you don't need the returned document.