Magento Indexing Caching

Magento Indexing and Caching

Overview

Magento's performance architecture relies on two distinct layers: indexers that pre-compute denormalized data (product prices, search indexes, category paths) into flat tables, and the Full-Page Cache (FPC) that stores complete rendered HTML pages. Varnish is the recommended FPC backend for production, replacing Magento's built-in file-based cache. Redis is the recommended backend for application cache, session storage, and the default cache (block HTML, config, layout). Misconfigured or stale indexes are one of the most common causes of incorrect pricing and product visibility issues.

When to Use This Skill

• When products appear with wrong prices or are invisible after catalog updates
• When implementing Varnish for the first time on a Magento production server
• When diagnosing slow category page loads caused by on-the-fly price calculation
• When configuring Redis for Magento's session and block cache storage
• When setting up cache invalidation after catalog or CMS updates via Magento's cache tags
• When managing indexer schedules on large catalogs (> 50,000 products) to prevent full reindex locking

Core Instructions

1. Understand Magento's indexers and their modes

   # List all indexers and their status
   bin/magento indexer:info
   bin/magento indexer:status

   # Example output:
   # catalog_category_product    Category Products    valid    Update on Save
   # catalog_product_category    Product Categories   valid    Update on Save
   # catalog_product_price       Product Price        invalid  Update by Schedule
   # catalogsearch_fulltext      Catalog Search       valid    Update by Schedule
   # catalogrule_product         Catalog Rule Product invalid  Update on Save

Set all production indexers to Update by Schedule mode to prevent checkout blocking:

   bin/magento indexer:set-mode schedule catalog_category_product
   bin/magento indexer:set-mode schedule catalog_product_price
   bin/magento indexer:set-mode schedule catalogsearch_fulltext
   bin/magento indexer:set-mode schedule catalog_product_flat
   bin/magento indexer:set-mode schedule catalog_category_flat

   # Verify modes
   bin/magento indexer:show-mode

2. Manage indexers and partial reindexing

   # Reindex a single indexer (less disruptive than full reindex)
   bin/magento indexer:reindex catalog_product_price

   # Reindex all (avoid on production during business hours)
   bin/magento indexer:reindex

   # Reset indexer to "invalid" to force next scheduled run
   bin/magento indexer:reset catalog_product_price

   # Check mview (materialized view) changelog tables — lists pending rows to process
   bin/magento indexer:show-changelog

For very large catalogs, use parallel indexing:

   # app/etc/env.php — enable parallel processing
   # In Admin → System → Index Management → Indexers → Configure
   # Or via config:
   bin/magento config:set dev/grid/async_indexing 1

   <?php
   // app/etc/env.php
   return [
       // ...
       'indexer' => [
           'batch_size' => [
               'catalog_product_price' => ['simple' => 200, 'configurable' => 50],
               'catalogsearch_fulltext' => ['simple' => 500],
           ],
       ],
   ];

3. Configure Varnish as Full-Page Cache

   # In Admin → System → Configuration → Advanced → System → Full Page Cache
   # Set Caching Application: Varnish Cache
   # Export Varnish configuration:
   bin/magento varnish:vcl:generate --export-version=6 --output-file=/etc/varnish/magento.vcl

Key sections of the generated VCL to understand:

   # /etc/varnish/magento.vcl (excerpts from generated config)

   sub vcl_recv {
       # Do not cache if private cookie is set (logged-in customer)
       if (req.http.cookie ~ "X-Magento-Vary=") {
           # Magento sets this cookie to vary cache by context (customer group, currency)
           # The cookie value changes only when the context changes — not per session
       }

       # Strip _ga, fbclid, utm_* params from cache key to prevent cache fragmentation
       set req.url = regsuball(req.url, "(\?|&)(utm_[^&]+|fbclid|gclid|mc_[^&]+)(&|$)", "\1");
   }

   sub vcl_hash {
       # Include Magento's context cookie in the cache hash
       if (req.http.cookie ~ "X-Magento-Vary=") {
           hash_data(regsub(req.http.cookie, ".*X-Magento-Vary=([^;]+).*", "\1"));
       }
   }

   sub vcl_backend_response {
       # Respect Magento's Cache-Control headers
       if (beresp.http.Cache-Control ~ "no-store") {
           set beresp.uncacheable = true;
           set beresp.ttl = 120s;
       }

       # Strip Set-Cookie on cacheable responses
       if (beresp.ttl > 0s) {
           unset beresp.http.set-cookie;
       }
   }

4. Configure Redis for application cache and sessions

   <?php
   // app/etc/env.php — Redis cache and session configuration

   return [
       'cache' => [
           'frontend' => [
               'default' => [
                   'id_prefix' => 'b0c_',
                   'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                   'backend_options' => [
                       'server'            => '127.0.0.1',
                       'database'          => '0',
                       'port'              => '6379',
                       'password'          => '',
                       'compress_data'     => '1',
                       'compression_lib'   => 'gzip',
                       'persistent'        => 'mgto_cache',
                   ],
               ],
               'page_cache' => [
                   'id_prefix' => 'b0c_',
                   'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                   'backend_options' => [
                       'server'        => '127.0.0.1',
                       'database'      => '1', // Separate DB for FPC
                       'port'          => '6379',
                       'compress_data' => '0', // FPC data is already compressed HTML
                   ],
               ],
           ],
       ],
       'session' => [
           'save'                  => 'redis',
           'redis' => [
               'host'              => '127.0.0.1',
               'port'              => '6379',
               'password'          => '',
               'timeout'           => '2.5',
               'persistent_identifier' => 'mgto_sessions',
               'database'          => '2', // Separate DB for sessions
               'compression_threshold' => '2048',
               'compression_library'   => 'gzip',
               'log_level'             => '1',
               'max_concurrency'       => '6',
               'break_after_frontend'  => '5',
               'max_lifetime'          => '7200',
               'disable_locking'       => '1', // Improves performance but reduces session safety
           ],
       ],
   ];

5. Implement cache tag invalidation for custom modules

Magento uses cache tags to selectively invalidate cached pages when data changes. Custom modules should tag and clean caches properly:

   <?php
   // Declare cache tags in your block/model
   namespace MyVendor\CustomModule\Block;

   use Magento\Framework\View\Element\Template;
   use Magento\Framework\DataObject\IdentityInterface;

   class CustomProductWidget extends Template implements IdentityInterface
   {
       private array $productIds = [];

       public function getIdentities(): array
       {
           // Return cache tags so Magento invalidates this block when any of these products change
           $tags = [\Magento\Catalog\Model\Product::CACHE_TAG];
           foreach ($this->productIds as $id) {
               $tags[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $id;
           }
           return $tags;
       }
   }

   <?php
   // In your observer or command — flush only relevant pages after product update
   namespace MyVendor\CustomModule\Observer;

   use Magento\Framework\Event\ObserverInterface;
   use Magento\Framework\App\Cache\TypeListInterface;
   use Magento\PageCache\Model\Cache\Type as PageCacheType;

   class ProductSaveObserver implements ObserverInterface
   {
       public function __construct(
           private readonly TypeListInterface $cacheTypeList,
           private readonly \Magento\Framework\App\Cache\Tag\Resolver $tagResolver
       ) {}

       public function execute(\Magento\Framework\Event\Observer $observer): void
       {
           $product = $observer->getEvent()->getProduct();

           // Invalidate only pages tagged with this product's cache tag
           // Varnish will purge via PURGE request using X-Magento-Tags-Pattern header
           $this->cacheTypeList->invalidate(PageCacheType::TYPE_IDENTIFIER);
       }
   }

Examples

Diagnose index lag causing wrong prices

# Check if mview changelog has pending rows
bin/magento indexer:show-changelog

# If catalog_product_price has > 0 pending rows, prices are stale
# Run the indexer to catch up
bin/magento indexer:reindex catalog_product_price

# Check cron_schedule table for missed cron jobs (should run indexers every minute)
bin/magento cron:run --group index &
# Or manually via:
mysql -u root -e "SELECT job_code, status, finished_at FROM magento.cron_schedule WHERE job_code LIKE '%index%' ORDER BY finished_at DESC LIMIT 20;"

Warm FPC after deployment

#!/bin/bash
# scripts/warm-cache.sh — warm the FPC after deploy
set -e

MAGENTO_BASE_URL="https://www.mystore.com"
SITEMAP_URL="$MAGENTO_BASE_URL/sitemap.xml"

echo "Flushing Magento caches..."
bin/magento cache:flush

echo "Warming FPC via sitemap..."
# Download sitemap and curl each URL with a warm-cache header
curl -s "$SITEMAP_URL" | \
  grep -oP '<loc>\K[^<]+' | \
  head -500 | \
  xargs -P 10 -I {} curl -s -o /dev/null -w "%{http_code} {}\n" \
    -H "X-Warm-Cache: 1" {}

echo "Cache warming complete."

Best Practices

• Always use `Update by Schedule` mode in production — Update on Save reindexes synchronously during admin saves, causing timeouts on large catalogs
• Separate Redis databases for cache, FPC, and sessions — using database 0 for everything causes key conflicts and makes it impossible to flush only one type
• Tag custom blocks with `IdentityInterface` — this ensures Varnish and Magento's FPC invalidate the right pages when your data changes
• Monitor indexer changelog table sizes — a catalog_product_price_cl table with millions of rows indicates cron is not running; fix cron before the table size causes reindex timeouts
• Use `bin/magento cache:clean` vs `cache:flush` — clean removes only invalid cache entries (safe); flush nukes everything including other framework caches sharing the same Redis
• Set distinct `id_prefix` per environment — staging and production can share a Redis server if each has a unique prefix, preventing cache poisoning
• Test Varnish with `curl -I` and `X-Cache` header — a HIT response confirms Varnish is serving from cache; MISS means it's hitting Magento for every request

Common Pitfalls

Related Skills

• @magento-module-development
• @magento-graphql
• @magento-multi-store
• @caching-strategies
• @infrastructure-performance

Custom Magento Module: Product Widget with Proper Cache Integration

Problem/Feature Description

A Magento agency has built a custom module called Acme_FeaturedProducts that displays a homepage widget showing hand-picked featured products. After deploying the module to production (which runs Varnish as the full-page cache), the client has reported that when a product price changes or a product is disabled, the homepage continues showing the old data for hours — sometimes until the next overnight cache flush. The Varnish server is running correctly and serving other pages fine.

The agency needs to fix the module so that Varnish properly invalidates cached pages when the featured products change. They also need to create a Magento observer that flushes the right cache entries when a product is saved, and ensure the development team knows which CLI command to use for routine cache maintenance going forward.

Output Specification

Produce the following files for the fixed module:

1. Block/FeaturedProducts.php — The corrected block class. The widget displays products from a hardcoded list of product IDs (use [1, 5, 12, 47] as the IDs for this implementation). The block should integrate with Magento's cache tagging system so Varnish knows which pages to invalidate.

2. Observer/ProductSaveObserver.php — An observer class that invalidates the full-page cache after a product is saved. Use the correct Magento API for this; the invalidation should work with Varnish's tag-based purging.

3. cache-maintenance.md — A one-page guide for the client's admin team explaining which Magento CLI cache commands to use for day-to-day maintenance and when each is appropriate.

Input Files

The following files represent the current (broken) state of the module. Extract them before beginning.

=============== FILE: current/Block/FeaturedProducts.php =============== <?php namespace Acme\FeaturedProducts\Block;

use Magento\Framework\View\Element\Template;

class FeaturedProducts extends Template { private array $productIds = [1, 5, 12, 47];

public function getProductIds(): array { return $this->productIds; } } =============== FILE: current/Observer/ProductSaveObserver.php =============== <?php namespace Acme\FeaturedProducts\Observer;

use Magento\Framework\Event\ObserverInterface;

class ProductSaveObserver implements ObserverInterface { public function execute(\Magento\Framework\Event\Observer $observer): void { // TODO: invalidate cache after product save } }

Magento Production Indexer Configuration

Problem/Feature Description

A mid-sized e-commerce retailer recently launched their Magento 2.4 store on a dedicated server. Since launch, the operations team has noticed several problems: admin users experience timeouts when saving products with large attribute sets, checkout occasionally stalls after promotions are applied, and the merchandising team has reported that newly imported products sometimes show incorrect prices for up to an hour after import.

The catalog contains approximately 80,000 SKUs including configurable products with many variants. The engineering team suspects the indexing configuration is contributing to both the timeout issues and the stale data problems, and they've asked you to produce a production-ready indexer setup and a runbook for diagnosing stale data incidents.

Output Specification

Produce the following files:

1. setup-indexers.sh — A shell script that configures the indexers appropriately for production use and enables any optimizations suitable for a large catalog. The script should include comments explaining what each step does and why.

2. env-indexer-snippet.php — A PHP snippet showing the relevant section to add to app/etc/env.php to optimize indexer performance for the catalog size described above. Include exact values.

3. indexer-runbook.md — A markdown runbook for the operations team covering:

• How to diagnose whether stale index data is causing wrong prices or invisible products
• Safe procedures for triggering a reindex on a live production system
• What to check when automatic reindexing appears to have stopped working

Redis Backend Configuration for Magento Production

Problem/Feature Description

A growing online retailer is migrating their Magento 2.4 store from a single-server setup to a dedicated infrastructure. Their current setup uses the default file-based cache and PHP sessions stored on disk, which is causing race conditions during flash sales and poor performance under load. They've provisioned a single Redis instance (running on 127.0.0.1:6379) to serve all cache and session needs.

The company runs two environments — staging and production — that will share this Redis instance to reduce infrastructure costs. The DevOps lead is concerned about cache poisoning between environments and session loss under high concurrency. They also want clear Redis server configuration recommendations to handle memory pressure gracefully without losing customer sessions.

Output Specification

Produce the following files:

1. env-redis.php — The complete Redis section to insert into app/etc/env.php for a production environment. The configuration should cover the default application cache, full-page cache, and session storage. Use placeholder values where credentials would go, but all other settings should be concrete and production-ready.

2. redis-server.conf — A redis.conf snippet (or a commented configuration file) showing the recommended memory eviction policy settings for the three logical Redis use cases in this setup (application cache, full-page cache, and sessions). Include comments explaining why each policy is appropriate.

3. redis-setup-notes.md — A brief technical notes document for the DevOps team explaining the multi-environment Redis sharing approach and any naming conventions used to prevent conflicts.