Elasticsearch

Elasticsearch — 分布式搜索与分析引擎

Elasticsearch 是基于 Apache Lucene 的分布式搜索和分析引擎，提供近实时的全文搜索、结构化搜索、聚合分析、地理空间查询和向量搜索能力。

Workflow — 从数据到搜索的 5 步决策流程

遇到 ES 相关需求时按以下顺序决策:

Step 1: 明确场景
├── 全文搜索?                    → Step 2
├── 结构化数据分析 (日志/指标)?   → Step 2
├── 向量/KNN 搜索?               → references/ 向量搜索
├── 地理空间查询?                 → references/ 地理查询
├── 集群运维/性能问题?             → references/ 07-cluster-ops
└── 数据管道/日志采集?             → references/ 08-elk-integration

Step 2: 设计索引 (Mapping + Analyzer)
├── 确定字段类型 (text/keyword/date/geo/nested…)
├── 选择分词器 (standard/ik/pinyin/自定义)
├── 规划主分片数 (每个分片 20-50GB), 建后不可改
└── 设置别名用于零停机重建

Step 3: 写入数据
├── Index API (单条), Bulk API (批量, 5-15MB/批)
├── _update (部分更新), _update_by_query (条件更新)
└── Logstash/Filebeat (日志采集管道)

Step 4: 构建查询
├── 全文搜索: match / multi_match / query_string
├── 精确过滤: term / terms / range / exists / bool filter
├── 复合查询: bool (must/should/filter/must_not)
├── 聚合分析: terms + avg → references/ 聚合专题
└── 排序/分页: sort / search_after (深度分页) / scroll (导出)

Step 5: 持续优化
├── 性能问题?  → Profile API + 慢查询日志
├── 数据增长?  → ILM + Rollover 自动化 (references/ 07-cluster-ops)
├── Mapping 变更? → Reindex + Alias 零停机 (examples/03)
└── 查询优化?  → filter 优先 query, 避免 script, 限制 _source

When to Use / When NOT to

核心原则：Elasticsearch 是搜索服务器，不是关系型数据库的替代品。

Boundary — 能力边界

核心概念速查

查询 DSL 速查

聚合速查

集群运维概述

Gotchas — 常见陷阱与反模式

FAQ

Q1: ES 是关系型数据库的替代品吗？ 不是。ES 是搜索服务器，不支持 ACID 事务、外键、复杂 JOIN。正确架构：业务数据存 RDBMS，ES 做搜索和聚合。

Q2: 数据写入后多久能查到？ 近实时。写入先到 buffer，默认 1s refresh 后才可搜索。可调 refresh_interval 或加 ?refresh 参数。

Q3: text 和 keyword 字段区别？ text：分词后索引，支持 match 搜索，不支持排序/聚合。keyword：完整值索引，支持 term 搜索、排序、聚合。

Q4: 主分片数为什么不能修改？ 路由规则 hash(_id) % shards，修改后已有数据无法定位。变更需 reindex。

Q5: 如何选择分片数？ 每个分片 20-50GB。500GB 原始数据 → 10-25 个主分片。每个节点 ≤25 分片/GB 堆内存。

Q6: ES 为什么搜索快？ 倒排索引：将每个词项映射到文档列表，查找直接定位。加分片并行 + filter 缓存。

Q7: 聚合 (Aggregation) 是什么？ Bucket = GROUP BY 分组，Metric = AVG/SUM/COUNT，Pipeline = 聚合结果的再分析。详见 references/03, references/04。

Q8: Green/Yellow/Red 状态？ Green=全正常，Yellow=主分片正常但副本未分配，Red=主分片丢失。

Q9: 如何零停机重建索引？ Alias + Reindex：创建新索引 → Reindex 数据 → 原子切换 Alias → 删除旧索引。见 examples/03。

Q10: ES 和 Solr 怎么选？ ES 集群管理内置、近实时搜索 1s、聚合强大、ELK 生态完整。Solr 依赖 ZK，配置复杂。

Q11: Mapping 可以修改吗？ 可新增字段，不可修改已有字段类型（如 text→keyword），需重建索引。

Q12: search_after 和 from+size 区别？ from+size 深度分页 OOM（限制 10000）。search_after 基于排序值翻页，性能与深度无关。

Q13: 数据备份怎么做？ Snapshot API 备份到 S3/GCS/FS。推荐 SLM 自动管理。增量快照只存变化。

Q14: 查询慢如何排查？ Profile API → 慢查询日志 → filter vs query → segments 数量 → GC 日志。

Q15: ILM 能解决什么问题？ 自动滚动（大小/时间阈值）、自动迁移（热→温→冷）、自动压缩、自动删除。

Keywords

elasticsearch, ES, 搜索引擎, 全文搜索, 倒排索引, Lucene, index, mapping, document, shard, replica, analyzer, ik, pinyin, 查询 DSL, match, term, bool, filter, range, multi_match, query_string, nested, geo, aggregation, 聚合, terms, date_histogram, avg, sum, cardinality, percentiles, pipeline, reindex, bulk, scroll, search_after, ILM, rollover, force_merge, alias, snapshot, cluster, ELK, Logstash, Kibana, Filebeat, KNN, dense_vector, 向量搜索, painless, ingest pipeline, RBAC, profile API, 慢查询, zero downtime

References

• references/01-query-dsl-fulltext.md — 全文查询（match/multi_match/query_string）
• references/02-query-dsl-term.md — 精确查询与复合查询（term/range/bool）
• references/03-aggregations-metric.md — 指标聚合（avg/sum/stats/cardinality/percentiles）
• references/04-aggregations-bucket.md — 桶聚合（terms/date_histogram/range）
• references/05-mapping-types.md — 映射与字段类型详解
• references/06-analyzers.md — 分词器（标准/IK/pinyin/自定义）
• references/07-cluster-ops.md — 集群运维（分片/监控/快照/ILM）
• references/08-elk-integration.md — Logstash/Filebeat/Kibana 配置
• examples/01-fulltext-search.md — 全文搜索实战
• examples/02-aggregation-report.md — 聚合报表实战
• examples/03-reindex-zero-downtime.md — 零停机重建索引
• examples/04-cluster-monitoring.md — 集群监控实战

示例: 全文搜索实战 — 电商商品搜索

场景

构建一个电商商品搜索功能，支持关键词搜索、多字段搜索、价格筛选、分页。

步骤

1. 创建索引与 Mapping

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "ik_max_word"
      },
      "category": { "type": "keyword" },
      "brand": { "type": "keyword" },
      "price": { "type": "float" },
      "tags": { "type": "keyword" },
      "status": { "type": "keyword" },
      "created_at": { "type": "date" },
      "stock": { "type": "integer" }
    }
  }
}

2. 导入示例数据

POST /products/_bulk
{"index":{"_id":1}}
{"title":"Apple MacBook Pro 16英寸 M3 Pro","description":"Apple M3 Pro芯片, 18GB统一内存, 512GB存储","category":"笔记本","brand":"Apple","price":19999,"tags":["electronics","laptop","apple"],"status":"active","stock":50,"created_at":"2024-01-15T10:30:00Z"}
{"index":{"_id":2}}
{"title":"华为 MateBook X Pro 2024","description":"13.9英寸 3K触控屏, 32GB内存, 1TB SSD","category":"笔记本","brand":"华为","price":14999,"tags":["electronics","laptop","huawei"],"status":"active","stock":30,"created_at":"2024-01-20T14:00:00Z"}
{"index":{"_id":3}}
{"title":"iPhone 15 Pro Max 256GB","description":"A17 Pro芯片, 4800万像素主摄, 钛金属设计","category":"手机","brand":"Apple","price":9999,"tags":["electronics","phone","apple"],"status":"active","stock":100,"created_at":"2024-02-01T09:00:00Z"}
{"index":{"_id":4}}
{"title":"Samsung Galaxy S24 Ultra","description":"Snapdragon 8 Gen 3, 200MP相机, S Pen","category":"手机","brand":"Samsung","price":12999,"tags":["electronics","phone","samsung"],"status":"active","stock":80,"created_at":"2024-02-10T11:00:00Z"}
{"index":{"_id":5}}
{"title":"机械革命 极光Pro 游戏本","description":"RTX4060显卡, i7-12650H, 15.6英寸165Hz","category":"笔记本","brand":"机械革命","price":6999,"tags":["electronics","laptop","gaming"],"status":"active","stock":20,"created_at":"2024-03-01T16:00:00Z"}
{"index":{"_id":6}}
{"title":"Apple MacBook Air M2","description":"M2芯片, 13.6英寸, 8GB内存, 256GB存储","category":"笔记本","brand":"Apple","price":8999,"tags":["electronics","laptop","apple"],"status":"inactive","stock":0,"created_at":"2023-06-01T10:00:00Z"}

3. 基本关键词搜索

# 搜索"笔记本" (精确匹配标题)
GET /products/_search
{
  "query": {
    "match": {
      "title": "笔记本"
    }
  }
}
# 返回: MacBook Pro, MateBook X Pro, 极光Pro

# 多字段搜索 (标题+描述)
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "苹果笔记本 M3",
      "fields": ["title^3", "description"],
      "type": "best_fields"
    }
  }
}
# title 权重 3 倍, MacBook Pro 排最前

4. 综合搜索 (关键词 + 筛选 + 分页)

# 搜索"笔记本", 过滤价格 5000-15000, 只返回 active 商品
GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "笔记本" } }
      ],
      "filter": [
        { "term": { "status": "active" } },
        { "range": { "price": { "gte": 5000, "lte": 15000 } } }
      ]
    }
  },
  "sort": [
    { "price": { "order": "asc" } }
  ],
  "_source": ["title", "brand", "price", "stock"]
}
# 返回: 极光Pro (6999), MacBook Air (8999), MateBook X Pro (14999)

5. 搜索建议 (Completion Suggester)

PUT /products/_mapping
{
  "properties": {
    "title_suggest": {
      "type": "completion"
    }
  }
}

POST /products/_update/1
{
  "doc": {
    "title_suggest": ["Apple MacBook Pro", "MacBook Pro"]
  }
}

POST /products/_update/3
{
  "doc": {
    "title_suggest": ["iPhone 15 Pro Max", "iPhone 15"]
  }
}

# 搜索建议
GET /products/_search
{
  "suggest": {
    "product_suggest": {
      "prefix": "mac",
      "completion": {
        "field": "title_suggest",
        "size": 5
      }
    }
  }
}
# 返回: ["Apple MacBook Pro", "MacBook Pro"]

6. 高亮显示

GET /products/_search
{
  "query": {
    "match": { "title": "笔记本" }
  },
  "highlight": {
    "fields": {
      "title": {},
      "description": {}
    },
    "pre_tags": ["<em>"],
    "post_tags": ["</em>"]
  }
}
# title 中"笔记本"会被 <em> 标签包裹

完整搜索 API 示例

# 前端搜索框调用的完整 API
GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "multi_match": {
          "query": "笔记本",
          "fields": ["title^3", "description", "brand"],
          "type": "best_fields"
        }}
      ],
      "filter": [
        { "term": { "status": "active" } },
        { "terms": { "category": ["笔记本", "平板"] }},
        { "range": { "price": { "gte": 3000, "lte": 20000 } }},
        { "term": { "brand": "Apple" }}
      ],
      "should": [
        { "term": { "tags": "hot" }},
        { "term": { "is_new": true }}
      ]
    }
  },
  "sort": [
    { "_score": { "order": "desc" }},
    { "created_at": { "order": "desc" }}
  ],
  "from": 0,
  "size": 20,
  "_source": ["title", "brand", "price", "stock", "category"],
  "highlight": {
    "fields": { "title": { "number_of_fragments": 0 } }
  }
}

示例: 聚合报表 — 电商销售分析

场景

基于订单数据构建销售分析仪表盘：销售额趋势、TOP 品类、品牌分布、价格区段。

步骤

1. 创建索引

PUT /orders
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "order_id": { "type": "keyword" },
      "user_id": { "type": "keyword" },
      "product_id": { "type": "keyword" },
      "product_name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
      "category": { "type": "keyword" },
      "brand": { "type": "keyword" },
      "amount": { "type": "float" },
      "quantity": { "type": "integer" },
      "status": { "type": "keyword" },
      "region": { "type": "keyword" },
      "order_date": { "type": "date" },
      "payment_date": { "type": "date" }
    }
  }
}

2. 按品类统计销售额 (terms + sum)

# 每个品类的总销售额, 降序排列
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": {
        "field": "category",
        "size": 20,
        "order": { "total_sales": "desc" }
      },
      "aggs": {
        "total_sales": { "sum": { "field": "amount" } },
        "avg_order": { "avg": { "field": "amount" } },
        "order_count": { "value_count": { "field": "order_id" } }
      }
    }
  }
}

3. 每日销售趋势 (date_histogram + sum)

# 按天统计销售额
GET /orders/_search
{
  "size": 0,
  "query": {
    "range": {
      "order_date": { "gte": "now-30d", "lte": "now" }
    }
  },
  "aggs": {
    "daily_sales": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "format": "yyyy-MM-dd",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2024-01-01",
          "max": "2024-12-31"
        }
      },
      "aggs": {
        "revenue": { "sum": { "field": "amount" } },
        "orders": { "value_count": { "field": "order_id" } }
      }
    }
  }
}

4. 价格区段分布 (range + stats)

# 价格区段 + 每个区段的统计
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "amount",
        "ranges": [
          { "key": "低价 (<100)", "to": 100 },
          { "key": "中低价 (100-500)", "from": 100, "to": 500 },
          { "key": "中价 (500-2000)", "from": 500, "to": 2000 },
          { "key": "高价 (2000-10000)", "from": 2000, "to": 10000 },
          { "key": "超高 (>10000)", "from": 10000 }
        ]
      },
      "aggs": {
        "amount_stats": { "stats": { "field": "amount" } },
        "brand_distribution": {
          "terms": { "field": "brand", "size": 5 }
        }
      }
    }
  }
}

5. 地区分布 + TOP 品牌 (嵌套聚合)

# 各地区销售额, 每个地区 TOP 品牌
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "by_region": {
      "terms": {
        "field": "region",
        "size": 10,
        "order": { "revenue": "desc" }
      },
      "aggs": {
        "revenue": { "sum": { "field": "amount" } },
        "order_count": { "value_count": { "field": "order_id" } },
        "top_brands": {
          "terms": {
            "field": "brand",
            "size": 5,
            "order": { "brand_revenue": "desc" }
          },
          "aggs": {
            "brand_revenue": { "sum": { "field": "amount" } }
          }
        }
      }
    }
  }
}

6. 综合仪表盘查询 (一次查询完成多项分析)

# 一次查询返回多种聚合结果
GET /orders/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "term": { "status": "completed" } },
        { "range": { "order_date": { "gte": "now-30d/d", "lte": "now" } } }
      ]
    }
  },
  "aggs": {
    "overview": {        // 总体指标
      "stats": { "field": "amount" }
    },
    "total_revenue": {
      "sum": { "field": "amount" }
    },
    "unique_customers": {
      "cardinality": { "field": "user_id" }
    },
    "sales_trend": {     // 日销售趋势
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_revenue": { "sum": { "field": "amount" } }
      }
    },
    "top_categories": {  // TOP 品类
      "terms": { "field": "category", "size": 10 },
      "aggs": {
        "cat_revenue": { "sum": { "field": "amount" } }
      }
    },
    "price_distribution": {  // 价格分布
      "percentiles": {
        "field": "amount",
        "percents": [25, 50, 75, 90, 99]
      }
    }
  }
}
# 一次请求返回: 总览统计 + 总销售额 + 去重用户数 + 日趋势 + TOP 品类 + 价格百分位

示例: 零停机重建索引 (Zero-Downtime Reindex)

场景

需要修改已有索引的 mapping（如添加字段、修改分词器），但生产环境不能停机。

核心思路

Alias（别名） + Reindex（数据迁移） + 原子切换。

步骤

1. 假设已有索引和别名

# 初始状态: 已有 products_v1 索引, 通过 products 别名访问
GET /products/_search
{
  "query": { "match_all": {} }
}
# 别名可以透明访问

2. 创建新索引 (v2) 使用新的 mapping

PUT /products_v2
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1,
    "refresh_interval": "-1"
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 },
          "pinyin": { "type": "text", "analyzer": "pinyin" }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "ik_max_word"
      },
      "price": { "type": "double" },
      "new_field": { "type": "keyword" },
      "search_all": {
        "type": "text",
        "analyzer": "ik_max_word"
      }
    }
  }
}

3. 执行 Reindex 数据迁移

# 使用 slices=auto 并行加速
POST /_reindex?slices=auto&wait_for_completion=false
{
  "source": {
    "index": "products_v1"
  },
  "dest": {
    "index": "products_v2"
  },
  "script": {
    "source": """
      // 如果新索引有 copy_to 字段, 在脚本中处理
      if (ctx._source.title != null && ctx._source.description != null) {
        ctx._source.search_all = ctx._source.title + ' ' + ctx._source.description;
      }
      // 添加新字段默认值
      ctx._source.new_field = 'migrated';
    """
  }
}
# 返回 task id, 可用 _tasks API 查看进度
GET /_tasks/<task_id>

4. 原子切换别名 (零停机)

# 关键步骤: 一次操作移除旧别名 + 添加新别名
# 这期间已有查询不会中断
POST /_aliases
{
  "actions": [
    { "remove": { "index": "products_v1", "alias": "products" } },
    { "add": { "index": "products_v2", "alias": "products" } }
  ]
}
# 原子操作, 切换完成后所有通过 products 别名的查询自动指向 v2

5. 验证并删除旧索引

# 验证新索引数据完整
GET /products/_search
{
  "query": { "match_all": {} },
  "size": 0
}

# 确认无误后删除旧索引
DELETE /products_v1

Reindex 进阶技巧

跨集群 Reindex

POST /_reindex
{
  "source": {
    "remote": {
      "host": "http://old-cluster:9200",
      "username": "elastic",
      "password": "pass"
    },
    "index": "products"
  },
  "dest": {
    "index": "products_v2"
  }
}

只迁移部分数据

POST /_reindex
{
  "source": {
    "index": "products_v1",
    "query": {
      "term": { "status": "active" }
    }
  },
  "dest": {
    "index": "products_active"
  }
}

冲突处理

# 跳过已存在的文档 (op_type=create)
POST /_reindex
{
  "source": { "index": "products_v1" },
  "dest": {
    "index": "products_v2",
    "op_type": "create"
  }
}

# 或者使用 version_type=external 保留旧版本

Reindex 大小和速度

# 限制批次大小 (默认 1000)
POST /_reindex
{
  "source": {
    "index": "products_v1",
    "size": 5000
  },
  "dest": {
    "index": "products_v2"
  }
}

# 限制速率 (每秒文档数)
PUT /_cluster/settings
{
  "transient": {
    "indices.recovery.max_bytes_per_sec": "200mb"
  }
}

注意事项

1. 大索引 Reindex：建议 slices=auto 并行执行，可根据数据节点数自动调整并行度 2. 源索引不影响：Reindex 期间源索引可继续读写服务 3. Refresh 优化：新索引设 refresh_interval: -1，迁移完再恢复 4. 副本优化：新索引设 number_of_replicas: 0，迁移完再调回 5. 验证数据完整性：迁移后对比文档数 (_cat/count) 6. 善后清理：确认新索引正常后删除旧索引释放空间

示例: 集群监控实战

场景

日常运维需要监控 Elasticsearch 集群的健康状态、性能指标和资源使用情况。

1. 集群健康检查

快速健康状态

# 绿色 = 全部正常, 黄色 = 副本未分配, 红色 = 主分片丢失
GET /_cluster/health?pretty

{
  "cluster_name": "production",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 3,
  "active_primary_shards": 125,
  "active_shards": 240,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 10,
  "delayed_unassigned_shards": 0,
  "active_shards_percent_as_number": 96.0
}

解读：unassigned_shards=10, active_shards_percent=96% → 有副本未分配。检查节点是否宕机。

详细健康诊断

# 查看所有未分配分片的原因
GET /_cluster/allocation/explain?pretty

# 重点关注字段:
# - current_node: 当前所在节点
# - can_remain_on_current_node: 能否保留
# - can_rebalance_cluster: 能否再平衡
# - node_decision: 节点决策

2. 节点监控

节点概览

# 节点角色和资源使用
GET /_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,master

# 示例输出:
# name     node.role heap.percent ram.percent cpu load_1m disk.used_percent master
# master-1 m                  42          35   8     2.3               42 *
# data-1   d                  67          72  35    12.5               56 -
# data-2   d                  55          60  28     9.8               52 -
# coord-1  -                  48          45  15     4.1               38 -

告警阈值：

• heap.percent > 85 → GC 压力大
• cpu > 80 → 需要扩容
• disk.used_percent > 85 → 需要清理或 ILM

热点线程排查 (CPU 突增)

# 当集群 CPU 突然飙高时
GET /_nodes/hot_threads

# 返回每个节点的热点线程堆栈
# 重点关注:
# - bulk 线程 → 写入压力大
# - search 线程 → 查询压力大
# - merge 线程 → 段合并 (大批量写入后)
# - GC 线程 → 内存问题

节点统计

# 获取节点级别详细统计
GET /_nodes/stats/indices,os,process,jvm,fs,transport,http

# 关注指标:
# - indices.indexing: 写入速率
# - indices.search: 查询速率
# - jvm.mem.heap_used_percent: 堆内存使用
# - jvm.gc.collectors.young.collection_time_in_millis: GC 时间
# - os.cpu.percent: CPU
# - fs.total.available_in_bytes: 磁盘空间

3. 索引监控

索引概览

# 查看所有索引的大小和文档数
GET /_cat/indices?v&h=index,docs.count,store.size,pri.store.size

# 查看索引分片分布
GET /_cat/shards?v

# 按大小排序
GET /_cat/indices?v&s=store.size:desc

段 (Segment) 监控

# 检查段的统计
GET /my_index/_segments

# 查看各索引段数
GET /_cat/segments?v&h=index,shard,segment,size,committed,search

段过多 (>100) 表示需要 force_merge：POST /my_index/_forcemerge?max_num_segments=1

4. 性能监控脚本

Shell 健康检查脚本

#!/bin/bash
# es_health_check.sh
ES_HOST="http://localhost:9200"

# 集群健康
echo "=== 集群健康 ==="
curl -s "$ES_HOST/_cluster/health?pretty" | python3 -c "
import json, sys
h = json.load(sys.stdin)
print(f'状态: {h[\"status\"]}')
print(f'节点: {h[\"number_of_nodes\"]} (数据节点: {h[\"number_of_data_nodes\"]})')
print(f'活跃分片: {h[\"active_shards\"]}/{h[\"active_primary_shards\"]}p')
print(f'未分配: {h[\"unassigned_shards\"]}')
print(f'活跃率: {h[\"active_shards_percent_as_number\"]:.1f}%')
"

# 节点资源
echo -e "\n=== 节点资源 ==="
curl -s "$ES_HOST/_cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent"

# 索引 TOP 10 大索引
echo -e "\n=== TOP 10 大索引 ==="
curl -s "$ES_HOST/_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc&limit=10"

# JVM 堆内存 Top 节点
echo -e "\n=== JVM 内存 (Top 5) ==="
curl -s "$ES_HOST/_nodes/stats/jvm" | python3 -c "
import json, sys
nodes = json.load(sys.stdin)['nodes']
for node_id, info in sorted(nodes.items(), key=lambda x: x[1]['jvm']['mem']['heap_used_percent'], reverse=True)[:5]:
    name = info['name']
    mem = info['jvm']['mem']
    gc = info['jvm']['gc']['collectors']
    print(f'{name}: heap={mem[\"heap_used_percent\"]}%  old_gc={gc[\"old\"][\"collection_count\"]}次({gc[\"old\"][\"collection_time_in_millis\"]//1000}s)')
"

5. 告警规则建议

6. Kibana 监控配置

# 启用 Kibana 监控 UI (Stack Monitoring)
# Management → Stack Monitoring
# 可查看:
# - 集群概览 (节点/索引/分片)
# - 节点 CPU/内存/磁盘/IO
# - 索引搜索/写入速率
# - GC 统计
# - 慢查询 Top N

性能排查清单

1. 是否 Yellow/Red → 检查未分配分片 (allocation/explain)
2. CPU 高 → hot_threads 排查
3. 查询慢 → Profile API + 慢查询日志
4. 写入慢 → 检查 refresh_interval + Bulk 优化
5. 磁盘满 → ILM 清理 + force_merge
6. GC 频繁 → 减少分片/字段, 增加堆内存
7. Segments 多 → force_merge (只读索引)
8. 深度分页 → 改用 search_after

全文查询 (Full Text Queries)

全文查询会对查询文本进行分词，然后搜索倒排索引。这些查询会计算相关性分数 (_score)。

match — 标准全文匹配

# 基本 match (默认 OR 逻辑)
GET /products/_search
{
  "query": {
    "match": {
      "title": "apple macbook"
    }
  }
}

# AND 逻辑 (必须匹配所有词项)
GET /products/_search
{
  "query": {
    "match": {
      "title": {
        "query": "apple macbook",
        "operator": "and"
      }
    }
  }
}

# minimum_should_match (至少匹配 75% 词项)
GET /products/_search
{
  "query": {
    "match": {
      "title": {
        "query": "apple macbook pro",
        "minimum_should_match": "75%"
      }
    }
  }
}

业务场景：商品搜索、文章搜索、文档搜索。用户输入关键词时自动分词匹配。

match_phrase — 短语匹配

# 精确短语匹配 (顺序必须一致)
GET /products/_search
{
  "query": {
    "match_phrase": {
      "title": "macbook pro"
    }
  }
}

# 带 slop 的短语匹配 (允许词项间插入 slop 个词)
GET /products/_search
{
  "query": {
    "match_phrase": {
      "title": {
        "query": "macbook pro",
        "slop": 1
      }
    }
  }
}

业务场景：搜索"红米手机"不希望匹配"红色小米手机"；搜索完整书名/产品名。

multi_match — 多字段匹配

# 同时在 title 和 description 中搜索
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "轻薄笔记本",
      "fields": ["title", "description"]
    }
  }
}

# 带权重 (title 字段权重 2 倍)
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "轻薄笔记本",
      "fields": ["title^2", "description", "tags^0.5"]
    }
  }
}

# type 说明:
# - best_fields (默认): 取最佳匹配字段分数
# - most_fields: 合并所有匹配字段分数
# - cross_fields: 将词项拆分到多字段 (适合姓名搜索)
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "Will Smith",
      "fields": ["first_name", "last_name"],
      "type": "cross_fields"
    }
  }
}

业务场景：搜索框同时匹配标题、描述、标签等字段；人名搜索。

query_string — 完整查询语法

# 支持 AND/OR/NOT, 通配符, 正则等
GET /products/_search
{
  "query": {
    "query_string": {
      "query": "title:(apple AND macbook) AND price:[10000 TO 20000]",
      "default_operator": "and"
    }
  }
}

# 支持短语和多字段
GET /products/_search
{
  "query": {
    "query_string": {
      "query": "\"macbook pro\" OR \"iphone\"",
      "fields": ["title^2", "description"]
    }
  }
}

注意：query_string 解析语法错误会抛出异常，生产环境推荐用 simple_query_string。

simple_query_string — 安全版查询语法

GET /products/_search
{
  "query": {
    "simple_query_string": {
      "query": "\"macbook pro\" +apple -samsung",
      "fields": ["title", "description"],
      "default_operator": "or"
    }
  }
}

支持语法：+ (AND), | (OR), - (NOT), " (短语), * (前缀)。语法错误不会抛异常。

精确查询与复合查询

精确查询不会对搜索词进行分词，直接匹配倒排索引中的精确值。

term / terms — 精确值匹配

# 精确匹配 keyword 字段 (不要对 text 字段用 term!)
GET /products/_search
{
  "query": {
    "term": {
      "tags": "electronics"
    }
  }
}

# 多值匹配
GET /products/_search
{
  "query": {
    "terms": {
      "tags": ["electronics", "laptop"]
    }
  }
}

# ids 查询
GET /products/_search
{
  "query": {
    "ids": {
      "values": ["1", "2", "3"]
    }
  }
}

⚠️ 常见陷阱：term 查询对 text 字段几乎永远不匹配，因为 text 字段被分词后存储的是词项而非完整内容。

range — 范围查询

# 数值范围
GET /products/_search
{
  "query": {
    "range": {
      "price": { "gte": 10000, "lte": 20000 }
    }
  }
}

# 日期范围 (支持日期数学)
GET /products/_search
{
  "query": {
    "range": {
      "created_at": { "gte": "now-7d/d", "lte": "now" }
    }
  }
}

# 日期数学: now-1h, now+1d, now/d, now-1M/M, 2024-01-01||+1y

exists — 存在性检查

# 查找有 description 字段的文档
GET /products/_search
{
  "query": {
    "exists": { "field": "description" }
  }
}

# 查找没有 description 的文档 (using must_not)
GET /products/_search
{
  "query": {
    "bool": {
      "must_not": [
        { "exists": { "field": "description" } }
      ]
    }
  }
}

bool 查询 — 最常用的组合查询

# bool 结构: must (AND+算分), filter (AND+缓存), should (OR), must_not (NOT)
GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "轻薄笔记本" } }
      ],
      "filter": [
        { "term": { "status": "active" } },
        { "range": { "price": { "gte": 3000, "lte": 8000 } } }
      ],
      "should": [
        { "match": { "description": "轻薄" } }
      ],
      "minimum_should_match": 1
    }
  }
}

filter vs query 选择：

• filter：精确匹配、范围过滤 → 可缓存，不贡献算分
• query：全文搜索 → 贡献算分，不可缓存

nested — 嵌套对象查询

// mapping 定义
PUT /orders
{
  "mappings": {
    "properties": {
      "items": {
        "type": "nested",
        "properties": {
          "product_id": { "type": "keyword" },
          "quantity": { "type": "integer" },
          "price": { "type": "float" }
        }
      }
    }
  }
}

# nested 查询 (保证跨字段关联正确)
GET /orders/_search
{
  "query": {
    "nested": {
      "path": "items",
      "query": {
        "bool": {
          "must": [
            { "term": { "items.product_id": "p100" } },
            { "range": { "items.quantity": { "gte": 2 } } }
          ]
        }
      }
    }
  }
}

其他特殊查询

# fuzzy — 模糊纠错
GET /products/_search
{
  "query": {
    "fuzzy": {
      "title": {
        "value": "macbok",
        "fuzziness": "AUTO",
        "prefix_length": 2,
        "transpositions": true
      }
    }
  }
}
# "macbok" → 匹配 "macbook"

# wildcard — 通配符 (性能差, 谨慎使用)
GET /products/_search
{
  "query": {
    "wildcard": {
      "title.keyword": "Mac*"
    }
  }
}

# regexp — 正则查询 (性能开销大)
GET /products/_search
{
  "query": {
    "regexp": {
      "title.keyword": "Macbook\\s(Pro|Air)"
    }
  }
}

# boosting — 权重控制 (降权不排除)
GET /products/_search
{
  "query": {
    "boosting": {
      "positive": { "match": { "title": "手机" } },
      "negative": { "term": { "status": "discontinued" } },
      "negative_boost": 0.2
    }
  }
}

指标聚合 (Metric Aggregations)

指标聚合对文档集中的某个字段进行数值计算，类似于 SQL 聚合函数。

基本结构

{
  "size": 0,
  "aggs": {
    "my_agg_name": {       // 自定义聚合名称
      "avg": {              // 聚合类型
        "field": "price"
      }
    }
  }
}

常用指标聚合

avg — 平均值

GET /products/_search
{
  "size": 0,
  "aggs": {
    "avg_price": {
      "avg": { "field": "price" }
    }
  }
}

sum / min / max

GET /products/_search
{
  "size": 0,
  "aggs": {
    "total_revenue": { "sum": { "field": "price" } },
    "min_price": { "min": { "field": "price" } },
    "max_price": { "max": { "field": "price" } }
  }
}

stats — 批量统计

# 一次性返回 count/min/max/avg/sum
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_stats": {
      "stats": { "field": "price" }
    }
  }
}

extended_stats — 扩展统计

# 额外返回方差、标准差、总和平方等
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_extended": {
      "extended_stats": { "field": "price" }
    }
  }
}

cardinality — 去重计数 (COUNT DISTINCT)

GET /products/_search
{
  "size": 0,
  "aggs": {
    "unique_brands": {
      "cardinality": {
        "field": "brand",
        "precision_threshold": 100   // 精度阈值 (默认 3000)
      }
    }
  }
}

说明：cardinality 基于 HyperLogLog++ 算法，近似去重。precision_threshold 越高越精确，但消耗更多内存。

value_count — 非空值计数

GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand" },
      "aggs": {
        "price_count": { "value_count": { "field": "price" } }
      }
    }
  }
}

通常作为子聚合，用于计算每个桶的样本数。

percentiles — 百分位

GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_percentiles": {
      "percentiles": {
        "field": "price",
        "percents": [1, 5, 25, 50, 75, 95, 99]
      }
    }
  }
}

percentile_ranks — 百分位排名

# 值落在指定阈值内的百分比
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_ranks": {
      "percentile_ranks": {
        "field": "price",
        "values": [5000, 10000]
      }
    }
  }
}

业务场景速查

桶聚合 (Bucket Aggregations)

桶聚合将文档分组到不同的"桶"中，类似 SQL 的 GROUP BY。每个桶可以嵌套子聚合。

terms — 分组聚合 (GROUP BY)

# 按标签分组统计
GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_tags": {
      "terms": {
        "field": "tags",
        "size": 20,
        "order": { "_count": "desc" }
      }
    }
  }
}

# 带子聚合: 每个分组的平均价格
GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_brand": {
      "terms": {
        "field": "brand",
        "size": 10,
        "order": { "avg_price": "desc" }
      },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } },
        "product_count": { "value_count": { "field": "id" } }
      }
    }
  }
}

注意事项：

• size 控制返回桶数（默认 10），超大 size 消耗内存
• order 支持按文档数 (_count)、聚合键 (_key)、子聚合排序
• keyword 字段上聚合性能最佳，text 字段需开启 fielddata

range / date_range — 范围分组

# 价格区间分组
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "key": "0-1000", "from": 0, "to": 1000 },
          { "key": "1000-5000", "from": 1000, "to": 5000 },
          { "key": "5000+", "from": 5000 }
        ]
      }
    }
  }
}

# 日期范围
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "date_ranges": {
      "date_range": {
        "field": "order_date",
        "ranges": [
          { "from": "now-30d/d", "to": "now" },
          { "from": "now-90d/d", "to": "now-30d/d" }
        ]
      }
    }
  }
}

histogram / date_histogram — 直方图

# 价格直方图 (间隔 1000)
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_histogram": {
      "histogram": {
        "field": "price",
        "interval": 1000,
        "min_doc_count": 1
      }
    }
  }
}

# 时间直方图 (按小时/天/月/年聚合)
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "orders_over_time": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",     // month, quarter, year
        "format": "yyyy-MM-dd",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2024-01-01",
          "max": "2024-12-31"
        }
      }
    }
  }
}

filter / filters — 过滤聚合

# 单一过滤
GET /products/_search
{
  "size": 0,
  "aggs": {
    "active_products": {
      "filter": { "term": { "status": "active" } },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}

# 多过滤
GET /products/_search
{
  "size": 0,
  "aggs": {
    "price_categories": {
      "filters": {
        "other_bucket": true,
        "filters": {
          "budget":  { "range": { "price": { "lte": 5000 } } },
          "mid":     { "range": { "price": { "from": 5000, "to": 15000 } } },
          "premium": { "range": { "price": { "gte": 15000 } } }
        }
      }
    }
  }
}

管道聚合 (Pipeline Aggregations)

# derivative — 环比增量
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "sales_per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_sales": { "sum": { "field": "amount" } },
        "sales_derivative": {
          "derivative": { "buckets_path": "daily_sales" }
        }
      }
    }
  }
}

# bucket_script — 计算桶间比例
GET /products/_search
{
  "size": 0,
  "aggs": {
    "total_products": { "value_count": { "field": "id" } },
    "active_products": {
      "filter": { "term": { "status": "active" } }
    },
    "active_ratio": {
      "bucket_script": {
        "buckets_path": {
          "activeCount": "active_products>_count",
          "totalCount": "total_products"
        },
        "script": "params.activeCount / params.totalCount * 100"
      }
    }
  }
}

# bucket_selector — 过滤桶 (类似 HAVING)
GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand", "size": 100 },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } },
        "brands_having_avg_gt_10000": {
          "bucket_selector": {
            "buckets_path": { "avgPrice": "avg_price" },
            "script": "params.avgPrice > 10000"
          }
        }
      }
    }
  }
}

业务场景速查

映射与字段类型详解

Mapping 定义文档中每个字段的数据类型和分析方式，相当于关系型数据库的 Schema。

字段类型速查

动态映射 (Dynamic Mapping)

映射参数详解

PUT /articles
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          },
          "pinyin": {
            "type": "text",
            "analyzer": "pinyin"
          }
        }
      },
      "price": {
        "type": "float",
        "coerce": false
      },
      "meta_data": {
        "type": "flattened"
      }
    }
  }
}

关键参数

别名 (Alias) 与索引模板

// 原子切换别名 (零停机重建)
POST /_aliases
{
  "actions": [
    { "remove": { "index": "products_v1", "alias": "products" } },
    { "add": { "index": "products_v2", "alias": "products" } }
  ]
}

// 索引模板
PUT /_index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" }
      }
    }
  }
}

索引设置

PUT /my_index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "30s",
    "max_result_window": 100000,
    "codec": "best_compression"
  },
  "mappings": { ... }
}

重要：number_of_shards 建后不可修改。refresh_interval 大批量写入时可设为 -1（关闭）。

分词器 (Analyzer)

分词器将文本拆分为词项 (Token)，用于建立倒排索引和搜索。

Analyzer 组成

Analyzer = Char Filter(s) + Tokenizer + Token Filter(s)
           (字符过滤)      (分词器)     (词项过滤)

输入: "I <3 Elasticsearch!"
    ↓
char_filter: HTML 解码 → "I <3 Elasticsearch!"
    ↓
tokenizer: 按空格/标点拆分 → ["I", "<3", "Elasticsearch"]
    ↓
token_filter: 小写化 → ["i", "<3", "elasticsearch"]
    ↓
输出: ["i", "<3", "elasticsearch"] (存入倒排索引)

测试分词器

# 测试分析器效果
POST /_analyze
{
  "analyzer": "standard",
  "text": "I love Elasticsearch 搜索引擎"
}

# 指定字段测试 (使用字段配置的分析器)
POST /products/_analyze
{
  "field": "title",
  "text": "Apple MacBook Pro 16英寸"
}

内置分词器

IK 分词器 (中文)

# 安装 (需重启 ES)
./bin/elasticsearch-plugin install \
  https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v8.12.0/elasticsearch-analysis-ik-8.12.0.zip

# ik_smart (粗粒度)
POST /_analyze { "analyzer": "ik_smart", "text": "中华人民共和国国歌" }
# → ["中华人民共和国", "国歌"]

# ik_max_word (细粒度, 穷尽所有可能)
POST /_analyze { "analyzer": "ik_max_word", "text": "中华人民共和国国歌" }
# → ["中华人民共和国", "中华人民", "中华", "华人", "人民共和国", "人民", "共和国", "共和", "国歌"]

拼音分词器

./bin/elasticsearch-plugin install \
  https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v8.12.0/elasticsearch-analysis-pinyin-8.12.0.zip

# 测试
POST /_analyze { "analyzer": "pinyin", "text": "王大力" }
# → ["wang", "da", "li", "wangda", "dali", "wangdali", "wdl"]

# 拼音字段配置
PUT /products
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "fields": {
          "pinyin": {
            "type": "text",
            "analyzer": "pinyin",
            "search_analyzer": "pinyin"
          }
        }
      }
    }
  }
}
# 搜索 "wdali" → 匹配 "王大力"

自定义分析器

PUT /my_index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "html_strip": { "type": "html_strip" },
        "my_mapping": {
          "type": "mapping",
          "mappings": ["& => and", "| => or"]
        }
      },
      "tokenizer": {
        "my_standard": { "type": "standard", "max_token_length": 100 }
      },
      "filter": {
        "my_stop": {
          "type": "stop",
          "stopwords": ["a", "an", "the", "is"]
        },
        "my_synonym": {
          "type": "synonym",
          "synonyms": [
            "笔记本, 笔记本电脑, laptop",
            "手机, 移动电话, smartphone"
          ]
        }
      },
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip", "my_mapping"],
          "tokenizer": "my_standard",
          "filter": ["lowercase", "my_stop", "my_synonym", "asciifolding"]
        }
      }
    }
  }
}

自动补全 (Edge Ngram)

PUT /autocomplete_index
{
  "settings": {
    "analysis": {
      "filter": {
        "autocomplete_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20
        }
      },
      "analyzer": {
        "autocomplete": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "autocomplete_filter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "standard"
      }
    }
  }
}

同义词注意事项

• 单向同义词: "laptop => 笔记本" (搜索 laptop 映射到笔记本)
• 双向同义词: "laptop, 笔记本" (互相等价)
• 最佳实践: 索引时不用同义词（保持原始词），搜索时用 search_analyzer

集群运维

节点类型

生产环境推荐：3 个专用 Master + N 个 Data + 2 个 Coordinating。

发现与选主

# elasticsearch.yml
discovery.seed_hosts: ["node1:9300", "node2:9300", "node3:9300"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

防脑裂：专用 Master 节点、奇数个 (≥3)、堆内存 ≤50% 物理内存。

分片分配与再平衡

# 查看分片分配
GET /_cat/shards?v

# 延迟分片分配 (重启节点时)
PUT /_all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}

# 取消再平衡 (维护时)
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.rebalance.enable": "none"
  }
}

# 重新路由
POST /_cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "my_index",
        "shard": 0,
        "from_node": "node-1",
        "to_node": "node-2"
      }
    }
  ]
}

快照与恢复 (Snapshot/Restore)

# 1. 注册快照仓库
PUT /_snapshot/my_backup
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-backups",
    "region": "us-east-1",
    "base_path": "elasticsearch/backups"
  }
}

# 2. 创建快照
PUT /_snapshot/my_backup/snapshot_20240101
{
  "indices": "logs-*,products",
  "ignore_unavailable": true,
  "include_global_state": true
}

# 3. 查看状态
GET /_snapshot/my_backup/snapshot_20240101/_status

# 4. 恢复
POST /_snapshot/my_backup/snapshot_20240101/_restore
{
  "indices": "products",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1"
}

# 5. SLM 自动管理
PUT /_slm/policy/daily_snapshot
{
  "name": "<daily-snap-{now/d}>",
  "repository": "my_backup",
  "schedule": "0 30 1 * * ?",
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

ILM — 索引生命周期管理

PUT /_ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d",
            "max_docs": 5000000
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": { "require": { "box_type": "warm" } }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_backup"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

生命周期：hot (热节点 SSD) → warm (温节点 HDD) → cold (可搜索快照) → delete。

监控命令

# 集群健康
GET /_cluster/health?pretty

# 节点信息
GET /_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,load_1m,master

# 索引信息
GET /_cat/indices?v&h=index,docs.count,store.size,pri.store.size
GET /_cat/shards?v

# 热点线程 (排查 CPU 突增)
GET /_nodes/hot_threads

# 待处理任务
GET /_cat/pending_tasks

# 节点统计
GET /_nodes/stats/indices,os,process,jvm,fs,transport,http

# 任务管理
GET /_tasks?detailed&actions=*byquery
POST /_tasks/<task_id>/_cancel

性能优化

分片黄金法则

• 每个分片 20-50GB (最佳)
• 每 GB 堆内存 20-25 个分片 (含副本)
• 案例: 1TB → 20-25 主分片 × 1 副本 = 40-50 总分片

写入优化

1. 关闭 refresh (refresh_interval: -1) 2. 增加 translog 同步间隔 (30s) 3. Bulk API (5-15MB/批) 4. 副本数设为 0 (写完恢复) 5. 合理 mapping (关闭 norms 等)

查询优化

• filter 优先 (可缓存, 不贡献算分)
• 避免 script 查询
• search_after 替代深度 from+size
• 限制 _source 返回字段

慢查询日志

PUT /_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.query.info": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s"
}

Profile API

GET /products/_search
{
  "profile": true,
  "query": {
    "match": { "title": "手机" }
  }
}

ELK Stack 集成

Elastic Stack 架构

Kibana (可视化/仪表盘/Dev Tools)
    ↓
Elasticsearch (存储/搜索/聚合)
    ↓
Logstash (ETL)  ←  Filebeat (日志)  ←  Metricbeat (指标)  ←  其他 Beats

Logstash 配置

# logstash.conf — 接收文件日志, 解析后写入 ES
input {
  beats {
    port => 5044
  }
  file {
    path => "/var/log/app/*.log"
    start_position => "beginning"
  }
}

filter {
  # 解析 Nginx 日志
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  # 解析 JSON 格式
  json {
    source => "message"
    target => "parsed"
    skip_on_invalid_json => true
  }
  # 添加时间戳
  date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
  }
  # 地理 IP 解析
  geoip {
    source => "client_ip"
    target => "geo"
  }
  # 字段处理
  mutate {
    remove_field => ["message", "original"]
    convert => ["response", "integer"]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${ES_PASSWORD}"
    ssl => true
    cacert => "/etc/elasticsearch/certs/ca.crt"
  }
}

Logstash 是 ETL 工具，适合复杂数据转换（grok/geoip/useragent）。纯日志采集场景推荐 Filebeat（更轻量）。

Filebeat 配置

# filebeat.yml — 轻量级日志采集
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    fields:
      service: nginx
      env: production
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    multiline:
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after       # 合并多行异常堆栈

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "${ES_PASSWORD}"
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.kibana:
  host: "localhost:5601"

Filebeat 特点：轻量级、资源占用低、支持多行合并 (Java 异常堆栈)、模块化配置。

Metricbeat 配置

# metricbeat.yml — 系统和应用指标采集
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
      - network
      - diskio
      - filesystem
      - process
    period: 10s
    enabled: true
  - module: elasticsearch
    metricsets:
      - node
      - node_stats
      - cluster_stats
      - index
    period: 10s
    hosts: ["localhost:9200"]

Kibana 核心功能

KQL 查询语法

service.name: "my-app" AND http.response.status_code >= 400
agent.hostname: "web-*" AND NOT response_time > 5s

实战: 构建 Nginx 日志分析管道

1. Filebeat 采集 /var/log/nginx/access.log 2. Logstash (可选) 解析日志 → grok 提取字段 + geoip 解析 IP 3. Elasticsearch 存储到 nginx-logs-YYYY.MM.dd 索引 (ILM 管理) 4. Kibana 创建 Dashboard: 请求量趋势、状态码分布、TOP URL、地理分布

完整 pipeline 参考：

# 使用 Filebeat Nginx 模块 (免手动配置)
filebeat modules enable nginx
filebeat setup