跳到主要内容

第6章 实战案例应用

6.1 电商搜索系统实战

业务需求分析

  1. 商品搜索功能

    • 支持中文商品名称全文搜索
    • 多维度筛选(品牌、类别、价格区间)
    • 搜索结果排序和分页
    • 搜索建议和自动补全
  2. 数据同步策略

    • MySQL作为主数据存储
    • Elasticsearch作为搜索引擎
    • 实时数据同步机制
    • 数据一致性保障
  3. 性能要求

    • 搜索响应时间 < 100ms
    • 支持高并发查询
    • 数据更新延迟 < 5s

数据架构设计

products 表结构

字段名数据类型描述
idINT自增主键
nameVARCHAR(255)商品名称(中文)
categoryENUM商品类别(英文)
brandVARCHAR(100)品牌(英文)
priceDECIMAL(10, 2)价格
descriptionTEXT商品描述(英文)
specificationsJSON商品规格(英文)
created_atTIMESTAMP创建时间
updated_atTIMESTAMP更新时间

示例数据

idnamecategorybrandpricedescriptionspecificationscreated_atupdated_at
1Apple/苹果 iPhone 16 Pro(A3294)256GB 原色钛金属 支持移动联通电信5G 双卡双待手机phoneApple8999.00Latest iPhone with A18 chip, 5G support.{"screen_size": "6.1 inches", "ram": "8GB", "storage": "256GB", "color": "Titanium"}2023-10-01 10:00:002023-10-01 10:00:00
2华为 Mate 60 Pro 12GB+512GB 雅川青 支持卫星通话 鸿蒙OS 4.0 5G手机phoneHuawei6999.00Huawei flagship with satellite calling.{"screen_size": "6.8 inches", "ram": "12GB", "storage": "512GB", "color": "Green"}2023-10-01 10:00:002023-10-01 10:00:00
3Apple/苹果AI笔记本/2024MacBookPro14英寸M4(10+10核) 16G 512G深空黑色笔记本电脑laptopApple14999.002024 MacBook Pro with M4 chip.{"screen_size": "14 inches", "processor": "M4 (10+10 cores)", "ram": "16GB", "storage": "512GB", "color": "Space Black"}2023-10-01 10:00:002023-10-01 10:00:00
4联想拯救者Y9000P 2024 AI元启 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4060 碳晶灰laptopLenovo12999.00Lenovo gaming laptop with i9 and RTX4060.{"screen_size": "16 inches", "processor": "i9-14900HX", "ram": "32GB", "storage": "1TB", "graphics": "RTX4060", "color": "Carbon Gray"}2023-10-01 10:00:002023-10-01 10:00:00
5Apple/苹果 iPhone 15 Pro Max(A3295)512GB 金色 支持移动联通电信5G 双卡双待手机phoneApple10999.00Latest iPhone with A17 chip, 5G support.{"screen_size": "6.7 inches", "ram": "8GB", "storage": "512GB", "color": "Gold"}2023-10-01 10:00:002023-10-01 10:00:00
6Apple/苹果 iPhone 14 Plus(A3293)128GB 星光色 支持移动联通电信5G 双卡双待手机phoneApple6999.00Affordable iPhone with A16 chip, 5G support.{"screen_size": "6.7 inches", "ram": "6GB", "storage": "128GB", "color": "Starlight"}2023-10-01 10:00:002023-10-01 10:00:00
7Apple/苹果 iPhone SE 2024(A3296)64GB 午夜黑 支持移动联通电信5G 双卡双待手机phoneApple3999.00Compact iPhone with A16 chip, 5G support.{"screen_size": "4.7 inches", "ram": "4GB", "storage": "64GB", "color": "Midnight"}2023-10-01 10:00:002023-10-01 10:00:00
8Apple/苹果AI笔记本/2024MacBook Air13英寸M3(8+8核) 16G 512G星光色笔记本电脑laptopApple9999.002024 MacBook Air with M3 chip.{"screen_size": "13 inches", "processor": "M3 (8+8 cores)", "ram": "16GB", "storage": "512GB", "color": "Starlight"}2023-10-01 10:00:002023-10-01 10:00:00
9Apple/苹果AI笔记本/2024MacBook Pro16英寸M4(12+12核) 32G 1T深空灰色笔记本电脑laptopApple19999.002024 MacBook Pro with M4 chip.{"screen_size": "16 inches", "processor": "M4 (12+12 cores)", "ram": "32GB", "storage": "1TB", "color": "Space Gray"}2023-10-01 10:00:002023-10-01 10:00:00
10华硕 ROG 幻16 2024 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4070 月耀白laptopASUS14999.00ASUS gaming laptop with i9 and RTX4070.{"screen_size": "16 inches", "processor": "i9-14900HX", "ram": "32GB", "storage": "1TB", "graphics": "RTX4070", "color": "Moonlight White"}2023-10-01 10:00:002023-10-01 10:00:00
11戴尔 Alienware m18 2024 18英寸电竞游戏本笔记本电脑 64G+2T 2.5k 240Hz i9-14900HX RTX4080 暗影黑laptopDell19999.00Dell gaming laptop with i9 and RTX4080.{"screen_size": "18 inches", "processor": "i9-14900HX", "ram": "64GB", "storage": "2TB", "graphics": "RTX4080", "color": "Shadow Black"}2023-10-01 10:00:002023-10-01 10:00:00

Elasticsearch索引设计

# 创建商品索引
PUT /products
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"ik_max_word_analyzer": {
"tokenizer": "ik_max_word"
},
"ik_smart_analyzer": {
"tokenizer": "ik_smart"
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "text",
"analyzer": "ik_max_word_analyzer",
"search_analyzer": "ik_smart_analyzer"
},
"category": {
"type": "keyword"
},
"brand": {
"type": "keyword"
},
"price": {
"type": "double"
},
"description": {
"type": "text",
"analyzer": "english"
},
"specifications": {
"type": "object",
"properties": {
"screen_size": {"type": "keyword"},
"ram": {"type": "keyword"},
"storage": {"type": "keyword"},
"processor": {"type": "keyword"},
"graphics": {"type": "keyword"},
"color": {"type": "keyword"}
}
},
"created_at": {
"type": "date"
},
"updated_at": {
"type": "date"
}
}
}
}

示例数据导入


# 插入 Apple/苹果 iPhone 16 Pro(A3294)256GB 原色钛金属 支持移动联通电信5G 双卡双待手机

POST /products/_doc/
{
"id": 1,
"name": "Apple/苹果 iPhone 16 Pro(A3294)256GB 原色钛金属 支持移动联通电信5G 双卡双待手机",
"category": "phone",
"brand": "Apple",
"price": 8999.00,
"description": "Latest iPhone with A18 chip, 5G support.",
"specifications": {
"screen_size": "6.1 inches",
"ram": "8GB",
"storage": "256GB",
"color": "Titanium"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 华为 Mate 60 Pro 12GB+512GB 雅川青 支持卫星通话 鸿蒙OS 4.0 5G手机

POST /products/_doc/
{
"id": 2,
"name": "华为 Mate 60 Pro 12GB+512GB 雅川青 支持卫星通话 鸿蒙OS 4.0 5G手机",
"category": "phone",
"brand": "Huawei",
"price": 6999.00,
"description": "Huawei flagship with satellite calling.",
"specifications": {
"screen_size": "6.8 inches",
"ram": "12GB",
"storage": "512GB",
"color": "Green"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果AI笔记本/2024MacBookPro14英寸M4(10+10核) 16G 512G深空黑色笔记本电脑

POST /products/_doc/
{
"id": 3,
"name": "Apple/苹果AI笔记本/2024MacBookPro14英寸M4(10+10核) 16G 512G深空黑色笔记本电脑",
"category": "laptop",
"brand": "Apple",
"price": 14999.00,
"description": "2024 MacBook Pro with M4 chip.",
"specifications": {
"screen_size": "14 inches",
"processor": "M4 (10+10 cores)",
"ram": "16GB",
"storage": "512GB",
"color": "Space Black"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 联想拯救者Y9000P 2024 AI元启 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4060 碳晶灰

POST /products/_doc/
{
"id": 4,
"name": "联想拯救者Y9000P 2024 AI元启 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4060 碳晶灰",
"category": "laptop",
"brand": "Lenovo",
"price": 12999.00,
"description": "Lenovo gaming laptop with i9 and RTX4060.",
"specifications": {
"screen_size": "16 inches",
"processor": "i9-14900HX",
"ram": "32GB",
"storage": "1TB",
"graphics": "RTX4060",
"color": "Carbon Gray"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果 iPhone 15 Pro Max(A3295)512GB 金色 支持移动联通电信5G 双卡双待手机

POST /products/_doc/
{
"id": 5,
"name": "Apple/苹果 iPhone 15 Pro Max(A3295)512GB 金色 支持移动联通电信5G 双卡双待手机",
"category": "phone",
"brand": "Apple",
"price": 10999.00,
"description": "Latest iPhone with A17 chip, 5G support.",
"specifications": {
"screen_size": "6.7 inches",
"ram": "8GB",
"storage": "512GB",
"color": "Gold"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果 iPhone 14 Plus(A3293)128GB 星光色 支持移动联通电信5G 双卡双待手机

POST /products/_doc/
{
"id": 6,
"name": "Apple/苹果 iPhone 14 Plus(A3293)128GB 星光色 支持移动联通电信5G 双卡双待手机",
"category": "phone",
"brand": "Apple",
"price": 6999.00,
"description": "Affordable iPhone with A16 chip, 5G support.",
"specifications": {
"screen_size": "6.7 inches",
"ram": "6GB",
"storage": "128GB",
"color": "Starlight"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果 iPhone SE 2024(A3296)64GB 午夜黑 支持移动联通电信5G 双卡双待手机

POST /products/_doc/
{
"id": 7,
"name": "Apple/苹果 iPhone SE 2024(A3296)64GB 午夜黑 支持移动联通电信5G 双卡双待手机",
"category": "phone",
"brand": "Apple",
"price": 3999.00,
"description": "Compact iPhone with A16 chip, 5G support.",
"specifications": {
"screen_size": "4.7 inches",
"ram": "4GB",
"storage": "64GB",
"color": "Midnight"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果AI笔记本/2024MacBook Air13英寸M3(8+8核) 16G 512G星光色笔记本电脑

POST /products/_doc/
{
"id": 8,
"name": "Apple/苹果AI笔记本/2024MacBook Air13英寸M3(8+8核) 16G 512G星光色笔记本电脑",
"category": "laptop",
"brand": "Apple",
"price": 9999.00,
"description": "2024 MacBook Air with M3 chip.",
"specifications": {
"screen_size": "13 inches",
"processor": "M3 (8+8 cores)",
"ram": "16GB",
"storage": "512GB",
"color": "Starlight"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 Apple/苹果AI笔记本/2024MacBook Pro16英寸M4(12+12核) 32G 1T深空灰色笔记本电脑

POST /products/_doc/
{
"id": 9,
"name": "Apple/苹果AI笔记本/2024MacBook Pro16英寸M4(12+12核) 32G 1T深空灰色笔记本电脑",
"category": "laptop",
"brand": "Apple",
"price": 19999.00,
"description": "2024 MacBook Pro with M4 chip.",
"specifications": {
"screen_size": "16 inches",
"processor": "M4 (12+12 cores)",
"ram": "32GB",
"storage": "1TB",
"color": "Space Gray"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 华硕 ROG 幻16 2024 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4070 月耀白

POST /products/_doc/
{
"id": 10,
"name": "华硕 ROG 幻16 2024 16英寸电竞游戏本笔记本电脑 32G+1T 2.5k 240Hz i9-14900HX RTX4070 月耀白",
"category": "laptop",
"brand": "ASUS",
"price": 14999.00,
"description": "ASUS gaming laptop with i9 and RTX4070.",
"specifications": {
"screen_size": "16 inches",
"processor": "i9-14900HX",
"ram": "32GB",
"storage": "1TB",
"graphics": "RTX4070",
"color": "Moonlight White"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}


# 插入 戴尔 Alienware m18 2024 18英寸电竞游戏本笔记本电脑 64G+2T 2.5k 240Hz i9-14900HX RTX4080 暗影黑

POST /products/_doc/
{
"id": 11,
"name": "戴尔 Alienware m18 2024 18英寸电竞游戏本笔记本电脑 64G+2T 2.5k 240Hz i9-14900HX RTX4080 暗影黑",
"category": "laptop",
"brand": "Dell",
"price": 19999.00,
"description": "Dell gaming laptop with i9 and RTX4080.",
"specifications": {
"screen_size": "18 inches",
"processor": "i9-14900HX",
"ram": "64GB",
"storage": "2TB",
"graphics": "RTX4080",
"color": "Shadow Black"
},
"created_at": "2023-10-01T10:00:00Z",
"updated_at": "2023-10-01T10:00:00Z"
}

搜索功能实现

基础搜索查询


# 查看索引信息

GET /_cat/indices


# 查看所有数据

GET /products/_search


# 根据指定ID查询

GET /products/_doc/OiCYgpQBoNDYZpxwux6n


# 根据品牌单一查询

GET /products/_search
{
"query": {
"term": {
"brand.keyword": {
"value": "Apple"
}
}
}
}


# 根据条件多个查询

GET /products/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"brand.keyword": "Apple"
}
},
{
"match": {
"category.keyword": "phone"
}
},
{
"range": {
"price": {
"gte": 8000,
"lte": 10000
}
}
}
]
}
}
}

高级搜索功能

# 搜索建议(自动补全)
GET /products/_search
{
"suggest": {
"product_suggest": {
"prefix": "苹果",
"completion": {
"field": "name_suggest",
"size": 5
}
}
}
}

# 聚合统计
GET /products/_search
{
"size": 0,
"aggs": {
"brands": {
"terms": {
"field": "brand",
"size": 10
}
},
"categories": {
"terms": {
"field": "category",
"size": 10
}
},
"price_ranges": {
"range": {
"field": "price",
"ranges": [
{"to": 5000},
{"from": 5000, "to": 10000},
{"from": 10000, "to": 20000},
{"from": 20000}
]
}
}
}
}

数据同步实现

# Python数据同步脚本示例
from elasticsearch import Elasticsearch
import mysql.connector
import json

class ProductSync:
def __init__(self):
self.es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
self.mysql = mysql.connector.connect(
host='localhost',
user='root',
password='password',
database='ecommerce'
)

def sync_product(self, product_id):
# 从MySQL获取数据
cursor = self.mysql.cursor(dictionary=True)
cursor.execute("SELECT * FROM products WHERE id = %s", (product_id,))
product = cursor.fetchone()

if product:
# 同步到Elasticsearch
doc = {
'id': product['id'],
'name': product['name'],
'category': product['category'],
'brand': product['brand'],
'price': float(product['price']),
'description': product['description'],
'specifications': json.loads(product['specifications']),
'created_at': product['created_at'].isoformat(),
'updated_at': product['updated_at'].isoformat()
}

self.es.index(index='products', body=doc)
print(f"Product {product_id} synced successfully")

6.2 日志分析平台实战

ELK Stack架构设计

系统架构

应用服务器 → Filebeat → Logstash → Elasticsearch → Kibana
↓ ↓ ↓ ↓ ↓
日志文件 日志采集 数据处理 数据存储 数据可视化

Filebeat配置

# filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
- /var/log/nginx/error.log
fields:
service: nginx
environment: production

- type: log
enabled: true
paths:
- /var/log/application/*.log
fields:
service: application
environment: production
multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after

output.logstash:
hosts: ["localhost:5044"]

processors:
- add_host_metadata:
when.not.contains.tags: forwarded

Logstash数据处理

# logstash.conf
input {
beats {
port => 5044
}
}

filter {
if [fields][service] == "nginx" {
grok {
match => {
"message" => "%{NGINXACCESS}"
}
}

date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}

mutate {
convert => {
"response" => "integer"
"bytes" => "integer"
"response_time" => "float"
}
}
}

if [fields][service] == "application" {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:level}\] %{GREEDYDATA:log_message}"
}
}

date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{[fields][service]}-%{+YYYY.MM.dd}"
}

stdout { codec => rubydebug }
}

Elasticsearch索引模板

# 日志索引模板
PUT /_index_template/logs-template
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.lifecycle.name": "logs-policy",
"index.lifecycle.rollover_alias": "logs"
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"level": {
"type": "keyword"
},
"service": {
"type": "keyword"
},
"environment": {
"type": "keyword"
},
"host": {
"properties": {
"name": {"type": "keyword"}
}
},
"message": {
"type": "text",
"analyzer": "standard"
},
"response": {
"type": "integer"
},
"bytes": {
"type": "long"
},
"response_time": {
"type": "float"
}
}
}
}
}

常用日志分析查询

# 错误日志统计
GET /logs-*/_search
{
"query": {
"bool": {
"must": [
{"range": {"@timestamp": {"gte": "now-1h"}}},
{"term": {"level": "ERROR"}}
]
}
},
"aggs": {
"error_count_by_service": {
"terms": {
"field": "service",
"size": 10
}
},
"error_timeline": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "5m"
}
}
}
}

# 响应时间分析
GET /logs-nginx-*/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "now-1h"
}
}
},
"aggs": {
"response_time_stats": {
"stats": {
"field": "response_time"
}
},
"slow_requests": {
"filter": {
"range": {
"response_time": {
"gt": 1.0
}
}
}
}
}
}

6.3 企业文档搜索实战

文档管理需求

业务场景

  • 企业内部文档知识库
  • 多格式文件支持(PDF、Word、PPT、TXT)
  • 权限控制和安全搜索
  • 相关性排序和推荐

技术架构

文件上传 → 文档解析 → 内容提取 → Elasticsearch索引 → 搜索API
↓ ↓ ↓ ↓ ↓
前端界面 文档处理 文本分析 数据存储 搜索结果

文档索引设计

# 创建文档索引
PUT /documents
{
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"content_analyzer": {
"tokenizer": "ik_max_word",
"filter": ["lowercase", "stop"]
}
}
}
},
"mappings": {
"properties": {
"title": {
"type": "text",
"analyzer": "content_analyzer",
"boost": 2.0
},
"content": {
"type": "text",
"analyzer": "content_analyzer"
},
"file_type": {
"type": "keyword"
},
"file_size": {
"type": "long"
},
"author": {
"type": "keyword"
},
"department": {
"type": "keyword"
},
"tags": {
"type": "keyword"
},
"created_at": {
"type": "date"
},
"updated_at": {
"type": "date"
},
"permissions": {
"type": "keyword"
}
}
}
}

文档搜索实现

# 智能搜索查询
GET /documents/_search
{
"query": {
"bool": {
"must": [
{
"multi_match": {
"query": "项目管理流程",
"fields": ["title^2", "content", "tags^1.5"],
"type": "best_fields",
"fuzziness": "AUTO"
}
}
],
"filter": [
{
"terms": {
"permissions": ["public", "user_department"]
}
},
{
"range": {
"created_at": {
"gte": "now-1y"
}
}
}
]
}
},
"highlight": {
"fields": {
"title": {},
"content": {
"fragment_size": 150,
"number_of_fragments": 3
}
}
},
"aggs": {
"departments": {
"terms": {
"field": "department",
"size": 10
}
},
"file_types": {
"terms": {
"field": "file_type",
"size": 5
}
}
},
"sort": [
"_score",
{"updated_at": {"order": "desc"}}
]
}

6.4 性能监控与优化

关键性能指标

集群健康监控

# 集群状态检查
GET /_cluster/health

# 节点统计信息
GET /_nodes/stats

# 索引统计信息
GET /_stats

# 慢查询日志
GET /_cat/indices?v&s=docs.count:desc

查询性能分析

# 查询性能分析
GET /products/_search
{
"profile": true,
"query": {
"bool": {
"must": [
{"match": {"name": "iPhone"}},
{"range": {"price": {"gte": 5000}}}
]
}
}
}

性能优化策略

索引优化

# 强制合并段
POST /products/_forcemerge?max_num_segments=1

# 刷新索引设置
PUT /products/_settings
{
"refresh_interval": "30s",
"number_of_replicas": 0
}

查询优化

  • 使用filter代替query进行精确匹配
  • 合理使用缓存机制
  • 避免深度分页
  • 使用scroll API处理大量数据

故障排除实践

常见问题诊断

  1. 集群变红:检查分片分配和磁盘空间
  2. 查询变慢:分析慢查询日志和Profile结果
  3. 内存不足:调整JVM堆内存和字段缓存
  4. 数据丢失:检查副本配置和备份策略

应急处理流程

# 紧急情况下的操作命令
# 1. 检查集群状态
curl -X GET "localhost:9200/_cluster/health?pretty"

# 2. 查看未分配分片
curl -X GET "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason"

# 3. 手动分配分片
curl -X POST "localhost:9200/_cluster/reroute" -H 'Content-Type: application/json' -d'
{
"commands": [
{
"allocate_empty_primary": {
"index": "products",
"shard": 0,
"node": "node-1",
"accept_data_loss": true
}
}
]
}'

本章小结

通过本章的三个实战案例,我们学习了:

  1. 电商搜索系统:从数据建模到搜索实现的完整流程
  2. 日志分析平台:ELK Stack的集成应用和运维实践
  3. 企业文档搜索:智能搜索和权限控制的实现方案
  4. 性能监控优化:监控指标和故障排除的最佳实践

这些案例覆盖了Elasticsearch在企业级应用中的主要场景,为实际项目开发提供了参考模板和解决方案。


扩展阅读

📝 更新时间: 2025-01-26
🎯 适用版本: Elasticsearch 8.x
📚 难度等级: 高级实战