es 坑

发表于 | 分类于 | 阅读次数

Update By Query

VersionConflictEngineException

由于没有事务更新存在version冲突，当多个update在refresh_interval内修改同一个doc就会发生error。

Update By Query doc

解决方式：

  • conflicts

    发生version conflicts abort or proceed 默认abort，设置成proceed碰到error会继续

  • refresh

    update后马上refresh一下

Too many dynamic script compilations within, max: [75/5m]

这是关于script的描述 5min内最多编译75次，不是执行，触发circuit_breaking_exception，超过了script-compilation-circuit-breaker配置

1
script.max_compilations_rate: xxx/5m

编译script对于es来说非常重，编译好的新script会放到cache。

  • 不要使用硬编码，每次修改参数的改变都会重新编译

    1
    "source": "doc['my_field'] * 2"

  • 使用params，只编译一次

    1
    2
    3
    4
      "source": "doc['my_field'] * multiplier",
  "params": {
    "multiplier": 2
  }

解决方法

  • 调整script.max_compilations_rate 参数
  • 使用params参数来传递参数
  • stored script 预先存储script到es

Painless script function return type

1
ctx._source.xx_time = Math.max(ctx._source.xx_time, params.xx_time)

传入的参数都是long类型，返回却是double

Even though Java provides several overloaded Math.max() methods for long, float and double, Painless only provides the one for double, probably because all other types (i.e. long and float) can be upcast to double.

Painless Script Math.max change my data type

ES 强转类型

coerce

“5”，5.0 强转成 5 “false” -> false

因为es这个特性mapping 设置了数据类型约等于没用，脏数据进去了照样可以保存，不报错，放进去后取出来还是原来脏数据。go这种强类型的语言可以在测试时检查到异常，使用Python的话会是灾难。

要阻止这个傻逼特性

1
"index.mapping.coerce": false

Boolean 类型也要留意

must 和 should 同时使用 should 失效

要想must 和 should同时有用，不能同级。should用bool包起来放在must下级

错误写法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
    "query":{
        "bool":{
            "must":[
                {"term":{"a":"1"}}
            ],
            "should":[
                {
                    "match":{
                        "content":"xxxx"
                    }
                }
            ]
        }
    }
}

正确写法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
    "query":{
        "bool":{
            "must":[
                {"term":{"a":"1"}}
                {
                    "bool":{
                        "should":[
                            {
                                "match":{
                                    "content":"xxxx"
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

go uint8 数组 存放es short数组 报错

因为go 里面byte是uint8的别名

json.Marshal把[]uint8当成[]byte然后会编码成base64 string，导致es报错

failed to parse field [xxx] of type [short] in document with id ‘xxx’. Preview of field’s value: ‘AQ==’

用其他数据类型替代

https://stackoverflow.com/questions/14177862/how-to-marshal-a-byte-uint8-array-as-json-array-in-go/14178407

es mapping 不能直接update

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
ES_URL="127.0.0.1:9200/"
JSON_HEADER="Content-Type: application/json"

function es_config() {
    curl -X PUT $ES_URL$1 -H "$JSON_HEADER" -d "$2"
}
function es_del_index() {
    curl -X DELETE $ES_URL$1
}

function es_reindx() {
  	curl -X PUT $ES_URL_reindex -H "$JSON_HEADER" -d "$1"
}
es_config "xxx" '
{
  "mappings": {
    "properties": {
      "ip": { "type": "keyword"},
      "delete_time": { "type": "long" }
    }
  }
}'

# add field
es_config "xxx/_mapping" '
{
    "properties":{
        "detection_engine":{
            "type":"short"
        },
        "threat_tag":{
            "type":"keyword"
        }
    }
}
'

修改index 属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
# 改reindex

es_config "new_index" '
{
  "mappings": {
    "properties": {
      "priority": { "type": "keyword"}
    }
  }
}'

es_reindx '
{
  "source": {
    "index": "old_index"
  },
  "dest": {
    "index": "new_index"
  }
}
'

es_del_index "old_index"


es_reindx '
{
  "source": {
    "index": "old_index"
  },
  "dest": {
    "index": "new_index"
  }
}
'

# 重建index
es_config "old_index" '
{
  "mappings": {
    "properties": {
      "priority": { "type": "keyword"}
    }
  }
}'

es_reindx '
{
  "source": {
    "index": "new_index"
  },
  "dest": {
    "index": "old_index"
  }
}
'

es_del_index "new_index"

别名 alias

能非常优雅的解决两个索引无缝切换的问题 es alias doc

1
PUT /my-index-000001/_alias/alias1

  • 在一个运行中的es集群中无缝的切换一个索引到另一个索引上

  • 分组多个索引，比如按月创建的索引，我们可以通过别名构造出一个最近3个月的索引

  • 查询一个索引里面的部分数据构成一个类似数据库的视图（views）可以用filter

    1
    2
    3
    4
    5
    6
    7
    8
    9
    PUT /users/_alias/user_12
{
  "routing" : "12",
  "filter" : {
    "term" : {
      "user_id" : 12
    }
  }
}

索引别名切换

1
2
3
4
5
6
7
POST /_aliases
{
    "actions": [
        { "remove": { "index": "my_index_v1", "alias": "my_index" }},
        { "add":    { "index": "my_index_v2", "alias": "my_index" }}
    ]
}

顺序执行的，先解除my_index_v1的别名，然后给my_index_v2添加新别名，my_index_v1和my_index_v2现在想通过索引别名来实现无缝切换

创建索引超时

背景：创建超时，索引没有分配Shard , 报错 no_shard_available_action_exception

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 查看集群状态
GET _cluster/health?pretty

{
    "cluster_name": "docker-cluster",
    "status": "red", # 集群不健康
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 22,
    "active_shards": 22,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 22,  #  分配不了Shard
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 4, # number_of_pending_tasks
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 59468473,
    "active_shards_percent_as_number": 50.0
}


1
2
3
# 查看索引的状态
GET /_cat/indices?v


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 通过以下语句查看具体原因

GET _cluster/allocation/explain

{
    "note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
    "index": "fafa",  # 信息
    "shard": 0,
    "primary": true,
    "current_state": "unassigned", # 信息
    "unassigned_info": {
        "reason": "INDEX_CREATED",  # 信息
        "at": "2024-03-28T09:52:31.299Z",
        "last_allocation_status": "no"
    },
    "can_allocate": "yes",   # 可配但是分配不了
    "allocate_explanation": "Elasticsearch can allocate the shard.",  #  信息
    ......
}

1
2
3
# 精确定位unassigned shard的位置
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED