机房意外掉电导致Elasticsearch的部分index无数据的修复过程

1. 环境 :华为大数据集群FusionInsight V100R002C800SPC200、Elasticsearch 6.1.3、Kibana
2. 问题产生原因：因机房意外掉电导致集群部分机器两次掉电导致Elasticsearch重启，Elasticsearch重启后看似正常但某些index无数据。经排查判断为Elasticsearch的部分index变成只读导致的。
3. 解决步骤：
   1. 检查集群所有index是否有只读的情况：Kibana执行GET /_settings得到的结果中有index显示结果为read_only_allow_delete:"true"，表明此index为只读

      	  "index_name": {"settings": {"index": {"mapping": {"total_fields": {"limit": "2000"}},"refresh_interval": "180s","translog": {"flush_threshold_size": "1GB","sync_interval": "120s","durability": "async"},"blocks": {"read_only_allow_delete": "true","write": "false"},"provided_name": "index_name","max_result_window": "100000","creation_date": "1591003803952","unassigned": {"node_left": {"delayed_timeout": "30m"}},"number_of_replicas": "1","uuid": "4h3Hzw6fQ3ajh8WW1iDTOQ","version": {"created": "6010399"},"routing": {"allocation": {"total_shards_per_node": "4"}},"number_of_shards": "48","merge": {"scheduler": {"max_thread_count": "1"},"policy": {"segments_per_tier": "20","max_merge_at_once": "20"}},"max_slices_per_scroll": "5000"}}},

   2. 修改只读：Kibana执行以下命令，结果返回true

      PUT index_name/_settings{"index.blocks.read_only_allow_delete":null}

   3. 验证：Kibana执行命令:GET jz_resource_3009/_settings查看结果里面不含以下内容则正常

      "blocks": {"read_only_allow_delete": "true","write": "false"},

      结果为以下

      {"index_name": {"settings": {"index": {"mapping": {"total_fields": {"limit": "2000"}},"refresh_interval": "180s","translog": {"flush_threshold_size": "1GB","sync_interval": "120s","durability": "async"},"blocks": {"write": "false"},"provided_name": "index_name","max_result_window": "100000","creation_date": "1591003803952","unassigned": {"node_left": {"delayed_timeout": "30m"}},"number_of_replicas": "1","uuid": "4h3Hzw6fQ3ajh8WW1iDTOQ","version": {"created": "6010399"},"routing": {"allocation": {"total_shards_per_node": "4"}},"number_of_shards": "48","merge": {"scheduler": {"max_thread_count": "1"},"policy": {"segments_per_tier": "20","max_merge_at_once": "20"}},"max_slices_per_scroll": "5000"}}}
      }