kibana 접속 안될때 (read-only 전환된 상태)

주요 에러 : FORBIDDEN/12/index read-only

"reason":"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]","caused_by":{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}]}}}}"}

Unhandled rejection [export_exception] Exception when closing export bulk :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"0e23f447-0789-46b8-aeb8-c75defc09208\",\"name\":\"ip-10-100-2-49.ap-northeast-2.compute.internal\",\"index\":\".kibana\",\"host\":\"0.0.0.0\",\"transport_address\":\"0.0.0.0:5601\",\"version\":\"6.5.4\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":0,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-lucene\"},\"rollups\":{\"index_patterns\":{\"total\":0},\"saved_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved_searches\":{\"total\":0}}},\"index\":\".kibana\",\"dashboard\":{\"total\":1},\"visualization\":{\"total\":11},\"search\":{\"total\":1},\"index_pattern\":{\"total\":1},\"graph_workspace\":{\"total\":0},\"timelion_sheet\":{\"total\":0},\"xpack\":{\"spaces\":{\"available\":true,\"enabled\":true,\"count\":1},\"reporting\":{\"available\":true,\"enabled\":true,\"browser_type\":\"chromium\",\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{},\"lastDay\":{\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{}},\"last7Days\":{\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{}}}},\"canvas\":{\"workpads\":{\"total\":1},\"pages\":{\"total\":2,\"per_workpad\":{\"avg\":2,\"min\":2,\"max\":2}},\"elements\":{\"total\":90,\"per_page\":{\"avg\":45,\"min\":44,\"max\":46}},\"functions\":{\"total\":144,\"in_use\":[\"esdocs\",\"mapColumn\",\"getCell\",\"if\",\"compare\",\"math\",\"revealImage\",\"asset\",\"image\",\"essql\",\"eq\",\"ply\",\"as\",\"staticColumn\",\"pointseries\",\"pie\",\"palette\",\"render\",\"rowCount\",\"filterrows\",\"any\",\"markdown\",\"font\",\"filters\",\"demodata\",\"containerStyle\",\"timelion\",\"plot\",\"seriesStyle\",\"formatNumber\",\"metric\"],\"per_element\":{\"avg\":3.130434782608696,\"min\":1,\"max\":7}}},\"infraops\":{\"last_24_hours\":{\"hits\":{\"infraops_hosts\":0,\"infraops_docker\":0,\"infraops_kubernetes\":0,\"logs\":0}}}},\"cloud\":{\"name\":\"aws\",\"id\":\"i-0c625c857208057b4\",\"vm_type\":\"r5.xlarge\",\"region\":\"ap-northeast-2\",\"zone\":\"ap-northeast-2a\",\"metadata\":{\"architecture\":\"x86_64\",\"marketplaceProductCodes\":null,\"imageId\":\"ami-047f7b46bd6dd5d84\",\"kernelId\":null,\"pendingTime\":\"2020-05-15T02:29:08Z\",\"ramdiskId\":null,\"version\":\"2017-09-30\"}},\"concurrent_connections\":0,\"os\":{\"load\":{\"1m\":0.13232421875,\"5m\":0.0830078125,\"15m\":0.029296875},\"memory\":{\"total_in_bytes\":33244717056,\"free_in_bytes\":618860544,\"used_in_bytes\":32625856512},\"uptime_in_millis\":30239455000,\"platform\":\"linux\",\"platformRelease\":\"linux-4.14.109-99.92.amzn2.x86_64\",\"distro\":\"Raspbian\",\"distroRelease\":\"Raspbian-2\"},\"process\":{\"event_loop_delay\":1.1623306274414062,\"memory\":{\"heap\":{\"total_in_bytes\":210165760,\"used_in_bytes\":191881488,\"size_limit\":1501560832},\"resident_set_size_in_bytes\":328794112},\"uptime_in_millis\":98508},\"requests\":{\"disconnects\":0,\"total\":68},\"response_times\":{\"average\":3.8823529411764706,\"max\":12},\"timestamp\":\"2021-04-30T02:20:12.377Z\"}\n","statusCode":500,"response":"{\"took\":3,\"ignored\":false,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"Exception when closing export bulk\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"bulk [default_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}}]}}}}"}

발생 원인 : 디스크 사용률이 80% 이상 되었을 때 발생

/dev/nvme0n1p1 40G 36G 4.5G 89% /

해결 방안 :

kibana가 설치 되어있는 elastic 서버에 접속해서 아래 명령어 실행하고 ( stage : elastic1 )

curl -X PUT "localhost:9200/settings?pretty" -H 'Content-Type: application/json' -d'
{
     "index": {
        "blocks": {
         "read_only_allow_delete": "false"
         }
     }
 }
'

kibana 재시작

sudo su elastic
/kibana/kibana-6.5.4-linux-x86_64/stop.sh
/kibana/kibana-6.5.4-linux-x86_64/start.sh

kibana web 접속 후 Monitoring 탭에 접근이 안된다면, setting 체크

GET _all/_settings

settings 에서 "read_only_allow_delete" : "ture" 가 보이면, Dev Tools에서 다음 명령어 수행

PUT _all/_settings
{
    "index": {
        "blocks": {
        "read_only_allow_delete": "false"
        }
    }
}

이후 작업 :

사용하지 않는 indices 삭제해서 서버의 용량 확보