deadlock error occurred in elasticsearch-transport-7.0.0 #656
Closed
Description
opened on May 10, 2019
Hi,
We send logs our elasticsearch with fluent-plugin-elasticsearch but deadlock error occurred when we restarted our elasticsearch.
Any solution?
2019-05-10 15:38:27 +0900 [warn]: #0 [...] failed to flush the buffer. retry_time=0 next_retry_seconds=2019-05-10 15:38:12 +0900 chunk="58882d06509e33d5aeb3ed48dff56386" error_class=ThreadError error="deadlock; recursive locking"
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/connections/connection.rb:133:in `synchronize'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/connections/connection.rb:133:in `resurrectable?'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/connections/connection.rb:125:in `resurrect!'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:112:in `block in resurrect_dead_connections!'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:112:in `each'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:112:in `resurrect_dead_connections!'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:87:in `get_connection'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:271:in `perform_request'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/client.rb:160:in `perform_request'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/elasticsearch-api-7.0.0/lib/elasticsearch/api/actions/bulk.rb:101:in `bulk'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-2.11.10/lib/fluent/plugin/out_elasticsearch.rb:621:in `send_bulk'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-2.11.10/lib/fluent/plugin/out_elasticsearch.rb:523:in `block in write'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-2.11.10/lib/fluent/plugin/out_elasticsearch.rb:522:in `each'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-2.11.10/lib/fluent/plugin/out_elasticsearch.rb:522:in `write'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:1110:in `try_flush'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:1389:in `flush_thread_run'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:444:in `block (2 levels) in start'
2019-05-10 15:38:27 +0900 [warn]: #0 /path/to/vendor/bundle/ruby/2.6.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
-
OS
CentOS Linux release 7.6.1810 (Core) -
Elasticsearch
7.0.1 -
Ruby
ruby 2.6.3
Gemfile.lock
GEM
remote: https://rubygems.org/
specs:
cool.io (1.5.4)
dig_rb (1.0.1)
elasticsearch (7.0.0)
elasticsearch-api (= 7.0.0)
elasticsearch-transport (= 7.0.0)
elasticsearch-api (7.0.0)
multi_json
elasticsearch-transport (7.0.0)
faraday
multi_json
ethon (0.12.0)
ffi (>= 1.3.0)
excon (0.64.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
ffi (1.10.0)
fluent-plugin-elasticsearch (2.11.10)
elasticsearch
excon
fluentd (>= 0.14.20)
fluent-plugin-ping-message (1.0.0)
fluentd (>= 0.14.0)
fluent-plugin-prometheus (1.0.1)
fluentd (>= 0.14.8, < 2)
prometheus-client
fluent-plugin-record-reformer (0.9.1)
fluentd
fluent-plugin-suppress (1.0.0)
fluentd (>= 0.14.8, < 2)
fluentd (1.2.5)
cool.io (>= 1.4.5, < 2.0.0)
dig_rb (~> 1.0.0)
http_parser.rb (>= 0.5.1, < 0.7.0)
msgpack (>= 0.7.0, < 2.0.0)
serverengine (>= 2.0.4, < 3.0.0)
sigdump (~> 0.2.2)
strptime (>= 0.2.2, < 1.0.0)
tzinfo (~> 1.0)
tzinfo-data (~> 1.0)
yajl-ruby (~> 1.0)
http_parser.rb (0.6.0)
msgpack (1.2.10)
multi_json (1.13.1)
multipart-post (2.1.0)
oj (3.6.7)
prometheus-client (0.9.0)
quantile (~> 0.2.1)
quantile (0.2.1)
serverengine (2.1.1)
sigdump (~> 0.2.2)
sigdump (0.2.4)
strptime (0.2.3)
thread_safe (0.3.6)
typhoeus (1.3.0)
ethon (>= 0.9.0)
tzinfo (1.2.5)
thread_safe (~> 0.1)
tzinfo-data (1.2019.1)
tzinfo (>= 1.0.0)
yajl-ruby (1.4.1)
PLATFORMS
ruby
DEPENDENCIES
fluent-plugin-elasticsearch (= 2.11.10)
fluent-plugin-ping-message (= 1.0.0)
fluent-plugin-prometheus (= 1.0.1)
fluent-plugin-record-reformer (= 0.9.1)
fluent-plugin-suppress (= 1.0.0)
fluentd (= 1.2.5)
oj (= 3.6.7)
typhoeus (= 1.3.0)
BUNDLED WITH
1.17.2
Metadata
Assignees
Labels
No labels
Activity