An unknown error occurred sending a bulk request to Elasticsearch

[duylong]

Hi,

Do you know why I get this error after a Logstash restart or HUP ? The problem appears since the last version of Logstash (7.4.0).

[2019-10-18T10:10:30,978][ERROR][logstash.outputs.elasticsearch][test] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:27:in `jruby_dump'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'", "org/jruby/RubyArray.java:1800:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:117:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:296:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:201:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:169:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start_workers'"]}

I must restart another time in order to fix Logstash.

[ebuildy]

Hello,

Can you check on elasticsearch side, the write thread pool, if you dont see any "rejected" call?

If yes, this means you send too many data to elasticsearch, I suggest to use logstash internal queue (or better, something like RabbitMQ in front of logstash), in order to throttle output.

[Rickvanderwaal]

We do have the same problem. It happens after rebooting the server.
We are using Logstash in Docker version 7.12.1.

We must restart Logstash another time to fix the issue.

[2021-07-19T09:19:30,344][ERROR][logstash.outputs.elasticsearch] 
An unknown error occurred sending a bulk request to Elasticsearch. 
We will retry indefinitely 

{
	:error_message=>"", 
	:error_class=>"LogStash::Json::GeneratorError", 
	:backtrace=>[
		"/usr/share/logstash/logstash-core/lib/logstash/json.rb:43:in `jruby_dump'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:120:in `block in bulk'", 
		"org/jruby/RubyArray.java:2577:in `map'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:120:in `block in bulk'", 
		"org/jruby/RubyArray.java:1809:in `each'", 
		"org/jruby/RubyEnumerable.java:1258:in `each_with_index'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:118:in `bulk'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:271:in `safe_bulk'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:192:in `submit'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:160:in `retrying_submit'", 
		"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch.rb:303:in `multi_receive'", 
		"org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", 
		"org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", 
		"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295:in `block in start_workers'"
	]
}

There is no related logging in Elasticsearch.

[kares]

with LogStash::Json::GeneratorError there must be some JSON formatting incompatibility in the library used as it tries to format the event to JSON. weird, what would help is having debug logging turned on and seeing the event (in the logs) before it's serialized.

[manzoor77]

Hello, Facing this issue while indexing postgres db table into elasticsearch

[2023-03-26T17:26:30,367][ERROR][logstash.outputs.elasticsearch][main][05e46f90336e41dd4584f84d1289fe291bed2b9c018e1ec2d92ae4bf4603cda5] An unknown error occurred sending a bulk request to Elasticsearch (will retry indefinitely) {:message=>""\xFF" from ASCII-8BIT to UTF-8", :exception=>LogStash::Json::GeneratorError, :backtrace=>["/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/logstash-core/lib/logstash/json.rb:43:in jruby_dump'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:127:in block in bulk'", "org/jruby/RubyArray.java:2589:in map'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:127:in block in bulk'", "org/jruby/RubyArray.java:1821:in each'", "org/jruby/RubyEnumerable.java:1258:in each_with_index'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:125:in bulk'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:291:in safe_bulk'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:223:in submit'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in retrying_submit'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:342:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/Users/manzoorfaisal/Desktop/Laptop-Migration-2/logstash-7.17.4/logstash-core/lib/logstash/java_pipeline.rb:300:in block in start_workers'"]}

[TheVastyDeep]

with LogStash::Json::GeneratorError there must be some JSON formatting incompatibility in the library used as it tries to format the event to JSON

One example of this is ISO8859 characters that cannot be mapped to UTF-8. If you add

ruby { code => 'event.set("theBomb", [0xD1].pack("C*"))' }

to a pipeline that routes to ES it will blow up every time, and retry indefinitely. It might be good to handle GeneratorError differently and not retry it (thus losing the event). Ideally route the event to the DLQ, but I don't know if that can be done at that point.