Can't (yet) decode flowset id 274 from source id 256 - Cisco ASR 1001-X

[imuab]

Hello,

I have some issues with logstash Netflow codec and Cisco ASR 1000.
I am using Netflow Version 9 and have following messages in my logstash logs:

[2019-10-08T15:36:22,517][ERROR][logstash.inputs.udp ] Exception in inputworker {"exception"=>java.lang.ClassCastException: class org.jruby.gen.RubyObject4 cannot be cast to class org.jruby.RubyFixnum (org.jruby.gen.RubyObject4 is in unnamed module of loader org.jruby.util.OneShotClassLoader @6b6def36; org.jruby.RubyFixnum is in unnamed module of loader 'app'), "backtrace"=>["org.jruby.runtime.invokedynamic.MathLinker.fixnum_op_equal(MathLinker.java:237)", "java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)", "org.jruby.runtime.invokedynamic.MathLinker.fixnumOperator(MathLinker.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:171)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)", "org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1197)", "org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1193)", "org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneBlock.call(JavaMethod.java:577)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:167)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode_netflow9$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:166)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:97)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:93)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$inputworker$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$inputworker$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}

[2019-10-08T15:36:46,329][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 275 from source id 256, because no template to decode it with has been received. This message will usually go away after 1 minute.

• Version: Logstash 7.4.0 / Netflow Codec 4.2.1
• Operating System: Docker 1.13.1 on Centos 7
• Config File (if you have sensitive info, please remove it):

input {
udp {
port => 2055
type => "netflow"
codec => netflow {
include_flowset_id => true
enable_metric => true
versions => [5, 9]
}
}
}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "netflow-%{+YYYY.MM.dd}"
user => elastic
password => changeme
}
stdout { codec => rubydebug }
}

• Sample Data:
  

• Steps to Reproduce:

[novaksam]

Have the same issue since I upgraded to 7.4.0; I just switched back to 7.3.2 and it's working fine with the same Netflow-Codec version

[imuab]

@novaksam Did you just downgrade Logstash to 7.3.2 or all instances like elastic, kibana etc?

[imuab]

I got it working with Logstash 7.3.2, thank you @novaksam!

But i got some problems with the built-in netflow visualizations and dashboards.
The visualizations are matched on an different index-pattern and i cant match them for my netflow-* index.

The dashboards are looking for data with the filter "input.type: netflow", but they cant find any data.
Are they any syntax problems in my netflow.conf? Im saying type => netflow , is that wrong?

When im looking into logs, the type is correct. It says netflow, as you can see.

I think I am missing some fields, right?

[smaxx1337]

Hey, we have the same problem here with decoding the template. We are also using a Cisco ASR 1001-X and Logstash 7.3.2.

[2019-10-10T09:03:47,120][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-10-10T09:03:47,120][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-10-10T09:03:47,125][WARN ][logstash.codecs.netflow  ] Unsupported field in template 258 {:type=>44999, :length=>32}
[2019-10-10T09:03:47,125][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.

[imuab]

@smaxx1337 the logstash netflow module is deprecated in 7.4.0. It would recommand using the filebeat netflow module for the future. I just changed my setup as well and it's working absolutly fine with filebeat 7.4.0.

https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-netflow.html

[robcowart]

@imuab Only the Netflow module (which was basically ElastiFlow 1.0.0) is deprecated, not the Logstash Netflow Codec.

[novaksam]

Looks like this will/should be fixed in LS 7.6
elastic/logstash#11196