Is the @timestamp field needed when using Logstash to store in Elasticsearch?
When a message is read by Logstash, Logstash treat the message as a Event. An event will have a timestamp and message log. Thus, the @timestamp field is requisite.
Therefore, if you want to delete the @timestamp field, it will causes an error.Logstash can't output the event to the elasticsearch.
Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `tv_sec' for nil:NilClass at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:239) at org.jruby.RubyString.gsub(org/jruby/RubyString.java:3062) at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:225) at RUBY.receive(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/outputs/elasticsearch.rb:153)
So far, not all @-prefix fields causes error, only remove @timestamp will cause this error.
Any @
-prefixed field is used internally by Logstash. Removing them tends to cause errors.
For example, I tried the following config file with Logstash 1.3.3:
input { generator { type => "timestrip" message => "This is a test message." count => 1 }}filter { mutate { remove_field => ["@timestamp"] }}output { elasticsearch_http { host => "127.0.0.1" flush_size => 1 }}
The ES output reports "Failed to flush outgoing items" with the following exception:
NoMethodError: undefined method `tv_sec' for nil:NilClass