logstash output to elasticsearch with document_id; what to do when I don't have a document_id?

You're close with the conditional idea but you can't place it inside a plugin block. Do this instead:

output {  if [document_id] {    elasticsearch_http {      host => "127.0.0.1"      document_id => "%{document_id}"    }   } else {    elasticsearch_http {      host => "127.0.0.1"    }   }}

(But the suggestion in one of the other answers to use the uuid filter is good too.)

One way to solve this is to make sure a document_idis always available. You can achieve this by adding a UUID filter in the filter section that would create the document_id field if it is not present.

filter {    if "" in [document_id] {        uuid {            target => "document_id"        }    }}

Edited per Magnus Bäck's suggestion. Thanks!

Reference : docinfo_fields

For any document added in elasticsearch, the _id is auto-generated if not specified during insert. We can use this same _id later to update/delete/search queries by using docinfo_fields feature.

Example :

filter {    json {        source => "message"    }        elasticsearch {        hosts => "http://localhost:9200/"        user => elastic        password => elastic        query => "..."        docinfo_fields => {          "_id" => "docid"          "_index" => "document_index"        }    }    if ("_elasticsearch_lookup_failure" not in [tags]) {        #... doc update logic ...    }}output {    elasticsearch {        hosts => "http://localhost:9200/"        user => elastic        password => elastic        index => "%{document_index}"        action => "update"        doc_as_upsert => true        document_id => "%{docid}"    }}