Deduping entries with logstash and elasticsearch

There's one annoying little problem with logstash and file / block based injestion. The whole file gets reprocessed in certain scenarios, causing lines that's already been processed before to be reprocessed. These scenarios include:

1. S3 files been reuploaded with additional data appended.
   Logstash process all files after the datetime marked by the last run of processing.
2. Local file system files being copy pasted over with updates from other locations
   Files are handled with descriptors, so they can be renamed or moved without affecting logstash tailing the file. However, when they are overwritten, they are considered to be a completely different file and all the data in the file will be reprocessed.
3. Losing / deleting the since db file used to track progress.
4. And I'm sure there can be more.

The solution is actually surprisingly simple. Calculate a hash of the event message and use that as the document id for elasticsearch. Here's a sample config:

input {

#something. anything.

}

filter {

mutate {

add_field => ["logstash_checksum", "%{message}"]

}

anonymize {

fields => ["logstash_checksum"]

algorithm => "MD5"

key => "a"

}

}

output {

elasticsearch {

host => "127.0.0.1"

document_id => '%{logstash_checksum}'

}

}

Note that this works best with events that already contains the timestamp such as web server access log from IIS, apache, etc, load balancer logs, etc. It would be a bad idea to apply this technique to stream based log entries that rely on timestamp at the time of injestion by logstash.