Introduction to Logstash

Table of Contents

What is Logstash?

A generic concept for receiving data, transforming it, and outputting it.

Downloading/Installing Logstash

Downloading is as easy as getting it from http://logstash.net, but I'll be using version 1.4 (which is currently beta as of this writing)

To install, untar the package somewhere, or use the .deb/.rpm repositories for your respective operating system.

Running Logstash

There are 2 main modes of running Logstash. Note that both can be run at once.

Agent

Running as an agent collects information, forwarding it to the backend (in our case, Elasticsearch)

Web

Runs the web UI (known as Kibana) bundled in Logstash

Logstash architecture

Logstash is a collection of:

  • Inputs
  • Codecs
  • Filters
  • Outputs

Simplest configuration

Starting with the simplest input, standard in:

input {
  stdin {}
}

And the simplest output, standard out (no filters for now):

output {
  stdout {}
}

To run this, you can do:

bin/logstash agent -f logstash-simple.conf

Changing the way data is represented

Let's change the codec (data representation) to print more information:

input {
  stdin {}
}
output {
  stdout {
    codec => rubydebug
  }
}

Reading input from files on disk

This time, instead of reading in from stdin, read from a file:

input {
  file {
    type => "apache"
    path => "/Users/hinmanm/intro-to-logstash/example.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Outputing to an embedded Elasticsearch

Logstash can output to many more places than just stdout, it comes with elasticsearch as an output option that can run embedded:

input {
  file {
    type => "apache"
    path => "/Users/hinmanm/intro-to-logstash/example.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    embedded => true
  }
}

Add a few logs to the file:

echo "this is a log message about foo" >> example.log
echo "this is a log message about bar" >> example.log
echo "this is a log message about baz" >> example.log

Logstash creates an index, notice that it created it for the day this was run. Logstash will create daily indices by default:

curl 'localhost:9200/_cat/health?v'
echo "------"
curl 'localhost:9200/_cat/shards?v'
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign
1395046372 02:52:52  elasticsearch yellow          2         1      5   5    0    0        5
------
index               shard prirep state      docs store ip             node
logstash-2014.03.17 2     p      STARTED       0   99b 172.22.255.231 Multiple Man
logstash-2014.03.17 2     r      UNASSIGNED
logstash-2014.03.17 0     p      STARTED       0   99b 172.22.255.231 Multiple Man
logstash-2014.03.17 0     r      UNASSIGNED
logstash-2014.03.17 3     p      STARTED       2 4.2kb 172.22.255.231 Multiple Man
logstash-2014.03.17 3     r      UNASSIGNED
logstash-2014.03.17 1     p      STARTED       1 3.9kb 172.22.255.231 Multiple Man
logstash-2014.03.17 1     r      UNASSIGNED
logstash-2014.03.17 4     p      STARTED       0   99b 172.22.255.231 Multiple Man
logstash-2014.03.17 4     r      UNASSIGNED

And you can search for log messages (here's an example query)

{
  "query": {
    "simple_query_string": {
      "query": "foo | bar",
      "fields": ["message"]
    }
  },
  "size": 3
}

And get back the results:

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 1248

{
  "took" : 76,
  "timed_out" : false,
  "_shards" : {
    "total" : 20,
    "successful" : 20,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.35355338,
    "hits" : [ {
      "_index" : "logstash-2014.03.17",
      "_type" : "apache",
      "_id" : "q8Eq-Ck2RjWwB70-rxz7bw",
      "_score" : 0.35355338, "_source" : {"message":"this is a log message about foo","@version":"1","@timestamp":"2014-03-17T08:52:39.789Z","type":"apache","host":"Xanadu.local","path":"/Users/hinmanm/intro-to-logstash/example.log"}
    }, {
      "_index" : "logstash-2014.03.17",
      "_type" : "apache",
      "_id" : "e0KXf2eCQjmm302UB6n60g",
      "_score" : 0.35355338, "_source" : {"message":"this is a log message about bar","@version":"1","@timestamp":"2014-03-17T08:52:39.791Z","type":"apache","host":"Xanadu.local","path":"/Users/hinmanm/intro-to-logstash/example.log"}
    }, {
      "_index" : "logstash-2014.03.13",
      "_type" : "apache",
      "_id" : "DXwFHMvTTsauxjr9lJ5Xcg",
      "_score" : 0.25427115, "_source" : {"message":"this is a log message about bar","@version":"1","@timestamp":"2014-03-13T08:45:22.446Z","type":"apache","host":"Xanadu.local","path":"/Users/hinmanm/intro-to-logstash/example.log"}
    } ]
  }
}

Outputing to a separate Elasticsearch

Embedded is great for development, but outputting to a different Elasticsearch server is better for production:

input {
  file {
    type => "apache"
    path => "/Users/hinmanm/intro-to-logstash/example.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    host => "localhost"
    port => 9300
    node_name => "logstash-agent-007"
    workers => 2
  }
}

Adding a filter into the mix

Filters allow you to modify output

The most useful is grok, but let's start with mutate. So the standard input/output configuration first:

input {
  stdin {}
  file {
    type => "apache"
    path => "/Users/hinmanm/intro-to-logstash/example.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Adding the mutate filter to add a field as well as lowercase the "message" field

filter {
  mutate {
    add_field => [ "myhost", "Hello from %{host}!"]
    lowercase => [ "message"]
  }
}

The "grok" filter

Logstash's arguably most useful filter. ~120 different patterns that can be comibned.

Again, standard boilerplate:

input {
  stdin {}
  file {
    type => "apache"
    path => "/Users/hinmanm/intro-to-logstash/example.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

And then a grok filter meant to match the text "name: John" in the inputs:

filter {
  grok {
    match => [ "message", "name: %{WORD:custom_name}" ]
  }
  mutate {
    lowercase => [ "custom_name" ]
  }
}

Combining everything together

Read from a file (this time an Elasticsearch log file), use the es-log type when putting the log message into Elasticsearch. Output will be written to a separate Elasticsearch cluster at localhost on port 9300:

input {
  file {
    type => "es-log"
    path => "/Users/hinmanm/intro-to-logstash/es/logs/elasticsearch.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    host => "localhost"
    port => 9300
  }
}

This example filter will match Elasticsearch's log format, extract the useful pieces of the log (time, level, package, node_name, and log message).

The mutate filter will then:

  • lowercase the log level (INFO => info)
  • strip the whitespace for the package ("indices.recovery " => "indices.recovery")

Additionally, the multiline filter will match lines that look like a Java Exception, and collapse them into a single message from the previous line.

filter {
  grok {
    match => [ "message",
               "^\[%{TIMESTAMP_ISO8601:time}\]\[%{LOGLEVEL:level}.*\]\[%{DATA:package}\] \[%{DATA:node_name}\] %{DATA:logmsg}$"]
  }

  mutate {
    lowercase => [ "level" ]
    strip => [ "package" ]
  }

  multiline {
    pattern => "(org\.elasticsearch\.Exception.+|(at .+))"
    what => "previous"
  }
}

The UI for Logstash

Logstash bundles Kibana, which can be used for visualizing data, and is as easy as running:

bin/logstash web

or both at once with:

bin/logstash agent -f logstash.conf -- web

Author: Lee Hinman

Created: 2014-03-19 Wed 01:07

Emacs 24.3.50.1 (Org mode 8.2.5h)

Validate