DevOps Runbook: splunk

<dashboard>
<label>My label</label>
<description>A description</description>

<row>
    <panel>
      <title>Datacentre 1</title>
      <table>
        <search>

          <query>
            (host=jupiter ORhost=neptune)
            sourcetype=apache* source=*check_endpoints_info.log HTTP_STATUS_CODE: |
            rex field=_raw "TIMESTAMP:(?<TIMESTAMP>.+)\s+APP" |
            rex field=_raw "APP:(?<SERVICE>.+)\s+PORT" |
            rex field=_raw "CONTEXT ROOT:(?<CONTEXT_ROOT>.+)\s+GET" |
            rex field=_raw "\s+GET:\s+(?<URI>.+)\s+HTTP.+Host" |
            rex field=_raw "\s+HTTP_STATUS_CODE:\s+(?<HTTP_STATUS>\d+).+" |
            rex field=_raw "PORT:(?<PORT>.+)CONTEXT ROOT" |
            stats max(TIMESTAMP) as TIMESTAMP by host,SERVICE,URI,HTTP_STATUS |
            sort host
          </query>

          <earliest>-210s@s</earliest>
          <latest>now</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>

        </search>

        <option name="count">18</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">row</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>

        <format type="color" field="HTTP_STATUS">
          <colorPalette type="map">{"200":#65A637,"404":#D93F3C,"500":#D93F3C,"503":#D93F3C}</colorPalette>
        </format>

      </table>
    </panel>

    <panel>
      <title>Datacentre 2</title>
      <table>
        <search>

          <query>
            (host=mars OR host=venus)
            sourcetype=apache* source=*check_endpoints_info.log HTTP_STATUS_CODE: |
            rex field=_raw "TIMESTAMP:(?<TIMESTAMP>.+)\s+APP" |
            rex field=_raw "APP:(?<SERVICE>.+)\s+PORT" |
            rex field=_raw "CONTEXT ROOT:(?<CONTEXT_ROOT>.+)\s+GET" |
            rex field=_raw "\s+GET:\s+(?<URI>.+)\s+HTTP.+Host" |
            rex field=_raw "\s+HTTP_STATUS_CODE:\s+(?<HTTP_STATUS>\d+).+" |
            rex field=_raw "PORT:(?<PORT>.+)CONTEXT ROOT" |
            stats max(TIMESTAMP) as TIMESTAMP by host,SERVICE,URI,HTTP_STATUS |
            sort host
          </query>

          <earliest>-4m@m</earliest>
          <latest>now</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>

        </search>

        <option name="count">18</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">row</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="color" field="HTTP_STATUS">
          <colorPalette type="map">{"200":#65A637,"404":#D93F3C,"500":#D93F3C,"503":#D93F3C}</colorPalette>
        </format>

      </table>
    </panel>
</row>
</dashboard>

The syntax is:

rex field=splunk data field "(?<variable to assign to>regex)”

For example:
If your splunk _raw field contained the line “The sky is blue” and you wanted to 
get the word blue and assign it to a variable of COLOUR, you would do the following:

sourcetype="your_source_type" source="/etc/foo/bar" | rex field=_raw "The sky is\s+(?<COLOUR>\w+)\.*"

i.e  “The sky is” followed by one of more spaces, followed by one or more word characters (which are assigned to the variable COLOUR) followed by 0 or more of any characters. i.e, standard regex but instead of putting the assignment braces around only the (\w+) you also insert ?<variable to assign to> to the left of it, so you end up with  (?<COLOUR>\w+)

Now you have a variable called COLOUR you can pipe it to a table

sourcetype="your_source_type" source="/etc/foo/bar" | rex field=_raw "The sky is\s+(?<COLOUR>\w+)\.*" | table COLOUR

Here’s a real world example, to pull out the http method, response code and uri from apache’s access logs and render them in a table

sourcetype="psd2:ihs" source="/usr/websphere*/ihs*" | rex field=_raw "(?<METHOD>POST|GET|PUT)\s+(?<URI>.*\s+)\.*HTTP/1.1\"##(?<CODE>\d+)" | table host, CODE, METHOD, URI

DevOps Runbook

Saturday, 13 January 2018

An example splunk dashboard XML

splunk regex