Kubernetes device mapper out of space - pod stuck in ContainerCreating state

If you have a pod in ContainerCreating state, follow these steps to held determine the issue:

• carry out a wide pod listing (kubectl get pods -o wide) to determine which node your erroneous pod is hosted on
• ssh to the node in question
• run (as root) journalctl --unit kubelet --no-pager | grep <your pod name>
• In this particular case the following message was present

Mar 15 12:35:49 jupiter.stack1.com kubelet[3146]: E0315 12:35:49.215454    3146 pod_workers.go:186] Error syncing pod ec706dba-2847-11e8-b8d8-0676d5f18210 ("kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)"), skipping: failed to "CreatePodSandbox" for "kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)\" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod \"kube-proxy-nwp6t\": Error response from daemon: devmapper: Thin Pool has 152128 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior"

• This is stating that the Docker device mapper is out of space (as set in /etc/sysconfig/docker via option --storage-opt dm.loopdatasize=40G) .
• This can usually be resolved by removing containers that have exited, dangling images and dangling volumes using the following steps:
  
  
  • Cleanup exited processes: 

docker rm $(docker ps -q -f status=exited)

• • Cleanup dangling volumes:

docker volume rm $(docker volume ls -qf dangling=true)

• • Cleanup dangling images:

docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Now run docker info to check your Device Mapper available space (meta and data)

Hitting kube-proxy directly with curl to get container logs

When you ask the API server to retrieve pod logs (i.e. kubectl logs <pod name>, it does this be connecting to kube-proxy on port 10250 (not quite sure where this is defined yet but will find out and update this post)  on the node hosting the pod. You can check this manually with curl as follows

curl -k https://<k8s node main IP address>:10250/containerLogs/<pod namespace>/<pod name>/<container name>

The no route to host message is misleading because it implies that you have a IP routing problem, which is not the case, as you'll notice if you ping the host. The actually error is caused by the kube-proxy port being blocked by firewalld on the target hosting the pod in question. Not sure why the no route to host message is given because this is obviously not a routing issue. Allowing port 10250 on the host in question resolved the issue when I experienced this problem.

re-running the curl command above will show you the logs from the container in question (no authentication needed)

NOTE: Make sure you don't have HTTP_PROXY and/or HTTPS_PROXY env vars set on the box you're running curl from 

Kubernetes Ingress de-mysified

Just finished a weekend of being totally confused about k8s ingress. I'm glad to say that I finally managed (using my OCD powers and lots of snacks/coffee) to work out what's going on. Here's what I learned. (NOTE: I'm doing this on Minikube with no add-ons) 

The confusion I experienced was made worse by several guides that skipped over the key detail that was confusing me, i.e the ingress-controller, because they either:
- Assumed I was running on GKE which handles the ingress-controller automatically
- Assumed I had enabled ingress in Minikube as an add-on which also handles the ingress-controller automatically

Now, call me old fashioned, but when I'm learning something I like to do everything myself so that I fully understand every piece.

Oddly enough, a couple of guides were plain wrong and exposed the services they were publishing via ingress as NodePort. This really confused me because the ingress controller is inside the cluster, along with the services it's routing to and therefore can contact the services "natively" on their ClusterIP. The point here (and this sounds obvious to me now but before the penny dropped it wasn't) is that the only service that needs to be exposed to the outside world when using ingress is the service that sits in front of the ingress controller itself.

The aim of this guide is to:
- Help me not forget what I learned (always good to write things down) 
- Maybe help other out people that might find ingress confusing.

So here goes...

The diagram below set's out the target system:

Here's the yaml for each component, starting from the top down:

ingress-controller-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingress-controller
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: http
    nodePort: 32000
  selector:
    app: ingress-controller

ingress-controller-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ingress-controller
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-controller
  template:
    metadata:
      labels:
        app: ingress-controller
    spec:
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80

ingress-resource.yaml

An interesting point to note from the yaml below is that I've set the host to minikube. I originally left this blank which means that any hostname or IP specified in the URL would be acceptable. however when I did this the nginx controller kept trying to redirect me (via a 308) to https. I googled around a bit and found out that this is expected behavior for the nginx controller and that specifying a host stops this from happening. So I tried putting my minikube IP address in, kubectl complained about this stating that hostnames and not IP's had to be used in the ingress host field. This being the case I just placed an entry in my local /etc/hosts

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cluster-ingress-rules
spec:
  rules:
  - host: minikube
    http:
      paths:
      - path: /hello-world
        backend:
          serviceName: hello-world
          servicePort: 8080

hello-world-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: hello-world
  labels:
    run: hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: hello-world

hello-world-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: hello-world
  labels:
    run: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      run: hello-world
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: hello-world
    spec:
      containers:
      - image: gcr.io/google-samples/node-hello:1.0
        name: hello-world
        ports:
        - containerPort: 8080

default-backend-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: default-backend
  labels:
    app: default-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-backend

default-backend-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-backend
  labels:
    app: default-backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-backend
  template:
    metadata:
      labels:
        app: default-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-backend
        image: gcr.io/google_containers/defaultbackend:1.4
        ports:
        - containerPort: 8080

Applying this lot to my cluster results in the following:

----- Ingress ------------------------------------------------------------------
NAME                    HOSTS      ADDRESS   PORTS     AGE
cluster-ingress-rules   minikube             80        20m

----- Services -----------------------------------------------------------------
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
default-backend      ClusterIP   10.100.69.250   <none>        80/TCP         20m
hello-world          ClusterIP   10.110.199.3    <none>        8080/TCP       20m
ingress-controller   NodePort    10.111.99.142   <none>        80:32000/TCP   20m
kubernetes           ClusterIP   10.96.0.1       <none>        443/TCP        16d

----- Deployments --------------------------------------------------------------
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default-backend      1         1         1            1           20m
hello-world          1         1         1            1           20m
ingress-controller   1         1         1            1           20m

----- Pods ---------------------------------------------------------------------
NAME                                  READY     STATUS    RESTARTS   AGE
default-backend-79f984dc75-2bbbf      1/1       Running   0          20m
hello-world-554998f545-wjj49          1/1       Running   0          20m
ingress-controller-5cddb96544-2k9w7   1/1       Running   0          20m

As can be seen from the above, the only service exposed to the outside world is the ingress-controller service

Running minikube service list produces the following result:

|-------------|----------------------|-----------------------------|
|  NAMESPACE  |         NAME         |             URL             |
|-------------|----------------------|-----------------------------|
| default     | default-backend      | No node port                |
| default     | hello-world          | No node port                |
| default     | ingress-controller   | http://192.168.99.100:32000 |
| default     | kubernetes           | No node port                |
| kube-system | kube-dns             | No node port                |
| kube-system | kubernetes-dashboard | http://192.168.99.100:30000 |
| kube-system | tiller-deploy        | No node port                |
|-------------|----------------------|-----------------------------|

and putting http://minikube:32000/hello-world into a browser results in the message Hello Kubernetes!

And there you have it, hopefully I haven't missed anything.

Kubernetes - serviceaccount authentication tokens

When pods communicate with the API server, they use a serviceaccount to authenticate. 

The token for this is defined in a secret, which is mounted as a volume into the pod in question.

The token is signed by the key specified by the API servers --service-account-key-file property

You can test this token my making a request to the API server with curl as follows:

curl -H  “Authorization: Bearer <token>"  https://<master ip addr>:6443/api/v1/namespaces/<namespace>/pods/<pod name>

You can derive the token from the relevant secret as follows:

kubectl describe  secrets <secret name> -n kube-system

The result will look similar to below:

Name:         default-token-jvrkf
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name=default
              kubernetes.io/service-account.uid=ba8add2d-12ab-11e8-9719-080027dd95a3

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1066 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOTcxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoCsKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqGxbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZpIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds_ug

Here's an example curl request:

curl -k -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOTcxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoCsKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqGxbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZpIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds_ug" https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods/kube-dns-54cccfbdf8-xqgz6 

You can see how the pod uses this token by running kubectl describe pods kube-dns-54cccfbdf8-xqgz6 -n kube-system and checking the Mounts section in the output.

Mounts:
  /kube-dns-config from kube-dns-config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-jvrkf (ro)

It's worth noting that if you run kubectl get secrets <secret name> -n kube-system -o yaml as opposed to kubectl describe secrets <secret name> -n kube-system, the token will be base64 encoded so you'll need to pipe it through base64 --decode before you copy and paste it into your curl command

If you regenerate your cluster certificates for any reason, you'll need to regenerate these tokens. To do this just delete the secrets with kubectl, new secrets will be generated automatically. However (and this is really important and had me going for a while)  these secrets will have a different name, therefore your pods will still be trying to mount the old secrets. To resolve this you also need to delete the pods (e.g. kubectl delete pod <pod1 name> <pod2 name> <pod3 name>) -n kube-system . If the pods are managed by deployments (and this is why you should use deployments by the way) they will automatically be regenerated and will mount the new secrets automatically. 

Your best friend by far in all of this is the command journalctl -u kubelet -f. If you have a pod stuck in ContainerCreating and therefore are not able to look at that it's container logs, simply log on the node in question and run journalctl -u kubelet -f | grep <pod name> .. priceless tool 

NOTE: if you're using Calico networking, this new token will need to be set in /etc/cni/net.d/10-calico.conf which is in turn derived from the ConfigMap calico-config. If you edit the contents of this file directly, the changes will be overwritten by the contents of the calico-config ConfigMap at the next kubelet restart.

Kubernetes ConfigMaps

Create a configMap:

kubectl create configmap env-config --from-literal=ENV_VAR_1=FOO --from-literal=ENV_VAR_2=BAR

you can view the result with 

kubectl get  configmap env-config -o yaml

Create a pod to use this configMap

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
      - configMapRef:
          name: env-config
  restartPolicy: Never

kubectl create -f test-pod.yaml

This pod will print it's env and then exit. You can view the output as follows

kubectl logs test-pod -c test-container

You should see something like...

KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=test-pod
SHLVL=1
HOME=/root
EXAMPLE_SERVICE_PORT_8080_TCP_ADDR=10.100.105.233
EXAMPLE_SERVICE_SERVICE_HOST=10.100.105.233
EXAMPLE_SERVICE_PORT_8080_TCP_PORT=8080
ENV_VAR_1=FOO
ENV_VAR_2=BAR
EXAMPLE_SERVICE_PORT_8080_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
EXAMPLE_SERVICE_PORT=tcp://10.100.105.233:8080
EXAMPLE_SERVICE_SERVICE_PORT=8080
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
EXAMPLE_SERVICE_PORT_8080_TCP=tcp://10.100.105.233:8080
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
PWD=/
KUBERNETES_SERVICE_HOST=10.96.0.1

ssh tunneling (localport)

The syntax is:

ssh  -L  local_port:target_host:target_port  bastion_host

Forward port 2525 on your local machine to port 25 on mail.exmaple.net via bastion.stack1.com

ssh  -L  40000:ldap.stack1.com:389  bastion.stack1.com:389

As above but don't start a shell on the bastion (-N) and run ssh in the background (-f)

ssh  -L   40000:ldap.stack1.com:389  bastion.stack1.com -N -f

Put a shortcut in .ssh/config 

Host ldap-tunnel
    HostName bastion-host
    LocalForward local-port  target-host:target-port
    User user-on-bastion-host
    IdentityFile ~/.ssh/ private-key-for-user-on-bastion-host

Host ldap-tunnel
    HostName bastion.stack1.com
    LocalForward 40000 ldap.stack1.com:389
    User clarkeb
    IdentityFile ~/.ssh/clarkeb_rsa

If I now ran ssh ldap-tunnel -f -N I could connect my ldap client to port 40000 on my local machine and this would tunnel through to port 389 on ldap.stack1.com 

show progress when uploading with curl

curl has a --progress-bar option, however it appears that STDOUT overwrites it's output. A workaround for this is to redirect STDOUT to /dev/null as shown in the following example:

curl -v -u 'my_username:my_pass' --upload-file foo.tar.gz http://nexus.stack1.com/content/sites/binaries/golang/ --progress-bar  > /dev/null

programatically enable and disable individual crontab entries

Running as root and using the user clarkeb as an example
To append:

(crontab -u clarkeb -l 2>&1 | grep -v "no crontab" ;echo "*/5 * * * * perl /home/clarkeb/test1.pl") | crontab -u clarkeb –
(crontab -u clarkeb -l 2>&1 | grep -v "no crontab" ;echo "*/5 * * * * perl /home/clarkeb/test2.pl") | crontab -u clarkeb –
(crontab -u clarkeb -l 2>&1 | grep -v "no crontab" ;echo "*/5 * * * * perl /home/clarkeb/test3.pl") | crontab -u clarkeb –

To check:

crontab -u clarkeb -l

To remove test1.pl only and leave test2.pl and test3.pl n place:

crontab -u clarkeb -l  2>&1 | grep -v "test1.pl" | crontab -u clarkeb –

An example splunk dashboard XML

<dashboard>
  <label>My label</label>
  <description>A description</description>

  <row>
    <panel>
      <title>Datacentre 1</title>
      <table>
        <search>

          <query>
            (host=jupiter ORhost=neptune)
            sourcetype=apache* source=*check_endpoints_info.log HTTP_STATUS_CODE: |
            rex field=_raw "TIMESTAMP:(?&lt;TIMESTAMP&gt;.+)\s+APP" |
            rex field=_raw "APP:(?&lt;SERVICE&gt;.+)\s+PORT" |
            rex field=_raw "CONTEXT ROOT:(?&lt;CONTEXT_ROOT&gt;.+)\s+GET" |
            rex field=_raw "\s+GET:\s+(?&lt;URI&gt;.+)\s+HTTP.+Host" |
            rex field=_raw "\s+HTTP_STATUS_CODE:\s+(?&lt;HTTP_STATUS&gt;\d+).+" |
            rex field=_raw "PORT:(?&lt;PORT&gt;.+)CONTEXT ROOT" |
            stats max(TIMESTAMP) as TIMESTAMP by host,SERVICE,URI,HTTP_STATUS |
            sort host
          </query>
         
          <earliest>-210s@s</earliest>
          <latest>now</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
       
        </search>

        <option name="count">18</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">row</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
       
        <format type="color" field="HTTP_STATUS">
          <colorPalette type="map">{"200":#65A637,"404":#D93F3C,"500":#D93F3C,"503":#D93F3C}</colorPalette>
        </format>

      </table>
    </panel>

    <panel>
      <title>Datacentre 2</title>
      <table>
        <search>

          <query>
            (host=mars OR host=venus)
            sourcetype=apache* source=*check_endpoints_info.log HTTP_STATUS_CODE: |
            rex field=_raw "TIMESTAMP:(?&lt;TIMESTAMP&gt;.+)\s+APP" |
            rex field=_raw "APP:(?&lt;SERVICE&gt;.+)\s+PORT" |
            rex field=_raw "CONTEXT ROOT:(?&lt;CONTEXT_ROOT&gt;.+)\s+GET" |
            rex field=_raw "\s+GET:\s+(?&lt;URI&gt;.+)\s+HTTP.+Host" |
            rex field=_raw "\s+HTTP_STATUS_CODE:\s+(?&lt;HTTP_STATUS&gt;\d+).+" |
            rex field=_raw "PORT:(?&lt;PORT&gt;.+)CONTEXT ROOT" |
            stats max(TIMESTAMP) as TIMESTAMP by host,SERVICE,URI,HTTP_STATUS |
            sort host
          </query>

          <earliest>-4m@m</earliest>
          <latest>now</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>

        </search>

        <option name="count">18</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">row</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="color" field="HTTP_STATUS">
          <colorPalette type="map">{"200":#65A637,"404":#D93F3C,"500":#D93F3C,"503":#D93F3C}</colorPalette>
        </format>

      </table>
    </panel>
  </row>
</dashboard>

splunk regex

The syntax is:

rex field=splunk data field "(?<variable to assign to>regex)”

For example:
If your splunk _raw field contained the line “The sky is blue” and you wanted to 
get the word blue and assign it to a variable of COLOUR, you would do the following:

sourcetype="your_source_type" source="/etc/foo/bar" | rex field=_raw "The sky is\s+(?<COLOUR>\w+)\.*"

i.e  “The sky is” followed by one of more spaces, followed by one or more word characters (which are assigned to the variable COLOUR) followed by 0 or more of any characters. i.e, standard regex but instead of putting the assignment braces around only the (\w+) you also insert ?<variable to assign to> to the left of it, so you end up with  (?<COLOUR>\w+)  

Now you have a variable called COLOUR you can pipe it to a table

sourcetype="your_source_type" source="/etc/foo/bar" | rex field=_raw "The sky is\s+(?<COLOUR>\w+)\.*" | table COLOUR

Here’s a real world example, to pull out the http method, response code and uri from apache’s access logs and render them in a table

sourcetype="psd2:ihs" source="/usr/websphere*/ihs*" | rex field=_raw "(?<METHOD>POST|GET|PUT)\s+(?<URI>.*\s+)\.*HTTP/1.1\"##(?<CODE>\d+)" | table host, CODE, METHOD, URI

Configure IBM's ikeycmd tool to support CMS

Surprisingly, when using ikeycmd recently to manipulate IHS keystores (kdb), I had to specifically configure the CMS storetype as follows:

../jre/bin/ikeycmd -DADD_CMS_SERVICE_PROVIDER_ENABLED=true -cert -details -label foo -db plugin-key.kdb -stashed

Passwordless openssl

To prevent being prompted for a password in openssl, you can provide a password as follows:

openssl pkcs12 -in key.pkcs12 -clcerts -nodes -out all.pem -passin pass:my_password

bash shell shorthand conditionals

This is a neat way in bash scripts to use conditionals. The basic syntax is

[ condition ] &&/|| action1

counting files in a directory

[ $(ls -1 | wc -l) -ne 1 ] && echo “More than one file in this directory”

checking the number of args passed

[ $# -eq 1 ] || expecting "Expecting one argument"

you can also group multiple actions together as follows

[ $# -eq 1 ] || { echo "Expecting one argument"; exit 1; }

Using grep to show lines adjacent to the search string

Using a file with the following contents:

line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10

cat test_data | grep -B2 'line 5'

will display:
line 3
line 4
line 5

cat test_data | grep -A2 'line 5'

will display:
line 5
line 6
line 7

cat test_data | grep -A2 -B2 'line 5'

will display:
line 3
line 4
line 5
line 6
line 7

Access tomcat manager from remote host

Alongside configuring access in $CATALINA_HOME/conf/tomcat-users.xml, an additional step is required if you want to access the manager application from a remote host. Edit the $CATALINA_HOME/webapps/manager/META-INF/context.xml file and ensure the allow attribute has a value of ^.*$.

For example:

<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="^.*$" />

The complete file will look similar to this:

<?xml version="1.0" encoding="UTF-8"?>
<Context antiResourceLocking="false" privileged="true" >
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="^.*$" />
<Manager sessionAttributeValueClassNameFilter="java\.lang\.(?:Boolean|Integer|Long|Number|String)|org\.apache\.catalina\.filters\.CsrfPreventionFilter$LruCache(?:$1)?|java\.util\.(?:Linked)?HashMap"/>
</Context>

Set up an ELK stack (CentOS)

# -------------------------------------------------------------------------------------------------------------------------------------

Elasticsearch

# -------------------------------------------------------------------------------------------------------------------------------------

install java:

yum install -y java

Install elasticsearch: 

yum install -y elasticsearch-5.5.2.rpm

Start elasticsearch: 

service elasticsearch start

Configure elasticsearch

/etc/elasticsearch/elasticsearch.yml

Test elasticsearch:

curl http://10.1.1.100:9200

this should return something like the following:

{
  "name" : "BT_A1W0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "85TTdoncTPqC2vAnEufDWQ",
  "version" : {
    "number" : "5.5.2",
    "build_hash" : "b2f0c09",
    "build_date" : "2017-08-14T12:33:14.154Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Create entry in elasticsearch:

curl -XPUT '10.1.1.100:9200/my_index/my_type/my_id' -H 'Content-Type: application/json' -d’
{
"user":"bob”,
"post_date":"2009-11-15T14:12:12”,
"message":”hello"
}’

The PUT must be of the form /index/type/id (at least I think this is the case)

List indexes

curl "http://10.1.1.100:9200/_cat/indices"

delete all indexes:

curl -X DELETE "http://10.1.1.100:9200/_all"

# -------------------------------------------------------------------------------------------------------------------------------------

# Logstash

# -------------------------------------------------------------------------------------------------------------------------------------

Install logstash:

yum install -y logstash-5.5.2.rpm

Configure logstash: 

Assuming you have Apache HTTP server installed, create /etc/logstash/conf.d/logstash.conf with the following contents:

input {
  file {
    path => "/var/log/httpd/access*"
    start_position => “end"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["10.1.1.100:9200"]
  }
}

Start logstash

To test that things are working correctly, you can start in the foreground as follows:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

Once you’re happy you may want to create a basic init.d script,  e.g:

#!/bin/bash

function start() {
        echo "Starting logstash..."
        /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.reload.automatic > /var/log/logstash/logstash.log 2>&1 &
}

function stop() {
        echo "Stopping logstash..."
        pkill -9 -f .+logstash/runner.rb
}

case $1 in
        start)
                start
                ;;

        stop)
                stop
                ;;

        restart)
                stop
                sleep 3
                start
                ;;

        *)
                echo ""
                echo "Usage $0 start|stop|restart"
                echo ""
                ;;
esac

Once things are up and running, hit your Apache server with an HTTP request. Because you’ve set up an STDOUT output you’ll get something similar to the following in /var/log/logstash/logstash.log:

{
        "request" => "/",
        "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36\"",
        "auth" => "-",
        "ident" => "-",
        "verb" => "GET",
        "message" => "10.1.1.1 - - [31/Aug/2017:20:47:12 +0000] \"GET / HTTP/1.1\" 302 233 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36\"",
        "path" => "/var/log/httpd/access_log",
        "referrer" => "\"-\"",
        "@timestamp" => 2017-08-31T20:47:12.000Z,
        "response" => “200",
        "bytes" => "233",
        "clientip" => "10.1.1.1",
        "@version" => "1",
        "host" => "jupiter",
        "httpversion" => "1.1",
        "timestamp" => "31/Aug/2017:20:47:12 +0000"
}

Also, because you set up an output to elasticsearch you will see this entry in there too:

Check for the index as follows:

curl "http://10.1.1.100:9200/_cat/indices/logstash-*"

Take a look at the index as follows:

curl "http://10.1.1.100:9200/logstash-*?pretty"

# -------------------------------------------------------------------------------------------------------------------------------------

# Kibana

# -------------------------------------------------------------------------------------------------------------------------------------

Install Kibana

tar xvzf kibana-5.5.2-linux-x86_64.tar.gz -C /opt
cd /opt
mv kibana-5.5.2-linux-x86_64 kibana

Configure Kibaba

Edit /opt/kibana/config/kibana.yml, set server.host and elasticsearch.url

Start kibana

cd /opt/kibana/bin && ./kibana &

Test kibana

• In a browser, navigate to http://10.1.1.100:5601
• Navigate to dev tools
• In the left panel type GET _cat/indices and click the green arrow to run the query
• In the results window you should see the index created by logstash (when you made a request from Apache earlier) similar to:

yellow open logstash-2017.08.31 uzTssKBfTuecbgGzth-ViA 5 1 2 0 23.2kb 23.2kb

• Navigate to Management
• Select Index patterns 
• Create an index pattern of logstash-*
• Our Grok filter that we set up in logstash should have created approx 38 feilds from the Apache log, you should see that many of these are agregatable which means they can be used in visualisations

• Navigate to Visualisations > create visualisations > vertical bar
• Select the logstash-* index pattern you created earlier 
• you’ll see the Y-axis is already set up as count 
• Under buckets select X-axis
• for aggregation select Date Histogram with a field of @timestamp
• Add a sub bucket with a bucket size of split series 
• select a sub aggregation of terms with a filed of response.keyword
• Run the visual by pressing the blue/white arrow (top left)

Ansible check if variable is defined

- hosts: localhost
  tasks:
 
  - name: check args
    fail:
        msg: "env param is not defined, please set using --extra-vars env=dev."
    when: 
        env is not defined

bash option parser

A nice way of parsing options passed to a bash script without relying on position:

#!/bin/bash

for i in "$@"
  do
    case $i in

      --node-port=*)
      NODEPORT="${i#*=}"
      ;;
 
      --jjp-host=*)
      JJPHOST="${i#*=}"
      ;;
 
      --jjp-port=*)
      JJPPORT="${i#*=}"
      ;;
 
      --spp-host=*)
      SPPHOST="${i#*=}"
      ;;
 
      --spp-port=*)
      SPPPORT="${i#*=}"
      ;;

      --help)
      echo ""
      echo "Usage example: start_app --spp-host=jupiter --spp-port=2000  --node-port=3000 --jjp-host=neptune --jjp-port=4000"
      echo ""
      ;;

 
      *)
      ;;
  
  esac
done
 
[ "${NODEPORT}XXX" == "XXX" ] && { echo "Warning: NODEPORT env var not set"; }
[ "${JJPHOST}XXX" == "XXX" ] && { echo "Warning: JJPHOST env var not set"; }
[ "${JJPPORT}XXX" == "XXX" ] && { echo "Warning: JJPPORT env var not set"; }
[ "${SPPHOST}XXX" == "XXX" ] && { echo "Warning: SPPHOST env var not set"; }
[ "${SPPPORT}XXX" == "XXX" ] && { echo "Warning: SPPPORT env var not set"; }
 
export NODEPORT
export JJPHOST
export JJPPORT
export SPPHOST
export SPPPORT
 
cd /applications/my_app
nohup npm run start &

Filter Git events within a Jenkinsfile

Assuming you're using Git webhooks to trigger your Jenkins builds, let's say you want to trigger a different set of pipeline stages depending on whether the Git event is a push vs. a pull request. I.e for a push you may only wish to run static analysis and unit tests, whereas for a pull request you may want to add Selenium tests.

The following Jenkinsfile snippet will achieve this

// Import JSON Slurper
import groovy.json.JsonSlurperClassic

// Get the Git payload
def payload = new JsonSlurperClassic().parseText(env.payload)


// Now define a method to filter the event type
private String getEventType ( payload ){

   if( payload.pull_request  && payload.action.toLowerCase().contains("opened") ){
      return "pull_req"
   }

   else if( payload.ref && payload.head_commit){
      if( payload.ref.split('/')[1].toLowerCase().contains('head') ){
         return "push"
      }

      else if( payload.ref.split('/')[1].toLowerCase().contains('tag') ){
         return "tag"
      }
   }
}


// Now decide what action to take 
def eventType = getEventType( payload )
switch (eventType) {
  case "push":
    ............
    do something 
    ............
    break;

  case "pull_req":
    ............
    do something else 
    ............
    break;

  default:
      println "Git event ignored";
      currentBuild.displayName = "Git event ignored"
      
      // Tidy up the Jenkins GUI by deleting the ignored build
      def buildNumber = env.BUILD_ID
      node {
            sh "curl -X POST http://<username>:<apikey>@<jenkins host>:443/job/execute-pipeline/${buildNumber}/doDelete"
      }




 

Vagrant provisioning scripts

An example "create-user.sh" script

 
  #!/bin/bash

  # Add the user
  sudo useradd -m clarkeb -s '/bin/bash'
  sudo echo -e 'password\npassword\n' | sudo passwd clarkeb

  # Set up sudo 
  echo "clarkeb ALL=(ALL:ALL) NOPASSWD: ALL" > /etc/sudoers.d/10_clarkeb

  # Allow password based login
  sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
  sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/g' /etc/ssh/sshd_config
  service sshd restart

An example Vagrantfile snippet to invoke the above script

 
  config.vm.define "core01" do |core01|

    core01.vm.provision "shell", path: "create-user.sh"

    core01.vm.hostname = "core01"
    core01.vm.network "private_network", ip: "10.1.1.101"
    core01.vm.provider "virtualbox" do |vb|
    vb.memory = "1024"
      vb.customize ["modifyvm", :id, "--cpus", "1"]
    end
  end

Vagrant and AWS

An example Vagrantfile

 require 'vagrant-aws'  
  
 Vagrant.configure('2') do |config|  
   
  config.vm.box = "aws-box"  
  config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"  
   
  config.vm.define "jupiter" do |jupiter|  
    jupiter.vm.provision "shell", path: "install_ansible.sh"  
    jupiter.vm.provider 'aws' do |aws, override|  
      aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']  
      aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']  
      aws.keypair_name = 'vagrant'  
      aws.region = 'us-west-2'  
      aws.ami = 'ami-20be7540'  
      aws.security_groups = ['default']  
      override.ssh.username = 'ansible'  
      override.ssh.private_key_path = '~/ec2/keys/vagrant.pem'  
      aws.tags = {  
       'Name' => "ansible"  
      }  
    end  
  end  

What IP's are on my subnet?

1.) Clear arp cache

sudo arp -ad

2.) Ping your subnet, for example...

ping 10.1.1.255

3.) Now look at arp entries

arp -a

Alternatively carry out an nmap ping scan (nmap uses ARP under the covers for this)

nmap -sn 10.1.1.0/24

port scanning

Use netcat to check if UDP port 53 is open at IP 10.0.2.3

nc -vz -u 10.0.2.3 53

Use natcat to check if TCP ports 2000 to 2010 are open at IP 192.168.2.100

nc -zv 192.168.2.100 2000-2010

Use nmap with TCP connect option to check if TCP ports 2000 to 3000 are open at IP 10.1.1.34

nmap -sT -p 2000-3000 10.1.1.34

Same as above but skip nmap's host discovery (in case the target host is blocking ping probes)

nmap -sT -p 2000-3000 10.1.1.34 -Pn

Use nmap with SYN scan option (the default) to check if TCP port 22 is open at IP 10.1.1.70

nmap -sS -p 22 10.0.2.15

Use nmap to check if UDP ports 53,161 and 162 are open at IP 192.168.1.1

nmap -sU -p 53,161,162 192.168.1.1

Linux: one-liner group, user account and sudo setup

Running the following as root will:

• create a group called ansible with an id of 30000
• create a group called devops with an id of 30001
• create a user called ansible with a password of mypass123 with a primary group of ansible and a secondary group of devops
• setup passwordless root access via sudo for the ansible account 

groupadd --gid 30000 ansible && \
groupadd --gid 30001 devops && \
useradd --uid 30000 ansible -m -g ansible --groups devops && \
echo -e "mypass123\nmypass123" | passwd ansible && \
echo "%devops  ALL=(ALL)  NOPASSWD: ALL" > /etc/sudoers.d/10_devops

Find which process is listening on a port with LSOF

You can usually use the -p option in netstat to find out which process is listening on a specific port as follows:

netstat -anp | grep 22

This would provide something to the following which is telling you that port 22 has been opened for listening by process sshd who's PID is 954

tcp  0  0  0.0.0.0:22  0.0.0.0:*  LISTEN  954/sshd 

If you're in a fix and need a different method, good old fashioned LSOF can do the job 

lsof -i :22

Which would provide something similar to the following:

COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd     954    root    3u  IPv4  16348      0t0  TCP *:ssh (LISTEN)
sshd     954    root    4u  IPv6  16357      0t0  TCP *:ssh (LISTEN)

Obtain information about installed packages on CentOS and Ubuntu

CentOS / RHEL / Fedora

List all installed packages

rpm -qa

Get information about a specific package

rpm -qi <package name> 

List all files in a specific package

rpm -ql <package name> 

Combine the -a and -l options, in this exmaple we're searching for the package elasticsearch-5.5.2-1.noarch

package=$(rpm -qa | grep elasticsearch) && rpm -ql $package; 

Ubuntu

List all installed packages

dpkg-query -l

List all files in a specific package

dpkg-query -L <package name>