Kubernetes to rsyslog via FluentD

September 14, 2023 2-minute read

kubernetes • rsyslog • flientd

After a few hours of scouring documentation and reviewing examples, I couldn’t find a complete working example of how to get FluentD running as a Daemonset to forward logs to rsyslog. Below is a complete working example that I cobbled together.

Rsyslog config

The following additions need to go into /etc/rsyslog.conf or /etc/rsyslog.d/your.conf.

Enable the udp listener for rsyslog:

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="1514")

Create a template for Kubernetes logs

$template KubernetesLogs,"/var/log/remote/kubernetes/%msg:R,ERE,1,BLANK:\"namespace_name\"=>\"([a-zA-z0-9_-]+)\",--end:secpath-replace%_%msg:R,ERE,1,BLANK:\"container_name\"=>\"([a-zA-z0-9_-]+)\",--end:secpath-replace%.log"

Create a directive for udp messages to get passed to the KubernetesLogs template

if ($inputname contains "imudp") then {
  *.* ?KubernetesLogs 
  & stop
}

Restart rsyslog

Now all logs received will be placed into /var/log/kubernetes/<namespace_name>_<container_name>.log.

Kubernetes Manifest

Replace the SYSLOG_HOST value with the ip/hostname of your syslog server then run kubectl create.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-syslog
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name:  SYSLOG_HOST
            value: "10.1.1.250"
          - name:  SYSLOG_PORT
            value: "514"
          - name:  SYSLOG_PROTOCOL
            value: "udp"
          - name: FLUENT_UID
            value: "0"
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluent*
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: syslog
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        # When actual pod logs in /var/lib/docker/containers, the following lines should be used.
        - name: dockercontainerlogdirectory
          mountPath: /var/lib/docker/containers
          readOnly: true
        # When actual pod logs in /var/log/pods, the following lines should be used.
        #- name: dockercontainerlogdirectory
        #  mountPath: /var/log/pods
        #  readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      # When actual pod logs in /var/lib/docker/containers, the following lines should be used.
      - name: dockercontainerlogdirectory
        hostPath:
          path: /var/lib/docker/containers
      # When actual pod logs in /var/log/pods, the following lines should be used.
      #- name: dockercontainerlogdirectory
      #  hostPath:
      #    path: /var/log/pods