Легковесный сборщик логов на примере FluentD в K8S и WERF

project: fluentd
configVersion: 1
---
image: ~
from: fluent/fluentd:v1.9.1-1.0
docker:
USER: root
ansible:
install:
- name: Install plugins
shell: gem install fluent-plugin-logzio
become_user: root
# werf helm create NAME [flags] [options]
.helm/templates/_helpers.tpl
.helm/templates/config.yaml
.helm/templates/deployment.yaml
.helm/values.yaml
werf.yml

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "fluentd.fullname" . }}
labels:
{{- include "fluentd.labels" . | nindent 4 }}
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
{{- include "fluentd.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "fluentd.selectorLabels" . | nindent 8 }}
kubernetes.io/cluster-service: "true"
spec:
volumes:
- name: {{ include "fluentd.fullname" . }}-logs
hostPath:
path: /var/log

- name: {{ include "fluentd.fullname" . }}-container-logs
hostPath:
path: /var/lib/docker/containers

- name: {{ include "fluentd.fullname" . }}-config-volume
configMap:
name: {{ include "fluentd.fullname" . }}-config
items:
- key: fluent.conf
path: fluent.conf

tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: {{ .Chart.Name }}
{{ werf_container_image . | indent 0 }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: {{ include "fluentd.fullname" . }}-logs
mountPath: /var/log

- name: {{ include "fluentd.fullname" . }}-container-logs
mountPath: /var/lib/docker/containers
readOnly: true

- mountPath: /fluentd/etc/fluent.conf
subPath: fluent.conf
readOnly: true
name: {{ include "fluentd.fullname" . }}-config-volume
resources:
{{- toYaml .Values.resources | nindent 12 }}

Config map (.helm/templates/config.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "fluentd.fullname" . }}-config
labels:
{{- include "fluentd.labels" . | nindent 4 }}
data:
fluent.conf: |
<source>
@type tail
path /var/log/containers/*apps*.log
pos_file /var/log/fluentd-containers.log.pos
tag apps.*
<parse>
@type json
</parse>
</source>

<match apps.**>
@type logzio_buffered
endpoint_url https://listener.logz.io:8071?token=&type=
output_include_time true
output_include_tags true
http_idle_timeout 10
<buffer>
@type memory
flush_thread_count 4
flush_interval 3s
chunk_limit_size 16m
queue_limit_length 4096
</buffer>
</match>

GitHub Action

Secrets:

name: Stage

on:
push:
branches:
- master

jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Set tag
id: vars
run: |
echo ::set-output name=tag::$(echo $GITHUB_REF | cut -d'/' -f 3)-$(echo $GITHUB_SHA | cut -c1-8)

- name: Registry login
run: echo $PASSWORD | docker login -u $USER --password-stdin
env:
USER: ${{ secrets.REGISTRY_USER }}
PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}

- name: Сonverge
uses: werf/actions/converge@master
env:
WERF_STAGES_STORAGE: ":local"
WERF_TAG_BY_STAGES_SIGNATURE: false
WERF_IMAGES_REPO: ${{ secrets.REGISTRY_REPOSITORY }}
WERF_TAG_CUSTOM1: ${{ steps.vars.outputs.tag }}
WERF_NAMESPACE: infrastructure
with:
kube-config-base64-data: ${{ secrets.KUBE_CONFIG_BASE64_DATA_STAGE }}
env: stage

Шаги workflow:

  1. Runner клонирует код;

Первые проблемы с разбором текста: JSON не совсем JSON.

2020–10–27 16:48:32 +0000 [warn]: #0 pattern not matched: "2020–10–27T19:48:31.859783645+03:00 stdout F {\"trace_id\": \"3a86554a4d41197e829adef23d431301\"}
<source>
@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag apps.*
read_from_head true
<parse>
@type regexp
expression /^(?<timestamp>.*?)\s(?<system>.*?)\s(?<log>\{.*\})$/
time_key timestamp
</parse>
</source>
<filter apps.**>
@type parser
key_name log
reserve_data true
remove_key_name_field true
replace_invalid_sequence true
reserve_time true
<parse>
@type json
json_parser json
</parse>
</filter>
{
"timestamp":"2020–10–28T12:25:43.787975275+03:00",
"system":"stdout F",
"trace_id":"bd4ba83bd42e9b4b036544db42fb33fc",
"remote_addr":"194.61.2.200",
"remote_user":"",
"time_local":"28/Oct/2020:09:25:43 +0000",
"request":"GET /health HTTP/1.1",
"status":"200",
"body_bytes_sent":"93",
"http_referer":"",
"http_user_agent":"kube-probe/1.19+"
}
<match fluent.**>
@type null
</match>

<match **fluentd**.log>
@type null
</match>

<match **kube-system**.log>
@type null
</match>

<source>
@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag apps.*
read_from_head true
<parse>
@type regexp
expression /^(?<timestamp>.*?)\s(?<system>.*?)\s(?<log>\{.*\})$/
time_key timestamp
</parse>
</source>

<filter apps.**>
@type parser
key_name log
reserve_data true
remove_key_name_field true
replace_invalid_sequence true
reserve_time true
<parse>
@type json
json_parser json
</parse>
</filter>

<match apps.**>
@type logzio_buffered
endpoint_url https://listener.logz.io:8071?token=&type=
output_include_time true
output_include_tags true
http_idle_timeout 10
<buffer>
@type memory
flush_thread_count 4
flush_interval 3s
chunk_limit_size 16m
queue_limit_length 4096
</buffer>
</match>

Итог:

  1. Ускорилась работа приложения. Больше не тратится время на установку. соединения с коллектором.
  2. Если коллектор перезагружается, то это никак не влияет на работу приложения.
  3. Потребление ресурсов сократилось.
  4. Если logz.io будет не доступен, то FluentD будет складывать логи в RAM, а если памяти выделенной на POD не останется, то FluentD просто перестанет читать логи до высвобождения оперативной памяти, после чего просто продолжит читать с места где остановился.

--

--

Castle builder, pragmatic, software architect

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Viktor Gievoi

Viktor Gievoi

Castle builder, pragmatic, software architect