Data Pipeline
Tail
https://docs.fluentbit.io/manual/pipeline/inputs/tail
The tail input plugin allows to monitor one or several text files. It has a similar behavior like tail -f shell command.
The plugin reads every matched file in the Path pattern and for every new line found (separated by a newline character (\n) ), it generates a new record. Optionally a database file can be used so the plugin can have a history of tracked files and a state of offsets, this is very useful to resume a state if the service is restarted.
I was a little confused, but the general flow for creating a TAG can be understood by looking at the code below.
[INPUT]
......
......
Path /var/log/containers/*-*-deploy-*-*_*_*-container-*.log
Tag_Regex containers\/(?<service_name>.+)-deploy-[^-]+-[^-]+_(?<ns_name>.+)_.+-container-[^-]+.log$
Tag <ns_name>.<service_name>
......
......Show OpenSearch Dashboard
I use FluentBit and OpenSearch (with OpenSearch Dashboard).
I can monitor container logs.
