Skip to content

Commit b67703f

Browse files
benjaminhuobenjaminhuo
authored
Merge pull request #314 from wenchajun/master
Organizing documents
2 parents aec1bc8 + 13e7700 commit b67703f

File tree

2 files changed

+107
-102
lines changed

2 files changed

+107
-102
lines changed

‎README.md

+5-102
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,11 @@ Once installed, the Fluent Operator provides the following features:
4444
- [Deploy the Kubernetes logging stack with Helm](#deploy-the-kubernetes-logging-stack-with-helm)
4545
- [Collect auditd logs](#collect-auditd-logs)
4646
- [Fluentd](#fluentd-1)
47-
- [Monitoring](#monitoring)
4847
- [Plugins](#plugins)
4948
- [Fluent Bit](#fluent-bit-1)
5049
- [Fluentd](#fluentd-2)
5150
- [Best Practice](#best-practice)
5251
- [Plugin Grouping](#plugin-grouping)
53-
- [Path Convention](#path-convention)
5452
- [Custom Parser](#custom-parser)
5553
- [Roadmap](#roadmap)
5654
- [Development](#development)
@@ -235,83 +233,6 @@ Fluentd acts as a log forward layer that receives logs from Fluent Bit or other
235233

236234
For more info on various use cases of Fluent Operator Fluentd CRDs, you can refer to [Fluent-Operator-Walkthrough](https://github.com/kubesphere-sigs/fluent-operator-walkthrough#fluent-bit--fluentd-mode).
237235

238-
## Monitoring
239-
240-
Fluent Bit comes with a built-in HTTP Server. According to the official [documentation](https://docs.fluentbit.io/manual/administration/monitoring) of fluentbit You can enable this by enabling the HTTP server from the fluent bit configuration file:
241-
242-
```conf
243-
[SERVICE]
244-
HTTP_Server On
245-
HTTP_Listen 0.0.0.0
246-
HTTP_PORT 2020
247-
```
248-
249-
When you use the fluent-operator, You can enable this from `FluentBitConfig` manifest. Example is below:
250-
251-
```yaml
252-
apiVersion: fluentbit.fluent.io/v1alpha2
253-
kind: ClusterFluentBitConfig
254-
metadata:
255-
name: fluent-bit-config
256-
labels:
257-
app.kubernetes.io/name: fluent-bit
258-
spec:
259-
filterSelector:
260-
matchLabels:
261-
fluentbit.fluent.io/enabled: 'true'
262-
inputSelector:
263-
matchLabels:
264-
fluentbit.fluent.io/enabled: 'true'
265-
outputSelector:
266-
matchLabels:
267-
fluentbit.fluent.io/enabled: 'true'
268-
service:
269-
httpListen: 0.0.0.0
270-
httpPort: 2020
271-
httpServer: true
272-
parsersFile: parsers.conf
273-
274-
```
275-
276-
Once HTTP server is enabled, you should be able to get the information:
277-
278-
```bash
279-
curl <podIP>:2020 | jq .
280-
281-
{
282-
"fluent-bit": {
283-
"version": "1.8.3",
284-
"edition": "Community",
285-
"flags": [
286-
"FLB_HAVE_PARSER",
287-
"FLB_HAVE_RECORD_ACCESSOR",
288-
"FLB_HAVE_STREAM_PROCESSOR",
289-
"FLB_HAVE_TLS",
290-
"FLB_HAVE_OPENSSL",
291-
"FLB_HAVE_AWS",
292-
"FLB_HAVE_SIGNV4",
293-
"FLB_HAVE_SQLDB",
294-
"FLB_HAVE_METRICS",
295-
"FLB_HAVE_HTTP_SERVER",
296-
"FLB_HAVE_SYSTEMD",
297-
"FLB_HAVE_FORK",
298-
"FLB_HAVE_TIMESPEC_GET",
299-
"FLB_HAVE_GMTOFF",
300-
"FLB_HAVE_UNIX_SOCKET",
301-
"FLB_HAVE_PROXY_GO",
302-
"FLB_HAVE_JEMALLOC",
303-
"FLB_HAVE_LIBBACKTRACE",
304-
"FLB_HAVE_REGEX",
305-
"FLB_HAVE_UTF8_ENCODER",
306-
"FLB_HAVE_LUAJIT",
307-
"FLB_HAVE_C_TLS",
308-
"FLB_HAVE_ACCEPT4",
309-
"FLB_HAVE_INOTIFY"
310-
]
311-
}
312-
}
313-
```
314-
315236

316237
## Plugins
317238

@@ -327,27 +248,9 @@ curl <podIP>:2020 | jq .
327248

328249
### Plugin Grouping
329250

330-
Input, filter, and output plugins are connected by label selectors. For input and output plugins, always create `Input` or `Output` CRs for every plugin. Don't aggregate multiple inputs or outputs into one `Input` or `Output` object, except you have a good reason to do so. Take the demo `logging stack` for example, we have one yaml file for each output.
331-
332-
However, for filter plugins, if you want a filter chain, the order of filters matters. You need to organize multiple filters into an array as the demo [logging stack](manifests/logging-stack/filter-kubernetes.yaml) suggests.
333-
334-
### Path Convention
335-
336-
Path to file in Fluent Bit config should be well regulated. Fluent Bit Operator adopts the following convention internally.
337-
338-
|Dir Path|Description|
339-
|---|---|
340-
|/fluent-bit/tail|Stores tail related files, eg. file tracking db. Using [fluentbit.spec.positionDB](docs/fluentbit.md#fluentbitspec) will mount a file `pos.db` under this dir by default.|
341-
|/fluent-bit/secrets/{secret_name}|Stores secrets, eg. TLS files. Specify secrets to mount in [fluentbit.spec.secrets](docs/fluentbit.md#fluentbitspec), then you have access.|
342-
|/fluent-bit/config|Stores the main config file and user-defined parser config file.|
343-
344-
> Note that ServiceAccount files are mounted at `/var/run/secrets/kubernetes.io/serviceaccount`.
345-
346-
## Custom Parser
347-
348-
To enable parsers, you must set the value of `FluentBitConfig.Spec.Service.ParsersFile` to `parsers.conf`. Your custom parsers will be included into the built-in parser config via `@INCLUDE /fluent-bit/config/parsers.conf`. Note that the parsers.conf contains a few built-in parsers, for example, docker. Read [parsers.conf](https://github.com/kubesphere/fluentbit-operator/blob/master/conf/parsers.conf) for more information.
251+
For fluenbit,input, filter, and output plugins are connected by label selectors. For input and output plugins, always create `ClusterInput` or `ClusterOutput` CRs for every plugin. Don't aggregate multiple inputs or outputs into one `ClusterInput` or `ClusterOutput` object, except you have a good reason to do so. Take the demo `logging stack` for example, we have one yaml file for each output.
349252

350-
Check out the demo in the folder `/manifests/regex-parser` for how to use a custom regex parser.
253+
However, for filter plugins, if you want a filter chain, the order of filters matters. You need to organize multiple filters into an array as the demo [logging stack](manifests/logging-stack/filter-kubernetes.yaml) suggests.For more info on various use cases of Fluent Operator Fluentd CRDs, you can refer to [Fluent-Operator-Walkthrough](https://github.com/kubesphere-sigs/fluent-operator-walkthrough#fluent-bit--fluentd-mode).
351254

352255
## Roadmap
353256

@@ -361,9 +264,9 @@ Check out the demo in the folder `/manifests/regex-parser` for how to use a cust
361264

362265
### Requirements
363266
- golang v1.16+.requirement
364-
- kubectl v1.16.13+.
365-
- kubebuilder v2.3+ (the project is build with v2.3.2)
366-
- Access to a Kubernetes cluster v1.16.13+
267+
- kubectl v1.19.2+.
268+
- kubebuilder v3.1+ (the project is build with v3.1.0)
269+
- Access to a Kubernetes cluster v1.19.2+
367270

368271
### Running
369272

‎docs/fluentbit.md

+102
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ This Document documents the types introduced by the fluentbit Operator.
2424
* [ParserSpec](#parserspec)
2525
* [Script](#script)
2626
* [Service](#service)
27+
* [Monitoring](#monitoring)
28+
* [Path Convention](#path Convention)
29+
* [Custom Parser](#custom Parser)
30+
2731
# ClusterFilter
2832

2933
ClusterFilter defines a cluster-level Filter configuration.
@@ -321,3 +325,101 @@ ParserSpec defines the desired state of ClusterParser
321325
| parsersFile | Optional 'parsers' config file (can be multiple) | string |
322326

323327
[Back to TOC](#table-of-contents)
328+
# Monitoring
329+
330+
Fluent Bit comes with a built-in HTTP Server. According to the official [documentation](https://docs.fluentbit.io/manual/administration/monitoring) of fluentbit You can enable this by enabling the HTTP server from the fluent bit configuration file:
331+
332+
```conf
333+
[SERVICE]
334+
HTTP_Server On
335+
HTTP_Listen 0.0.0.0
336+
HTTP_PORT 2020
337+
```
338+
339+
When you use the fluent-operator, You can enable this from `FluentBitConfig` manifest. Example is below:
340+
341+
```yaml
342+
apiVersion: fluentbit.fluent.io/v1alpha2
343+
kind: ClusterFluentBitConfig
344+
metadata:
345+
name: fluent-bit-config
346+
labels:
347+
app.kubernetes.io/name: fluent-bit
348+
spec:
349+
filterSelector:
350+
matchLabels:
351+
fluentbit.fluent.io/enabled: 'true'
352+
inputSelector:
353+
matchLabels:
354+
fluentbit.fluent.io/enabled: 'true'
355+
outputSelector:
356+
matchLabels:
357+
fluentbit.fluent.io/enabled: 'true'
358+
service:
359+
httpListen: 0.0.0.0
360+
httpPort: 2020
361+
httpServer: true
362+
parsersFile: parsers.conf
363+
364+
```
365+
366+
Once HTTP server is enabled, you should be able to get the information:
367+
368+
```bash
369+
curl <podIP>:2020 | jq .
370+
371+
{
372+
"fluent-bit": {
373+
"version": "1.8.3",
374+
"edition": "Community",
375+
"flags": [
376+
"FLB_HAVE_PARSER",
377+
"FLB_HAVE_RECORD_ACCESSOR",
378+
"FLB_HAVE_STREAM_PROCESSOR",
379+
"FLB_HAVE_TLS",
380+
"FLB_HAVE_OPENSSL",
381+
"FLB_HAVE_AWS",
382+
"FLB_HAVE_SIGNV4",
383+
"FLB_HAVE_SQLDB",
384+
"FLB_HAVE_METRICS",
385+
"FLB_HAVE_HTTP_SERVER",
386+
"FLB_HAVE_SYSTEMD",
387+
"FLB_HAVE_FORK",
388+
"FLB_HAVE_TIMESPEC_GET",
389+
"FLB_HAVE_GMTOFF",
390+
"FLB_HAVE_UNIX_SOCKET",
391+
"FLB_HAVE_PROXY_GO",
392+
"FLB_HAVE_JEMALLOC",
393+
"FLB_HAVE_LIBBACKTRACE",
394+
"FLB_HAVE_REGEX",
395+
"FLB_HAVE_UTF8_ENCODER",
396+
"FLB_HAVE_LUAJIT",
397+
"FLB_HAVE_C_TLS",
398+
"FLB_HAVE_ACCEPT4",
399+
"FLB_HAVE_INOTIFY"
400+
]
401+
}
402+
}
403+
```
404+
405+
[Back to TOC](#table-of-contents)
406+
# Path Convention
407+
408+
Path to file in Fluent Bit config should be well regulated. Fluent Bit Operator adopts the following convention internally.
409+
410+
| Dir Path | Description |
411+
| --------------------------------- | ------------------------------------------------------------ |
412+
| /fluent-bit/tail | Stores tail related files, eg. file tracking db. Using [fluentbit.spec.positionDB](docs/fluentbit.md#fluentbitspec) will mount a file `pos.db` under this dir by default. |
413+
| /fluent-bit/secrets/{secret_name} | Stores secrets, eg. TLS files. Specify secrets to mount in [fluentbit.spec.secrets](docs/fluentbit.md#fluentbitspec), then you have access. |
414+
| /fluent-bit/config | Stores the main config file and user-defined parser config file. |
415+
416+
> Note that ServiceAccount files are mounted at `/var/run/secrets/kubernetes.io/serviceaccount`.
417+
418+
[Back to TOC](#table-of-contents)
419+
# Custom Parser
420+
421+
To enable parsers, you must set the value of `FluentBitConfig.Spec.Service.ParsersFile` to `parsers.conf`. Your custom parsers will be included into the built-in parser config via `@INCLUDE /fluent-bit/config/parsers.conf`. Note that the parsers.conf contains a few built-in parsers, for example, docker. Read [parsers.conf](https://github.com/kubesphere/fluentbit-operator/blob/master/conf/parsers.conf) for more information.
422+
423+
Check out the demo in the folder `/manifests/regex-parser` for how to use a custom regex parser.
424+
425+
[Back to TOC](#table-of-contents)

0 commit comments

Comments
 (0)