Skip to content

Introduction

Data sources, are in charge of dictating to crowdsec-agent which source should be read.

Data sources are configured via /etc/crowdsec/acquis.yaml or via the directory specified in acquisition_dir.

Info

Please note that the /etc/crowdsec/acquis.yaml should be auto generated by the wizard in most case.

Acquisition configuration

The acquisition configuration specifies lists of logs sources that crowdsec-agent will ingest and feed to parsers. Acquisition provides at least two information about a given log :

  • a source (a path to a file, a journalctl filter, a cloudwatch group/stream etc.)
  • a type, given in the form of a label

Extra parameters are as well supported :

  • a name that will ease tracking and logs readability
  • a log_level to individually configure the log level of a given source

The type label is crucial as it's later used in the process to determine which parser(s) can handle lines coming from this source.

Acquisition can be found in /etc/crowdsec/acquis.yaml, for example :

Acquisition example

source: file
log_level: debug
filenames:
  - /var/log/nginx/access-*.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
source: journalctl
journalctl_filter:
 - "_SYSTEMD_UNIT=ssh.service"
labels:
  type: syslog
---
source: cloudwatch
group_name: /aws/group/name
aws_profile: production
labels:
  type: syslog

Testing and viewing acquisition

At startup

At startup, you will see the monitored files in /var/log/crowdsec.log :

...
INFO[23-11-2020 15:21:17] [file datasource] opening file '/tmp/test.log' 
WARN[23-11-2020 15:21:17] [file datasource] no results for /tmp/ratata.log 
INFO[23-11-2020 15:21:17] [journald datasource] Configured with filters : [--follow _SYSTEMD_UNIT=ssh.service] 
...

At runtime

cscli allows you to view crowdsec-agent metrics info via the metrics command. This allows you to see how many lines are coming from each source, and if they are parsed correctly.

You can see those metrics with the following command:

sudo cscli metrics
cscli metrics example
$ sudo cscli metrics
...
...
INFO[0000] Acquisition Metrics:     
+--------------------------------------+------------+--------------+----------------+------------------------+
|                SOURCE                | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------+------------+--------------+----------------+------------------------+
| /tmp/test.log                        |         10 |           10 | -              |                     11 |
| journalctl-_SYSTEMD_UNIT=ssh.service |         36 |           12 |             24 |                     17 |
+--------------------------------------+------------+--------------+----------------+------------------------+
...
...

Info

All these metrics are actually coming from crowdsec-agent's prometheus agent. See prometheus directly for more insights.

Reference documentation

Link to acquisition reference documentation