Parsers

Listing installed parsers

Parsers are yaml files in /etc/crowdsec/config/parsers/<STAGE>/parser.yaml.

Info

Alphabetical file order dictates the order of stage and the orders of parsers within stage.

You can use the following command to view installed parsers:

cscli list parsers
cscli list example
# cscli list parsers
INFO[0000] Loaded 9 collecs, 14 parsers, 12 scenarios, 1 post-overflow parsers 
--------------------------------------------------------------------------------------------------------------------
 NAME                       📦 STATUS    VERSION  LOCAL PATH                                                        
--------------------------------------------------------------------------------------------------------------------
 crowdsec/iptables-logs     ✔️  enabled  0.3      /etc/crowdsec/config/parsers/s01-parse/iptables-logs.yaml     
 crowdsec/dateparse-enrich  ✔️  enabled  0.4      /etc/crowdsec/config/parsers/s02-enrich/dateparse-enrich.yaml 
 crowdsec/sshd-logs         ✔️  enabled  0.3      /etc/crowdsec/config/parsers/s01-parse/sshd-logs.yaml         
 crowdsec/whitelists        ✔️  enabled  0.4      /etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml       
 crowdsec/http-logs         ✔️  enabled  0.4      /etc/crowdsec/config/parsers/s02-enrich/http-logs.yaml        
 crowdsec/nginx-logs        ✔️  enabled  0.3      /etc/crowdsec/config/parsers/s01-parse/nginx-logs.yaml        
 crowdsec/syslog-logs       ✔️  enabled  0.4      /etc/crowdsec/config/parsers/s00-raw/syslog-logs.yaml         
 crowdsec/geoip-enrich      ✔️  enabled  0.4      /etc/crowdsec/config/parsers/s02-enrich/geoip-enrich.yaml     
--------------------------------------------------------------------------------------------------------------------

Installing parsers

From the hub

Crowdsec Hub allows you to find needed scenarios, just paste the command on your machine :

Hub Screenshot

# cscli install parser crowdsec/nginx-logs
INFO[0000] Loaded 9 collecs, 14 parsers, 12 scenarios, 1 post-overflow parsers 
INFO[0000] crowdsec/nginx-logs : OK                     
INFO[0000] Enabled parsers : crowdsec/nginx-logs        
INFO[0000] Enabled crowdsec/nginx-logs                  
# systemctl reload crowdsec

Your own parsers

Write your parser configuration and deploy yaml file in /etc/crowdsec/config/parsers/<STAGE>/.

Monitoring parsers behavior

cscli allows you to view Crowdsec metrics info via the metrics command. This allows you to see how many logs were ingested and then parsed or unparsed by said parser.

You can see those metrics with the following command:

cscli metrics
cscli metrics example
# cscli metrics
...
INFO[0000] Parser Metrics:                              
+---------------------------+--------+--------+----------+
|          PARSERS          |  HITS  | PARSED | UNPARSED |
+---------------------------+--------+--------+----------+
| crowdsec/sshd-logs        |  62424 |  12922 |    49502 |
| crowdsec/syslog-logs      | 667417 | 667417 |        0 |
| crowdsec/whitelists       | 610901 | 610901 |        0 |
| crowdsec/http-logs        |    136 |     21 |      115 |
| crowdsec/iptables-logs    | 597843 | 597843 |        0 |
| crowdsec/nginx-logs       |    137 |    136 |        1 |
| crowdsec/dateparse-enrich | 610901 | 610901 |        0 |
| crowdsec/geoip-enrich     | 610836 | 610836 |        0 |
| crowdsec/non-syslog       |    137 |    137 |        0 |
+---------------------------+--------+--------+----------+

Going further

If you're interested into understanding how parsers are made or writing your own, please have a look at this page.