AWS S3

This module allows the Security Engine to acquire logs from a S3 bucket.

It supports reading plain text file and gzip file (detection is performed based on the file extension).

Configuration example

To monitor a S3 bucket detecting new objects from a SQS queue:

source: s3
polling_method: sqs
sqs_name: test-sqs-s3-acquis
labels:
  type: foo

To monitor a S3 bucket detecting new objects by listing the bucket content:

source: s3
polling_method: list
bucket_name: my_bucket
polling_interval: 30
labels:
  type: foo

The list polling method is mostly intended for testing purposes, and its usage is not recommended in production. It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow.

When using the sqs polling method, make sure the Security Engine is the only reader of the queue. If other processes read from the queue, then the Security Engine will miss some events.

Look at the configuration parameters to view all supported options.

Parameters

polling_method

How to detect new files in a bucket. Must be one of sqs or list. sqs is the recommended mode.

polling_interval

How often in seconds the Security Engine will check for new objects in a bucket when using the list polling method. Defaults to 60.

sqs_name

Name of the SQS queue to poll. Required when polling_method is sqs.

sqs_format

Format of the body inside the SQS messages. Can be eventbridge or s3notification.

If not set, the Security Engine will automatically select the format based on the first valid event received from the queue.

bucket_name

Name of the bucket to poll. Required when polling_method is list.

prefix

Only read objects matching this prefix when polling_method is list. Optional, ignored when polling_method is sqs.

aws_profile

The AWS profile to use, relies on your ~/.aws/config/.

Optional, the data source will automatically use the standard AWS env vars if present.

aws_region

The AWS region.

Optional, the data source will automatically use the standard AWS env vars if present.

aws_endpoint

Endpoint for AWS API. Optional, the data source will automatically use the standard AWS env vars if present.

Can be used to point the Security Engine to a S3-compatible object storage.

source

Must be s3

DSN and command-line

This module supports acquisition directly from the command line, to read files in one shot.

A single s3 URI is accepted with the -dsn parameter, but you don't have to specify a specific object.

If no object is specified (either just a bucket, or a bucket and a prefix), the Security Engine will read all files matching the prefix.

If you don't specify an object, the path must end with /.

crowdsec -type syslog -dsn s3://my_bucket/

crowdsec -type syslog -dsn s3://my_bucket/my_prefix/

crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log

You can specify the log_level parameter to change the log level for the acquisition:

crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug

AWS SDK behaviour can be configured with the standard AWS environment variables.