Skip to main content
Version: Next

AWS S3

This module allows the Security Engine to acquire logs from a S3 bucket.

It supports reading plain text file and gzip file (detection is performed based on the file extension).

Configuration example​

To monitor a S3 bucket detecting new objects from a SQS queue:

source: s3
polling_method: sqs
sqs_name: test-sqs-s3-acquis
use_time_machine: true
labels:
  type: foo

To monitor a S3 bucket detecting new objects by listing the bucket content:

source: s3
polling_method: list
bucket_name: my_bucket
polling_interval: 30
use_time_machine: true
labels:
  type: foo

It is strongly recommended to set use_time_machine: true when using the S3 data source. Since files from S3 are not read in real time, the parser must rely on the timestamps within the log lines themselves to process events accurately.

The list polling method is mostly intended for testing purposes, and its usage is not recommended in production. It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow.

When using the sqs polling method, make sure the Security Engine is the only reader of the queue. If other processes read from the queue, then the Security Engine will miss some events.

Look at the configuration parameters to view all supported options.

Parameters​

polling_method​

How to detect new files in a bucket. Must be one of sqs or list. sqs is the recommended mode.

polling_interval​

How often in seconds the Security Engine will check for new objects in a bucket when using the list polling method. Defaults to 60.

sqs_name​

Name of the SQS queue to poll. Required when polling_method is sqs.

sqs_format​

Format of the body inside the SQS messages. Can be eventbridge, s3notification or sns.

If not set, the Security Engine will automatically select the format based on the first valid event received from the queue.

bucket_name​

Name of the bucket to poll. Required when polling_method is list.

prefix​

Only read objects matching this prefix when polling_method is list. Optional, ignored when polling_method is sqs.

aws_profile​

The AWS profile to use, relies on your ~/.aws/config/.

Optional, the data source will automatically use the standard AWS env vars if present.

aws_region​

The AWS region.

Optional, the data source will automatically use the standard AWS env vars if present.

aws_endpoint​

Endpoint for AWS API. Optional, the data source will automatically use the standard AWS env vars if present.

Can be used to point the Security Engine to a S3-compatible object storage.

source​

Must be s3

DSN and command-line​

This module supports acquisition directly from the command line, to read files in one shot.

A single s3 URI is accepted with the -dsn parameter, but you don't have to specify a specific object.

If no object is specified (either just a bucket, or a bucket and a prefix), the Security Engine will read all files matching the prefix.

If you don't specify an object, the path must end with /.

crowdsec -type syslog -dsn s3://my_bucket/
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/
crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log

You can specify the log_level parameter to change the log level for the acquisition:

crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug

AWS SDK behaviour can be configured with the standard AWS environment variables.

IAM Permissions​

Because the component needs to interact with AWS resources, it need the proper permissions.

Here is the set of required permissions:

{
    "Statement": [
        {
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ListDeadLetterSourceQueues",
                "sqs:ListQueues"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:::test-sqs-s3-acquis"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:DescribeJob",
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::my_bucket:*"
        }
    ],
    "Version": "2012-10-17"
}

For the permissions, we recommend to restrict the S3 permissions to read only operations, to avoid the ability to destroy logs from the CrowdSec agent. If you are using S3 polling, the SQS part of the permissions can be omitted.