Amazon S3 (w/ SQS)

AWS Simple Storage Service

Fluency has the ability to directly integration with Amazon S3 Buckets to read (and write) the data objects within.

A limitation of the AWS S3 API disallows the listing or sorting of objects by time. Instead, only a lexicographical sort of the object name is allowed. As such, directly consuming a data stream from an S3 bucket require that the objects are created/added in order, both lexicographically and chronologically (i.e. using timestamps are part of the filename).

For buckets where data is added non-lexicographically, AWS S3 buckets has integration with AWS SQS (Simple Queue Service), by way of Event Notifications. This SQS addition allows a consumer to be "notified" when a new object has been added to the S3 bucket, regardless of the sorting of the object name.

By default, Fluency suggests making use of this SQS addition to be notified of new data as they arrive in S3.

Existing S3 Buckets

For clients with existing S3 Buckets, AWS allows the attachment of an SQS queue. This can be done manually, or by following the following CloudFormation script:

https://fluency-cloudformation.s3.us-east-2.amazonaws.com/FluencyPlatformS3AddNotification.yaml

Manual Process

1. Create an SQS queue
2. Create / Attach IAM policy to allow S3 Bucket to post Event Notifications to SQS queue
3. Configure S3 Bucket to send Event Notifications
   1. Select S3 Bucket, Click "Properties" → "Event notifications" → "Create event notification"
   2. Select "All object create events"
   3. Select "SQS queue" as the destination, choosing the SQS queue from step 1

Fluency Configuration

Login to the Fluency portal.

Open the Main Menu and select the Integrations option under the Platform section.