Skip to main content
Version: 1.2.x

Sink S3

Purpose​

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Next to Amazon's S3 there are now various object storage providers which grant S3 compatible access to their storage solutions as well (e.g. Google Cloud Storage, IONOS, et al).

This UI helps to define the outbound connection parameters for an AWS S3 sink.

This Asset can be used by:​

Asset typeLink
Output ProcessorsStream Output Processor

Prerequisite​

You need:

Configuration​

Name & Description​

Name : Name of the Asset. Spaces are not allowed in the name.

Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

Required roles​

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

S3 Settings​

S3 Connection (A)​

Select the AWS Connection to use for this Asset. If it does not exist, you need to create it first.

S3 Bucket (B)​

  • S3 bucket name (B): Once you picked a S3 Connection above the system will try to test the connection and list bucket names it can find. These will be available for selection here. You can check how many buckets could be found at (6)

  • Prefix (1): If you pick a valid bucket (B), then available prefixes (folders) will be available for selection here. You can check how many prefixes could be found for a given bucket at (7)

  • Object Prefix / Object Suffix (2): You can narrow down your bucket selection further by putting object prefix / suffix filters.

  • "Object already exists"-Handling (3): Define your required handling in case the object "in process" already exists.

  • Use path style bucket access (4): The S3 API allows accessing objects via legacy "path style" or "virtual hosted style". Check this box if you want to access objects via legacy path style access. You can read more about this here

  • Include sub folders (5): Check this box if you want sub-folders to be included when querying the source. This means that all objects from sub-folders within a bucket/prefix combination will be considered for processing.

While you are entering and changing S3 bucket parameters, layline.io frequently tries to connect to the endpoint and retrieve bucket and prefix information. The status of these attempts is displayed at the bottom of the group box.

Internal​

External​

Potential problems​

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .