# Source S3

# Purpose

Defines the specific bucket and folder source of an S3 connected endpoint.

# Prerequisite

You need:

# Configuration

# Name & Description

Name & Description (S3 Source)

Name : Name of the Asset. Whitespaces are not allowed in the name.

Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

# Required roles

Required Roles (S3 Source)

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

# Polling & Processing

Polling & Processing (S3 Source)

S3 is not a stream, but an object based storage source which does not signal the existence of new objects to observers. We therefore need to define how often we want to look-up (poll) the source for new objects to process.

You can choose between Fixed rate polling and Cron tab style polling:

# Fixed rate

Use Fixed rate if you want to poll in constant and frequent intervals.

  • Polling interval [sec]: Enter the interval in seconds in which the configured source should be queried for new objects.

# Cron tab

Cron Tab (S3 Source)

Use Cron tab if you want to poll at determined times. This follows the cron tab style conventioni which may be familiar to you. In all other cases you can read more about crontab and the syntax here (opens new window). You can simulate cron settings using this smart website (opens new window).

Examples:

# Polling timeout

The Polling timeout [sec] defines the time in seconds to wait until a polling request fails. Depending on the endpoint and its responsiveness you may want to change this number to something higher or lower. You should set it high enough, so that you are confident that the endpoint responds under normal operation.

# Ordering

When listing objects from the S3 store for processing, you can define in what order they should be processed. Pick one of the following self-explanatory settings:

  • Alphabetically, ascending
  • Alphabetically, descending
  • Last modified, ascending
  • Last modified, descending

# S3 Connection

S3 Connection (S3 Source)

Select the previously configured S3 Connection to use for this Source.

# S3 Bucket

S3 Bucket (S3 Source)

  • S3 bucket name (1): Once you picked a S3 Connection above the system will try to test the connection and list bucket names it can find. These will be available for selection here. You can check how many buckets could be found at (5)

  • Prefix (2): If you pick a valid bucket (1), then available prefixes (folders) will be available for selection here. You can check how many prefixes could be found for a given bucket at (6)

  • Use path style bucket access (3): The S3 API allows accessing objects via legacy "path style (e.g. https://s3.amazonaws.com/etargett-public/images/example.jpeg)" or "virtual hosted style (e.g. https://etargett-public.s3.amazonaws.com/images/example.jpeg)". Check this box if you want to access objects via legacy path style access. You can read more about this here (opens new window)

  • Include sub folders (4): Check this box if you want sub-folders to be included when querying the source. This means that all objects from sub-folders within a bucket/prefix combination will be considered for processing.

While you are entering and changing S3 bucket parameters, layline.io frequently tries to connect to the endpoint and retrieve bucket and prefix information. The status of these attempts is displayed at the bottom of the group box.

S3 bucket connection success (S3 Source)

In case of error, you can hover the mouse over the red output and view what the problem is:

S3 bucket connection failure (S3 Source)

This usually helps to resolve the issue.

# Internal

# External

# Potential problems

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. Should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .

Last Updated: 7/5/2022, 4:50:56 PM

← Source Timer →