# Source FTP

# Purpose

Defines the specific bucket and folder source of a FTP connected endpoint. This Asset is required by:

Stream Input Processor

# Prerequisite

You need:

FTP Connection

# Configuration

# Name & Description

Name & Description (FTP Source)

Name : Name of the Asset. Whitespaces are not allowed in the name.

Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

# Required roles

Required Roles (FTP Source)

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

# Polling & Processing

Polling & Processing (FTP Source)

FTP is not a stream, but an object based storage source which does not signal the existence of new objects to observers. We therefore need to define how often we want to look-up (poll) the source for new objects to process.

You can choose between Fixed rate polling and Cron tab style polling:

# Fixed rate

Use Fixed rate if you want to poll in constant and frequent intervals.

Polling interval [sec]: Enter the interval in seconds in which the configured source should be queried for new objects.

# Cron tab

Cron Tab (FTP Source)

Use Cron tab if you want to poll at determined times. This follows the cron tab style conventioni which may be familiar to you. In all other cases you can read more about crontab and the syntax here (opens new window). You can simulate cron settings using this smart website (opens new window).

Examples:

0/2 2 3 4 5 (opens new window): "At every 2nd minute from 0 through 59 past hour 2 on day-of-month 3 and on Friday in April."
0 22 * * 1-5 (opens new window): "At 22:00 on every day-of-week from Monday through Friday."

# Polling timeout

The Polling timeout [sec] defines the time in seconds to wait until a polling request fails. Depending on the endpoint and its responsiveness you may want to change this number to something higher or lower. You should set it high enough, so that you are confident that the endpoint responds under normal operation.

# Ordering

When listing objects from the FTP source for processing, you can define in what order they should be processed. Pick one of the following self-explanatory settings:

Alphabetically, ascending
Alphabetically, descending
Last modified, ascending
Last modified, descending

# FTP Connection

FTP Connection (FTP Source)

Select the previously configured FTP Connection to use for this Source.

# Directories

An FTP source requires the definition of three different directories:

Input Directory : The directory to read new files from.
Done Directory : The directory to which read files are moved after reading.
Error Directory : Files which caused problems during processing are moved to the Error Directory for further analysis.

# Input Directory

Input Directory (FTP Source)

Input Directory : The directory to read files from. The path of the directory must be accessible to the Reactive Engine trying to access the FTP source. You can use ${...} placeholders to expand variables defined in environment variables.
Filter regular expression : Regular expression to filter which files in the directory are pulled.
File prefix regular expression : A regular expression filter which is applied to the beginning of a file name. E.g. XYZ. will lead to only those files read which filename starts with XYZ followed by anything.
File suffix regular expression : A regular expression filter which is applied to the end of a file name. E.g. .zip will lead to only those files read which filename ends with zip preceded by anything.
Include sub-directories : Scan sub-directories to the input directory also.

# Done Directory

Done Directory (FTP Source)

Done Directory : The directory to which files are moved when fully processed. The path of the directory must be accessible to the Reactive Engine trying to access the FTP source. You can use ${...} placeholders to expand variables defined in environment variables.
Done prefix : Prefix to add to the filename of the processed file after move to the done directory. E.g. done_ will add the done_-prefix to the beginning of the filename when moved to the done directory.
Done suffix : Suffix to add to the filename of the processed file after move to the done directory. E.g. _done will add the _done-suffix to the end of the filename when moved to the done directory.

# Error Directory

Error Directory (FTP Source)

Error Directory : The directory to which files are moved in case of a problem with the file during processing. The path of the directory must be accessible to the Reactive Engine trying to access the FTP source. You can use ${...} placeholders to expand variables defined in environment variables.
- Error prefix : Prefix to add to the filename of the processed file after move to the error directory. E.g. error_ will add the error_-prefix to the beginning of the filename when moved to the error directory.
- Error suffix : Suffix to add to the filename of the processed file after move to the error directory. E.g. _error will add the _error-suffix to the end of the filename when moved to the error directory.

# Internal

# External

# Potential problems

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. Should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .

← Throttle Source Kafka →