Input Stream

Purpose

Defines input parameters to ingest data from a Service Source.

This Asset is used within a Workflow definition.

Prerequisites

You need:

A Format

• Format

A Stream Source, of either

• File System Source
• FTP Source
• OneDrive Source
• S3 Source
• SharePoint Source
• SMB Source
• TCP Source
• WebDav Source

Name & Description

Name : Name of the Asset. Spaces are not allowed in the name.

Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

Output Ports

This Processor can have one-to-many Output Ports to send messages on within the Workflow.

A port can have a name and description. Names must exist and be unique within the processor.

Termination Settings

Using termination settings you can control how layline.io should behave upon shutdown of a Workflow that this Asset is part of. A Workflow shutdown at most goes through three phases:

1. Signalling shutdown: layline.io signals inputs that it wants to shut down.
2. Signalling abort: In case the shutdown was not confirmed, layline.io sends an abort request.
3. If the abort was not confirmed, layline.io terminates the input without further wait.

• Shutdown grace period [ms] : Time to wait for the input processor to gracefully confirm shutdown once a shutdown request has been received.

• Abort grace period [ms] : In case the shutdown signal was not confirmed in due time, an abort request will be issued. If the abort is not confirmed in the configured time interval, a hard termination will be issued. The abort timeout is consecutive to the shutdown timeout.

Format

Data must be read in a format which must have been previously defined by you.

If you have defined such a format, then you can select it from the list of available formats by clicking on the button next to the field.

Source

You need to assign a Source which you have defined (see Prerequisites). The Source defines which topics can be read from. The Source must have been previously defined.

Select the Sink by clicking on the button next to the field (1).

Other Settings

Compression Mode

In case your incoming stream data is in compressed format, you can define the algorithm to be used for decompression here. If you do not know the compression-algorithm, simply select Auto detect, and the system will try to determine the correct decompression automatically. For uncompressed data, please leave the setting on None.

Rollback on mapping errors

Check this box, if you want the system to issue a rollback request in case the incoming data cannot be mapped to the assigned format. The default is on. If you are unsure about this, then just leave the default.

Failure Handling

Processing within an Input Processor like this one can fail. In this section, you can define how the system should behave in case of problems.

Failure Types

One type of failure is observable:

#	Failure observables / Reaction	Ignore	Retry Event/Message	Retry Stream	Rollback Stream
1	Source failure handling A problem occurred with the Source of the messsages.			✔	✔

Failure Type Reactions

The following two failure reactions are supported.

Rollback Stream

Rollback the complete stream. In the case of batch/file processing for example the complete file (which represents the stream) will be rolled back and put into error. This is the default behavior.

warning

A rollback signal will be issued to all participating Workflow Processors. Each Processor needs to ensure itself how to deal with a rollback. A Javascript Flow Processor, for example, which directly interacts with a database will have to react to a rollback signal:

Rollback example in Javascript

   function onRollback() {
    if (connection) {
        try {
            connection.rollbackTransaction();
            connection.closeConnection();
        } catch (err) {
        } finally {
            connection = null;
        }
    }
}

Retry Stream

Don't simply give up. Try to process the whole stream again. This option allows you to define how often and in what intervals the retries should be performed.

Stream Retry Settings

• Max. Retries: The number of retries which should be performed. For example "5".
• Min. Backoff [ms]: Wait at least x milliseconds between each retry. For example "12000" (12 seconds).
• Max. Backoff [ms]: Wait at max x milliseconds between each retry. For example "150000" (150 seconds).

Based on these parameters, the system will try to balance the defined number of retries within the time boundaries of min. backoff and max. backoff.
Taken the example numbers from above, the five retries would happen in this timespan:

---

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .