Output Stream

Purpose

The Output Stream Asset acts as a final endpoint of a Workflow. You end a Workflow with an Output Stream Asset if you want to output to one of the Sinks mentioned in the Prerequisites list below.

This Asset is used within a Workflow definition.

Prerequisites

You need:

A defined Format

• Formats

A Stream Sink:

• Sink File System
• Sink FTP
• Sink OneDrive
• Sink S3
• Sink Sharepoint
• Sink SMB
• Sink WebDav
• Sink WebSocket

Name & Description

Name : Name of the Asset. Spaces are not allowed in the name.

Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

Input Ports

This Processor can only have one Input Port from which it receives data to process.

A port can have a name and description. A Name is mandatory.

You cannot delete the only Input Port.

Format

Data output by this Asset must be written in a specific format.

If you have defined such a format, then you can select it from the list of available formats by clicking on the button next to the field.

Sink

You need to assign a Sink which defines the physical output of the data.

Select the Sink by clicking on the button next to the field (1).

Other Settings

Compression Mode

In case your outgoing stream data shall be written in compressed format, you can define the algorithm to be used for decompression here. For uncompressed data, please leave the setting on None.

Stream of individual messages

Check this box, if the data format does only describe a single message, which is each message written should be considered a complete output and final output.

Explainer: Any data format described and selected in Format above, can be simply, or complex. A complex format, for example, could require the data to be sent to be only complete, if it contains a header record, one to many detail records, and a trailer record. To the contrary, a simple format could define only the structure of a single message.

If we assume that we have selected a complex format like above as the output, then the system will observe and enforce, that the written output does comply with the format between starting and stopping a stream. I.e. it must contain a header, one or many details, and a trailer record (example). If this is not the case, then an error will be generated.

If we on the other hand have a simple format which in structure only describes a single message structure, then you may want the system to write individual messages as final transactions for every single message. In this case, and only in this case, you can check this box. The default is off. If you are unsure about this, then just leave the default.

Failure Handling

Processing within an Output Processor like this one can fail for various reasons. In this section you can define how the system should behave in case of such problems.

Failure Types

Two types of failures are observable:

#	Failure observables / Reaction	Ignore	Retry Event/Message	Retry Stream	Rollback Stream
1	Sink failure handling A problem occurred during the output of a stream			✔	✔
2	Event write failure handling A problem occurred during writing of a message.	✔	✔	✔	✔

Failure Type Reactions

Ignore

Don't do anything.

Rollback Stream

Rollback the complete stream. In the case of batch/file processing for example the complete file (which represents the stream) will be rolled back and put into error.

warning

A rollback signal will be issued to all participating Workflow Processors. Each Processor needs to ensure itself how to deal with a rollback. A Javascript Flow Processor, for example, which directly interacts with a database will have to react to a rollback signal:

Rollback example in Javascript

   function onRollback() {
    if (connection) {
        try {
            connection.rollbackTransaction();
            connection.closeConnection();
        } catch (err) {
        } finally {
            connection = null;
        }
    }
}

Retry Stream

Don't simply give up. Try to process the whole stream again. This option allows you to define how often and in what intervals the retries should be performed.

Stream Retry Settings

• Max. Retries: The number of retries which should be performed. For example "5".
• Min. Backoff [ms]: Wait at least x milliseconds between each retry. For example "12000" (12 seconds).
• Max. Backoff [ms]: Wait at max x milliseconds between each retry. For example "150000" (150 seconds).

Based on these parameters, the system will try to balance the defined number of retries within the time boundaries of min. backoff and max. backoff.
Taken the example numbers from above, the five retries would happen in this timespan:

Retry Event/Message

Pick this reaction if you want to retry processing the current message. As is the case with the Retry Stream reaction you can define how often and in what intervals the retries should be performed.

The settings are the same as with the Retry Stream reaction. So please refer to this. There is one additional setting, however, which is When final retry failed.

You here have the option to decide what to do if the message cannot be processed, even after the final retry:

• Ignore: Don't do anything.

• Rollback Stream: Fallback to rolling back the whole stream.

• Retry Stream: Retry the whole stream once again. If you pick this option then you can again define all relevant Retry Stream parameters.

  

---

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .