Queue Service

Purpose

Define a service for the usage of a file-based queue.

Prerequisites

None

Configuration

Name & Description

• Name : Name of the Asset. Spaces are not allowed in the name.

• Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

Required roles

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

Queue Configuration

• Queue Type: Select the type of queue from the drop-down box. At this stage File Queue is the only available type. It is working by FIFO (First-In-First-Out) principle. In theory, Queue Type can be other storages like RDBMS, Kafka, et al. as well. Please don't hesitate to contact us at support@layline.io in case you are in need of other Queue Type.

• Queue Directory (macro supported): the directory used to store the file-based queue data.

• Use a service instance specific subdirectory: activating this checkbox, layline.io will create a unique named subdirectory per service instance (recommended)

Example: Using the Queue Service

File-based queues need to be considered whenever the processing requires a majority or even all messages from an input to be checked and kept before they can be sent resp. written to an output source. An approach to achieve this could be to keep the data in main memory, while this often creates an issue with available capacities. Hence, a temporary file-based storage helps to support these use cases. Typical scenarios for making use of file-based queues are:

• Assembly of multiple messages to form an output message, spanning one dedicated file
• Header messages populated with accumulated detail message information, for instance TAP-File creation

In our example we have a simple Workflow which reads a file with input data (1), then in the next step (2) intermediately writes that data into a queue provided by the Queue Service and ensures to read that data after all messages have been processed. Finally, the data is sent and written to an output file. There is no other purpose in this Workflow than to demonstrate how to use the Service.

In the middle of the Workflow we find a JavaScript Processor by the name of "Queued-Data". This Processor writes and reads information to / from the configured Queue Service.

What needs to be done to implement this?

Link Queued-Data Processor to Queue Service

To use the Queue Service in the JavaScript Processor, we first have to assign the Service within the JavaScript Processor like so:

• Physical Service: The Queue Service which has been configured.

• Logical Service Name: The name by which we want to use the Service within JavaScript. This could be the exact same name as the Service or a name which you can choose. Must not include whitespaces.

Access the Service from within JavaScript

Before we start looking into dedicated JavaScript sample code using the Queue Service, it is important to understand that layline.io processing is based on "dynamic push/pull mode". More information can be found here.

Using a Queue Service in a JavaScript Processor means that this Processor becomes a "producer" of additional messages requiring the forcing of the pull-mode to be used. Hence, the onPullMessage usage becomes a must-have.

Let’s finally use the service within JavaScript:

Writing to and Reading from Queue endpoint

const OUTPUT_PORT = processor.getOutputPort('Output-1');
let queue = null;
let trailerReceived = false;
let totalRecordsInFile = 0;

export function onStreamStart() {
    // Invoke Service
    // using a Queue-Service for writing data into it requires to "open it" via the method openConnection
    queue = services.QueueService.openConnection();
    trailerReceived = false;
    totalRecordsInFile = 0;
}

export function onStreamEnd() {
    if (queue) {
        // any used Queue-Service should be closed after usage 
        queue.closeConnection();
        queue = null;
    }
}

export function onMessage() {
    if (message.data.SMPL_IN.RECORD_TYPE == 'H') {
        // write message to Queue-Service
        queue.WriteMessage(message);
    } else if (message.data.SMPL_IN.RECORD_TYPE == 'D') {
        onDetail(message);
    } else if (message.data.SMPL_IN.RECORD_TYPE == 'T') {
        onTrailer(message)
    }
}

/**
 * Handle detail
 */
function onDetail (detail) {
    totalRecordsInFile++;
    // write message to Queue-Service
    queue.WriteMessage(detail);
}

/**
 * Handle the Trailer message
 * indicates that full file has been processed
 */
function onTrailer(trailer) {
    trailerReceived = true;
    // write message to Queue-Service
    queue.WriteMessage(trailer);
}

/**
 * Handle the request to pull a message
 */
export function onPullMessage() {
    if (trailerReceived && queue) {
        let message = null;
        do {
            // read message from Queue-Service
            message = queue.ReadMessage();
            if (message) {
                if (message.data.SMPL_IN.RECORD_TYPE == 'H') {
                    onProcessAndEmitHeader(message)
                } else if (message.data.SMPL_IN.RECORD_TYPE == 'D') {
                    onProcessAndEmitDetail(message)
                } else if (message.data.SMPL_IN.RECORD_TYPE == 'T') {
                    onProcessAndEmitTrailer(message);
                }
            }             
        } while (message);

        if (message == null) {
            // any used Queue-Service should be closed after usage 
            queue.closeConnection();
            queue = null;
        }
    }
}

Service Testing

layline.io provides a test facility for testing your Services before you deploy them. In this way, you save time and effort by testing your Services without having to deploy and activate a whole Project with Workflows.

Once you have configured your Service(s), you can test them: Within your Asset Configuration tab (1), switch to the Test tab (2) to test your Service.

Test Facility Toolbar

The toolbar provides the following options:

The Testing tab provides two major views:

1. Testcase configuration: This is where you define the testcases to be executed.
2. Testcase execution: This is where you can execute the testcases and see the results.

You switch between these two views by clicking on the leftmost icon in the toolbar (1).

Let's start with the Testcase configuration view.

Testcase Configuration

The concept of the Testing is to define a set of Testcases which can be executed in a batch or individually. For this purpose, you can define multiple Testcases and configure them individually. I.e. each Testcase groups a number of indidivual tests which can be executed individually or in a batch.

Adding a Testcase

Click Add Testcase in the toolbar to add a new testcase:

A new Testcase is added. It is automatically named New<Service Asset Name>Test (3) and added to the list of Testcases (2).

• Service test name (3): You can change the name of the Testcase here.
• Service test description (4): You can add a description to the Testcase here.

Test Case Setup

Basics

In this section you define the individual tests to be executed for this Testcase.

To start, click # END in the toolbar:

A new test is added to the list of tests (1), and the test is opened for configuration (2).

Next we fill in the details:

• Test name (3): You can change the name of the Test here.

• Test description (4): You can add a description to the Test here.

• Service function to test (5): Select the Service function to test here.

  This list contains all Service functions which are defined in the Service Asset. Pick the one you want to test.

  

  Once a Service function is selected, the system will automatically create a skeleton to fill in the respective parameters for the selected Service function.

  

Service Function Input Parameters

• Service Function Input Parameters (6): Fill in the respective parameters for the selected Service function.

  In our example we have a function GetAlertsForSite which takes two parameters baseurl and riskId. If we click on Add member in the skeleton table the system will allow you to select the respective parameter from the list of available parameters:

  

  Once you have selected the parameter, the system will automatically add the respective parameter name. You then add the respective value for the parameter:

  

Service Function Evaluation Parameters

To automatically evaluate the result, you can add a script which analyzes the results.

Testcase Execution

Once you have configured your Testcases, you can execute them.

There are two ways on how to trigger execution:

• Option 1: Select Run selected test in the toolbar (1) to execute the currently selected Testcase.

  

  Executing a test this way will switch the tab to the Testcase execution view, execute the test and show the results.

• Option 2: Switch to the Testcase execution view by clicking on the leftmost icon in the toolbar (1) select the test to execute, and then hit the play button next to the test.

  

Each option will take us to the Testcase execution view:

In this view you can find the Testcase (1) and the Tests (2) we have created. If we had created additional tests for this Testcase, they would be listed here as well.

Question marks indicate that the test has not yet been executed.

We can now either execute all tests, or run them individually:

• Run all Tests (1): Click this button to execute all tests.

• Run Testcase (2): Click this button to a Testcase with all its underlying individual tests.

  

• Run individual Test (3): Click this button next to a test to execute this individual test.

  

Once a test has been executed, the question mark will be replaced by a green check mark or a red cross depending on whether the test was successful or not.

The right hand-panel will show the results of the test execution respectively:

In case of errors, the system will show the error message for further investigation.

---

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .