Batch component

Overview

The batch component allows Connect messages to be aggregated into simple collections for processing in batches.

Integrators would need to split an integration logic into two parts.

The first should use the action Add message to batch, which starts to collect messages.

The second should start with the Get ready batches trigger.

Flow parts

It uses the Maester service as a package repository. The legacy component versions support MongoDB (4.0 and higher stable releases) as a batches repository.

The component does not support external MongoDB as a data store since version 2.0.0 (disabled support for credentials in the component).

Requirements

Maester microservice, the official Connect object-storage client, is enabled by default as the Batch component relies on it.

Triggers

Get ready batches

Emits all batches that are ready to be processed.

Get ready batches

Input fields description

Input field Required Description Example

Max time in milliseconds before batch is ready.

No.

Timeout in milliseconds after batch creation, which makes batch ready for processing. Defaults to 60000 ms.

3000

Max records number in batch.

No.

Maximum count of items in batch, which makes batch ready for processing. Defaults to 100.

10

Max size of batch in bytes.

No.

Maximum bytes size of items in batch which makes batch ready for processing 1000000(1 MB).

500000

Correlation ID.

Yes.

Correlation ID between action and trigger. Correlation ID helps identify what collection of batches should be used.

Flow1

Do not delete batch after retrieval.

No.

If selected, the batch will not be deleted after emit.

False

Use the same configuration in action for correct batch processing.

Actions

Add message to batch

Stores a message in a batch and emits a created/updated batch with a processed item only.

Add message to batch

Input fields description

Input field Required Description Example

Max time in milliseconds before batch is ready.

No.

Timeout in milliseconds after batch creation, which makes batch ready for processing. Defaults to 60000 ms.

3000

Max records number in batch.

No.

Maximum count of items in batch, which makes batch ready for processing. Defaults to 100.

10

Max size of batch in bytes.

No.

Maximum bytes size of items in batch which makes batch ready for processing 1000000(1 MB).

500000

Correlation ID.

Yes.

Correlation ID between action and trigger. Correlation ID helps identify what collection of batches should be used.

Flow1

Use the same configuration in trigger for correct batch processing.

Additional info

  1. Objects created in Maester will have the default TTL. Please contact support if you require the value of this property.

  2. Please ensure that the value of Max time in ms before batch is ready is less than Maester’s default TTL. Otherwise, the object in Maester will be deleted before the batch is ready.

Known limitations

  1. The component uses Maester microservice, so the component is affected by its limitations.

  2. Emit rate logic is not implemented.

  3. The library does not guarantee a sequence of batch items.

© 2015- Squiz Pty Ltd