AWS S3

The AWS S3 component allows Connect to store and retrieve objects (for example, files, such as documents or images) from Amazon AWS S3, from any place on the Internet.

General information

Purpose

The component provides ability to connect to Amazon Simple Storage Service (Amazon S3) object storage service.

Following actions are available:

Write file
Read file
Get filenames
Delete file
Stream to CSV
Rename file

Description

This is the component for working with AWS S3 object storage service on Squiz Connect platform.

SDK version

The component is based on AWS S3 SDK version 2.683.0.

Requirements

Environment variables

Name Mandatory Description Values

Name	Mandatory	Description	Values
`ATTACHMENT_MAX_SIZE`	false	For Connect attachments configuration. Maximum possible attachment size in bytes. By default set to 1000000 and according to platform limitations CAN’T be bigger than that.	Up to `1000000` bytes
`ACCESS_KEY_ID`	false	For integration-tests is required to specify this variable
`ACCESS_KEY_SECRET`	false	For integration-tests is required to specify this variable
`REGION`	false	For integration-tests is required to specify this variable

ATTACHMENT_MAX_SIZE

false

For Connect attachments configuration. Maximum possible attachment size in bytes.

By default set to 1000000 and according to platform limitations CAN’T be bigger than that.

Up to 1000000 bytes

ACCESS_KEY_ID

false

For integration-tests is required to specify this variable

ACCESS_KEY_SECRET

false

For integration-tests is required to specify this variable

REGION

false

For integration-tests is required to specify this variable

NOTE:The component LOG_LEVEL environment variable has been deprecated. Log levels are now controlled per each step of the flow.

Credentials

Access keys consist of three parts:

An access key ID.
A secret access key.
A region.

Similar to a user name and password, you must use both the access key ID and secret access key together to authenticate your requests.

According to AWS documentation for buckets created in Regions launched after March 20, 2019 Region is required for AWS credential.

Access Key Id

An access key ID (for example, AKIAIOSFODNN7EXAMPLE).

Secret Access Key

A secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).

Region

Example: ca-central-1.

Triggers

Get New and Updated S3 Objects

Triggers to get all new and updated s3 objects since last polling.

List of Expected Config fields

Bucket Name and folder

The name of the S3 bucket from which to read files.

Emit Behaviour

Options are:

Emit Individually: Emits each object in separate message. This is the default.
Fetch All: Emits all objects in one message

Start Time

Start datetime of polling. Default min date:-271821-04-20T00:00:00.000Z

End Time*

End datetime of polling.

Default max date

+275760-09-13T00:00:00.000Z

Enable File Attachments

If selected, the contents of the file will be exported in addition to the file metadata.

Output metadata:

{
  "type": "object",
  "properties": {
    "Key": {
      "type": "string",
      "required": true
    },
    "LastModified": {
      "type": "string",
      "required": true
    },
    "ETag": {
      "type": "string",
      "required": true
    },
    "Size": {
      "type": "number",
      "required": true
    },
    "StorageClass": {
      "type": "string",
      "required": true
    },
    "Owner": {
      "type": "object",
      "properties": {
        "ID": {
          "type": "string",
          "required": true
        }
      }
    }
  }
}

Write file to S3 from a provided attachment

Given a filename and a URL to an attachment stored in the platform, transfers the contents of the attachment to AWS S3.

The component returns a summary of the written file.

AWS S3 always overwrites the contents of the file if it already exists.

Expected input metadata

bucketName: The name of S3 bucket to write the file to; Sufficient write permission is required.
fileName: The name of file/S3 object to write. Use / characters in the filename to create folders.
attachmentUrl: URL to the attachment stored in the platform. The contents of this attachment will be written to S3 without any transformation;

Limitations

It is not possible to set the file/object Metadata in S3.
Files/objects can not be so large that they can not fit in the memory of the component’s docker container.
Files/objects can not be more that 5 GB in size.
It is not possible to set the AWS S3 Storage Class for written files/objects. They will always be written with the standard storage class.
It is not possible to set file/object tags.
It is not possible to compress objects/files (with zip, gzip, or other archive types).
It is not possible to encrypt object/files.

Read file

Reads a file from an S3 bucket. This action reads file from S3 bucket by provided name.

The result is storing in the output body (for JSON or XML) or in the output attachment (for other types). File type resolves by it’s extension.

The name of attachment would be same to filename.

List of expected config fields

Default Bucket Name and folder: The name of S3 bucket to read file from (by default, if bucketName is not provided in metadata).

Expected input metadata

filename: The name of the S3 bucket file to read.
bucketName: The name of S3 bucket from which to read the file (will replace Default bucket name and folder if provided). The field is optional.

Input metadata:

{
  "type": "object",
  "properties": {
    "filename": {
      "type": "string",
      "required": true
    },
    "bucketName": {
      "type": "string",
      "required": false
    }
  }
}

Expected output metadata

{
  "type": "object",
  "properties": {
    "filename": {
      "type": "string",
      "required": true
    }
  }
}

Get filenames

Emit all the filenames, individually, from the S3 bucket.

If you provide a bucket and folder by name (for example; eio-dev/inbound), the name of the root folder (inbound/) will be returned along with all the file names.

List of Expected Config fields

Default Bucket Name and folder: The name of the S3 bucket from which to read files (by default, if bucketName is not provided in metadata).

Expected input metadata

bucketName: The name of the S3 bucket from which to write files (this will replace Default bucket name and folder if provided). This field is optional.

Input metadata:

{
  "type": "object",
  "properties": {
    "bucketName": {
      "type": "string",
      "required": false
    }
  }
}

Expected output metadata

{
  "type": "object",
  "properties": {
    "ETag": {
      "type": "string",
      "required": true
    },
    "Location": {
      "type": "string",
      "required": false
    },
    "Key": {
      "type": "string",
      "required": true
    },
    "Bucket": {
      "type": "string",
      "required": true
    }
  }
}

Known limitations

It is possible to retrieve a maximum of 1000 file names.

Delete file

This action deletes a file from S3 bucket.

The action will emit single filename of removed file.

List of Expected Config fields

Default Bucket Name and folder: The name of the S3 bucket from which to delete the file (by default, if bucketName is not provided).

Expected input metadata

filename: The name of the file to delete from the S3 bucket.
bucketName: The name of S3 bucket and folder from which to delete file. This will replace Default bucket name and folder if provided. This field is optional).

Input metadata:

{
  "type": "object",
  "properties": {
    "filename": {
      "type": "string",
      "required": true
    },
    "bucketName": {
      "type": "string",
      "required": false
    }
  }
}

Expected output metadata

{
  "type": "object",
  "properties": {
    "filename": {
      "type": "string",
      "required": true
    }
  }
}

Rename file

This action renames a file in an S3 bucket and folder.

The action will emit properties of renamed file.

Expected input metadata

bucketName: The name of the S3 bucket where the file is located.
folder: The name of folder where file is located. This is optional.
oldFileName: The original name of file that to be renamed.
newFileName: The new name of the file.

{
  "type": "object",
  "properties": {
    "bucketName": {
      "title":"Bucket Name and folder",
      "type": "string",
      "required": true
    },
    "folder": {
      "type": "string",
      "required": false
    },
    "oldFileName": {
      "type": "string",
      "required": true
    },
    "newFileName": {
      "type": "string",
      "required": true
    }
  }
}

Expected output metadata

{
  "type": "object",
  "properties": {
    "Key": {
      "type": "string",
      "required": true
    },
    "LastModified": {
      "type": "string",
      "required": true
    },
    "ETag": {
      "type": "string",
      "required": true
    },
    "Size": {
      "type": "number",
      "required": true
    },
    "StorageClass": {
      "type": "string",
      "required": true
    },
    "Owner": {
      "type": "object",
      "required": true,
      "properties": {
        "ID": {
          "type": "string",
          "required": true
        }
      }
    }
  }
}

Known Limitations

The maximum size for an attachment is 10 MB.
The attachment mechanism does not work with the deprecated Local agent component.

Help Center

Menu

AWS S3

General information

Purpose

Description

SDK version

Requirements

Environment variables

Credentials

Access Key Id

Secret Access Key

Region

Triggers

Get New and Updated S3 Objects

List of Expected Config fields

Write file to S3 from a provided attachment

Expected input metadata

Limitations

Read file

List of expected config fields

Expected input metadata

Expected output metadata

Get filenames

List of Expected Config fields

Expected input metadata

Expected output metadata

Known limitations

Delete file

List of Expected Config fields

Expected input metadata

Expected output metadata

Rename file

Expected input metadata

Expected output metadata

Known Limitations