Help Center

Menu

Multi-collection

Multi-collection requests are requests that can get documents from multiple collections in a single API request.

The endpoint exists at https://YOUR-DATA-SERVICE.datastore.squiz.cloud.

It is a unique endpoint available automatically and does not need to be configured in your blueprint. The API for this endpoint is very similar to the collection schema applicable for regular collection requests; however, a significant difference is that you must perform a POST request to get documents. You also will not be able to create documents under this endpoint, only request them.

The endpoint expects an application/json content type.

Authorization

The https://YOUR-DATA-SERVICE.datastore.squiz.cloud endpoint is public; however, for a request to be satisfied, the requester must satisfy the collection ACL statement for all requested collections. Collection ACL statements cannot contain parent.

Additionally, all requested collections must have identical document-level ACL statements, and only documents where that ACL statement is satisfied will be returned.

Like collection and document requests, you should provide a JWT with a payload to satisfy any ACL statement trusted data requirements.

Requesting collections

To specify which collections you wish to request documents from, the JSON object in the body of your application/json request should contain a collections object key. Its value should be an array of collections starting with a forward slash that maps to a root level collection or sub-collection in your Data service.

This example uses CURL to request data from the /students and /staff collection:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ "collections": ["/students", “/staff”] }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

The response of a multi-collection request returns a JSON object with the resultant documents data in the object variable "data" array. Injected into every document result is the key "$documentPath" to identify to which collection each document belongs.

Following on from our student/staff multi-collection example, the result is as follows:

{
    "data":[
        {
            "id":"23",
            "Name": "Madison",
            “$documentPath”:”/student/23”
        },
        {
            "id":"24",
            "Name": "Avery",
            “$documentPath”:”/staff/24”
        },
    ]
}

A maximum of 100 collections can be specified in the collections array.

Sub-collections and wildcards

When targeting sub-collections, you may wish to target all sub-collections of a collection. Since the sub-collection endpoint is partially made from the `documentid`s in the collection, it can be difficult to know those ids and impossible to target them all if there are more than 100.

This is where the collection wildcard operator /:{*} can be used. This can be used to replace the documentid portion of a sub-collection endpoint path.

For example, if we wanted to make a multi-collection request to all student and staff absent days collection, we could do the following:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ "collections": ["/students/:{*}/absent-days", “/staff/:{*}/absent-days”] }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

Filtering multi-collection requests

Just like collection requests, multi-collection responses can be filtered. This is done by providing the filter string that is usually in the query string into the request’s body under the filters key.

Read the Filtering documentation for more information about the format of the filters string.

An example of filtering students and staff by name is below:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ \
    "collections": ["/students", “/staff”], \
    “filters”: "document.name===Madison”, \
        }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

Sorting multi-collection requests

Just like collection requests, multi-collection responses can be sorted. The properties of how documents are sorted that apply to collection requests also apply to multi-collection requests Sorting is done by providing the sort string that is usually in the query string into the request’s body under the sort key.

Read the Sorting documentation for more information about the format of the sort string.

An example of sorting students and staff by name is below:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ \
    "collections": ["/students", “/staff”], \
    “sort”: "document.name”, \
        }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

Paginating multi-collection requests

Just like collection requests, multi-collection responses can be paginated.

Pagination is done by providing the page object that is usually in the query string into the request’s body under the page key.

Read the Pagination documentation for more information about the format of the page.

An example of paginating through students and staff:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ \
    "collections": ["/students", “/staff”], \
    “page”: { \
        "size": 1, \
        "v": 2, \
    } \
        }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

Like collection requests, the response contains the pagination information.

{
    "data":[
        {
            "id":"23",
            "Name": "Madison",
        },
        {
            "id":"24",
            "Name": "Avery",
        },
    ],
    "page": {
        "v": 2
        "size": 1,
        "before": null,
        "after": "/students/23"
    }
}

Aggregating multi-collection requests

Just like collection requests, multi-collection responses can be aggregated. Aggregation is done by providing the aggregation object in the query string into the request’s body under the agg key.

Read the Aggregation documentation for more information about the format of the agg object.

An example of counting all students and staff is below:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ \
    "collections": ["/students", “/staff”], \
    "agg": { \
        "func": “count”, \
        “prop”: “document.id”,\
    } \
        }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

Multi-collection aggregation responses are identical to collection requests.

Extends tree

The extends tree is relationship information that can be loaded for every document showing in the response of the multi-collection request. It will include the path of every document a document extends and then every document that extends any of those documents. In short, it shows the full extending relationship tree of every document in the response. As this is a lot of data, the extends tree can only be loaded on multi-collection requests that are limited to a page size of 100 documents or less. Additionally, when requesting the extends tree, the document level ACL statements cannot contain document.

The Personalization SDK uses the extends tree for finding the most relevant document to a user in personalization use cases.

When traversing the extends tree, all collections and documents are traversed; however, before becoming part of the result, Datastore will remove any document that:

  • Exists in a collection that is not part of the multi-collection request.

  • Exists in a collection the requester does not have collection ACL access to.

  • The requestor does not have document access to.

Requesting the extends tree is done by providing the includes key in the request’s body. That key should hold an array with an extendsTree element within.

In the following example, we will be loading assignments. The base information for an assignment is contained in the /assignments document; however, a student may have a document that extends the assignment to inherit most values and overriding others. A value inherited may be the title of the assignment, but as this student has received an extension, the assignment’s due date is overwritten. The extends relationship is controlled through the overrides property in this example.

To load this assignment information, the following request can be made:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    --silent \
    -d '{ \
        "collections": ["/assignments", “/students/:{*}/assignments”], \
        “page”: { \
            "size": 20, \
            "v": 2,\
        } \
        “includes”: [“extendsTree"] \
    }' \
    -fS \ https://YOUR-DATA-SERVICE.datastore.squiz.cloud/__resources/collections

The response to this request might look like this:

{
    "data":[
        {
            "id":"1",
            "title": "Crossing Boundaries Essay",
            “dueDate”: “2022-03-20 17:00:00+11”,
            "overrides": "",
            “$documentPath”: “/assignments/1”
        },
        {
            "id":"1",
            "title": "Crossing Boundaries Essay",
            “dueDate”: “2022-03-27 17:00:00+11”,
            "overrides": "/assignments/1",
            “$documentPath”: “/students/23/assignments/1”
        },
    ],
    "page": {
        "v": 2
        "size": 1,
        "before": null,
        "after": "/students/23"
    },
    "extendsTree": {
        “/assignments/1”: [“/students/1/assignments/1”],
        “/students/1/assignments/1”: [“/assignments/23”],
    },
}

Response codes

Collection requests return the following HTTP response codes:

HTTP Method HTTP code Notes

GET

200 OK

Returned when authorized to get documents for the endpoint.
Code Description JSON

400 Bad Request

Returned when in the POST body the collections object value is not an array.

 
{
"title": "Invalid collections request",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Collections list must be an array"
    }
]
}

400 Bad Request

Returned when in the POST body the collections object value is an empty array.

 
{
"title": "Empty \"collections\" is not allowed",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Collections list cannot be empty"
    }
]
}

400 Bad Request

Returned when in the POST body the collections object value array has more than 100 items.

 
{
"title": "More than 100 "collections" passed",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Cannot request from more than 100 \"collections\""
    }
]
}

400 Bad Request

Returned when in the POST body the collections object value array has one or more collections that are not correctly formatted. Collection paths must start with "/" and contain no path traversal, and consist of letters, numbers or the following characters "-", "_", ".", "%", "~".

 
{
"title": "Invalid collection path",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Collection paths must start and not end with a \"/\", contain no path traversal and consist of letters, numbers or the following characters \"-\", \"_\", \".\", \"%\", \"~\""
    }
]
}

400 Bad Request

Returned when one or more collections being requested from have a document level ACL rule that differs from another collection’s document level ACL rule in the same request.

 
{
"title": "Invalid collections request",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Document ACL statements containing \"document.\" must exactly match across all collections"
    }
]
}

400 Bad Request

Returned when a collection being requested is not found within the Data Service.

 
{
"title": "Invalid collections request",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Collections not found: %PROBLEM-COLLECTION%"
    }
]
}

400 Bad Request

Returned when a collection being requested has a document level ACL statement that contains "parent.".

 
{
"title": "Invalid collections request",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "Document ACL statements containing \" parent.\" are not allowed for wildcard requests."
    }
]
}

400 Bad Request

Returned when a collection being requested has a document level ACL statement that contains "parent.".

 
{
"title": "Invalid sort",
"status ": "400",
"invalid-params": [
    {
        "name": "sort",
        "reason": "Must be valid document property: %PROBLEM-PROPERTY%"
    }
]
}

400 Bad Request

Returned when in the POST body the func element is not found in the agg object.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Missing aggregation function."
    }
]
}

400 Bad Request

Returned when aggregating multiple collections and the defined aggregation function is not supported.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Aggregation function is not supported."
    }
]
}

400 Bad Request

Returned when in the POST body the prop element is not found in the agg object.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Missing aggregation property."
    }
]
}

400 Bad Request

Returned when aggregating multiple collections and the defined aggregation property that does not exist.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Must be valid document property: %PROBLEM-PROPERTY%"
    }
]
}

400 Bad Request

Returned when aggregating multiple collections and attempting to group by a property that does not exist.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Must be valid document property: %PROBLEM-PROPERTY%"
    }
]
}

400 Bad Request

Returned when aggregating multiple collections and attempting to group by an internal property like $documentid.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Can not group by internal property: %PROBLEM-PROPERTY%"
    }
]
}

400 Bad Request

Returned when aggregating multiple collections and attempting to group by a property of type array.

 
{
"title": "Invalid aggregation",
"status ": "400",
"invalid-params": [
    {
        "name": "aggregation",
        "reason": "Can not group by array property: %PROBLEM-PROPERTY%"
    }
]
}

400 Bad Request

Returned when requesting the extends tree, and a collection with document level ACL statement is requested.

 
{
"title": "Invalid extendsTree request",
"status ": "400",
"invalid-params": [
    {
        "name": "resources",
        "reason": "extendsTree cannot be requested when document level ACL statements are in use"
    }
]
}

400 Bad Request

Returned when requesting the extends tree and not paginating or having a page size greater than 100.

 
{
"title": "Invalid extends request",
"status ": "400",
"invalid-params": [
    {
        "name": "extendsTree",
        "reason": "The page size must be less than or equal to 100"
    }
]
}

403 Forbidden

Returned when the request satisfied no collection’s collection level ACL rule.

 
{
"title": "Forbidden",
"status": "403"
}