Document and collection organisation

One of Datastore’s key features is the ability it provides you with to create your own structures (defined by the API specifications and data models that make up your blueprints) to store data, along with the methods used to query data in these structures.

Datastore stores your actual data in objects called documents, where groups of related documents are organised into collections.

These documents and collections are analogous to files and directories (respectively) on a file system.

Datastore implements two API standards:

  • OpenAPI - an initiative to standardise how RESTful APIs are described. This standard is used to create your API specification that defines the API endpoints and the HTTP methods they support. These files are typically defined in YAML format.

  • JSON Schema - as documented on this site, this is a vocabulary for annotating and validating JSON documents. This standard is used to define the property data (or data models) that each endpoint accepts through requests, as well as their responses. Refer to the json.org web site for a brief summary of this syntax.

How documents and collections are organised

From an organisational perspective:

  • A document must always be placed inside a collection.

  • A collection must be created at the root level, from which multiple documents can be created and accessed. (Documents cannot be created at the root level.)

  • A document may contain a collection, which is known as a sub-collection since it is not a collection at the root level.

  • Multiple collections can be created at the root level.

For example, a collection called comments could be used to store comments submitted through an application, while each comment could have a sub-collection called replies used to store the replies to each comment.

This creates a storage structure like the following:

Example storage structure for comments and their respective replies
/comments
/comments/comment#1
/comments/comment#1/replies
/comments/comment#1/replies/reply#1
/comments/comment#1/replies/reply#2
/comments/comment#2
/comments/comment#2/replies
/comments/comment#2/replies/reply#1

This alternating collection > document > collection > etc. data structure pattern is used to construct the API endpoint URLs within Datastore, in the form /collection/document/collection/…​, where either a collection or unique document identifier (ID) name is used at each level of the URL for a given API call.

Within a URL, the collection (for comments and their replies) can retain its user-friendly name as this name only needs to be unique at a given level of the URL.

The documents (i.e. the actual comments and replies themselves) use a unique ID that Datastore assigns to them. While this document name only needs to be unique within a given collection, Datastore allocates a version 4 universally unique identifier (UUID) for each document name. A UUID generated by Datastore is a 36 character string separated into 5 groups using hyphens, for example: 0cad4697-8422-47c0-9d4b-5f16d7f5baf3.

Following on from the comment and reply example above, the following URL paths can be generated:

Example of comment and reply endpoint URL paths (with truncated UUIDs)
/comments
/comments/9ff23854-af...
/comments/9ff23854-af.../replies
/comments/9ff23854-af.../replies/ec0a929b-0c...
/comments/9ff23854-af.../replies/b7a6e61e-ab...
/comments/0dc67f6d-3b...
/comments/0dc67f6d-3b.../replies
/comments/0dc67f6d-3b.../replies/0cad4697-84...
Since a document name only needs to be unique within a given collection, your application can assign a unique identifier to a document instead of one allocated by Datastore. This allows you to create friendlier URLs with more memorable document identifiers.

© 2015- Squiz Pty Ltd