Help Center

Menu

Tutorial 8 - Filtering documents

Your event app currently allows you to add new user accounts, as well as create, edit, delete and sort event documents as a specific user, which only that user can access through the API.

When a collection of event documents is retrieved, all documents in the collection are returned.

In this tutorial, you will extend the event app functionality (utilising the JS SDK) to implement document filtering features in collection retrieval.

This tutorial takes around 20-30 minutes to complete.

Filtering implementation approach

Like sorting documents (in the previous tutorial), apps that perform client-side document filtering work well with small collections of documents, but rapidly encounter performance problems as the number of documents per collection increases, along with the number of documents being filtered out.

Therefore, Datastore natively provides server-side document filtering capabilities, which allows collections of documents to be filtered rapidly by properties defined in the document’s JSON schema files of your API specification, before the documents are sent back to your client app in the response.

Copy the 'tutorial 8' files over to your event-app directory

Copy the next level of your app’s functionality (which handles sorting) from the source tutorial repository over to your event-app directory.

  1. From the event-app directory, copy the content of the step9 directory to the event-app directory:

    $ cp -r ../tutorial/step9/*.* .

  2. Refresh or re-open the initial index.html file in the event-app directory.

Test the updated app

  1. Ensure you have logged in with an account you created in a previous tutorial, and notice the Filter Events section along the left of the event app’s page.

  2. Ensure you have created at least three events, each with different Category, Cost and Do you have a ticket? values.

  3. In the Filter Events section, choose a Category field’s value (ensuring you 'clear' all other fields' values first by ensuring their 'empty' values have been chosen from their lists), and click the Update list button. You should notice that only events whose Category field matches your chosen value appears in the events list.

  4. Do the same for the Category field’s other values, and notice how the events list is filtered.

  5. Repeat the previous two steps for the Price range and Do you have a ticket? fields.

  6. Try applying multiple field values together, as the filter values are additive.

The JS SDK where(), and(), and or() filtering methods

The JS SDK includes where(), and(), and or() filtering methods (which operate on a collection() method that returns a collection request object), to filter documents in the collection of the Datastore service’s response, based on the values specified in these filtering methods' arguments.

The where( …​ , …​ , …​ ), and( …​ , …​ , …​ ), and or( …​ , …​ , …​ ) methods each accept the following three arguments (listed in their order of usage), where all arguments are mandatory:

  • property - the name of the property (as a string value) as defined in the document’s JSON schema file,

  • comparison - a string value of one of the following valid comparison operators:

    • === - equals,

    • !== - is not equal to,

    • > - greater than,

    • >= - greater than or equal to,

    • < - less than, and

    • <= - less than or equal to.

  • value - the value to which the document’s property value (in the Datastore service) is being compared. If the document’s property value and this argument’s value value evaluates to true (based on the comparison's operator value), then the document is included in the collection from the Datastore service’s response when the get() method is called.

Using the filtering methods

Following a collection() method, the first filtering method must be where(), which can then followed by and() or or() methods. The and() and or() methods are used to handle logical groupings of each comparison, where and() has a higher priority than or().

For example, the following JS SDK call:

datastore.collection('mycollection').where('a', '===', 'b').and('b', '===', 'c').or('c', '===', 'd').and('d', '===', 'e');

is equivalent to the following expression:

(('a' === 'b') && ('b' === 'c')) || (('c' === 'd') && ('d' === 'e'))

Examine the filtering values in the index.html file

In the event-app directory, open the index.html file and scroll about 1/4 of the way through the file to the start of the <!-- Filtering --> section.

The following code snippet shows how HTML <select/> elements are used for your event app’s filtering features, whose <option/> elements' value attribute values (which are indirectly used by the when() and and() methods in the JavaScript code of your event app), have been specified:

    <!-- Filtering -->
    <div class="form-group">
        <label for="formControlSelect-Category">Category</label>
        <select class="form-control" id="formControlSelect-Category">
            <option value=""></option>
            <option value="Live Music">Live Music</option>
            <option value="Sporting Event">Sporting Event</option>
            <option value="Movie">Movie</option>
        </select>
    </div>
    <div class="form-group">
        <label for="formControlSelect-Cost">Price range</label>
        <select class="form-control" id="formControlSelect-Cost">
            <option value=""></option>
            <option value="0">$0-50</option>
            <option value="50">$50-$100</option>
            <option value="100">$100 or more</option>
        </select>
    </div>
    <div class="form-group">
        <label for="formControlSelect-Ticket">Do you have a ticket?</label>
        <select class="form-control" id="formControlSelect-Ticket">
            <option value=""></option>
            <option value="true">Yes</option>
            <option value="false">No</option>
        </select>
    </div>
    <!-- end Filtering -->
Be aware that any appropriate HTML elements or structured data within your HTML files can be used in your own JavaScript apps to filter Datastore documents in collections.

Examine the filterList function in the main.js file

In the event-app directory, open the main.js file and examine the revised const filterList …​ function definition towards the end of the file.

The following code snippet shows how the filterSettings array’s values are populated, for use in the JS SDK call to retrieve a list of filtered event documents from a specific user’s events collection.

/**
 * Gets the filter vars.
 */
const filterList = () => {
    filterSettings = [];
    const filterCategory = document.querySelector('#formControlSelect-Category').value; (1)
    const filterCost = document.querySelector('#formControlSelect-Cost').value;
    const filterTicket = document.querySelector('#formControlSelect-Ticket').value;

    if (filterTicket) { (2)
        if (filterTicket === 'true') {
            filterSettings.push({property: 'haveTicket', comparison: '===', value: true});
        } else {
            filterSettings.push({property: 'haveTicket', comparison: '===', value: false});
        }
    }

    if (filterCost) { (3)
        if (filterCost === '0') {
            filterSettings.push({property: 'cost', comparison: '>=', value: 0});
            filterSettings.push({property: 'cost', comparison: '<=', value: 50});
        } else if (filterCost === '50') {
            filterSettings.push({property: 'cost', comparison: '>=', value: 50});
            filterSettings.push({property: 'cost', comparison: '<=', value: 100});
        } else if (filterCost === '100') {
            filterSettings.push({property: 'cost', comparison: '>=', value: 100});
        }
    }

    if (filterCategory) { (4)
        filterSettings.push({property: 'category', comparison: '===', value: filterCategory});
    }

    refreshEvents();
};
1 JavaScript code to select the select element in your event app’s index.html file above and assigns the currently selected option element’s value attribute’s value to the filterCategory const.
The code in the following two lines behaves similarly to this line, but assigns the relevant currently selected option element’s value attribute’s value to the appropriate filterCost or filterTicket const.
2 If a filterTicket value has been specified (from your event app’s Do you have a ticket? field), this JavaScript code adds this filter’s appropriate property, comparison and value values as an object to the filterSettings array.
3 If a filterCost value has been specified (from your event app’s Price range field), this code adds this filter’s property, comparison and value values as an object to the filterSettings array.
4 If a filterCategory value has been specified (from your event app’s Category field), this code adds this filter’s property, comparison and value values as an object to the filterSettings array.

Examine the revised getEvents function in the main.js file

Keep the main.js file and examine the revised const getEvents …​ function definition towards the top of the file.

The following code snippet shows the JS SDK call to retrieve a list of filtered event documents from a specific user’s events collection.

/**
 * Gets all events for My Event Manager
 */
const getEvents = () => {
    const eventsCollection = datastore.collection('users').doc(currentUser.userid).collection('events'); (1)

    // Sorting.
    const sortSettings = document.querySelector('#sortEvents').value;
    if (sortSettings) {
        ...
        // Code for handling the current sort settings.
        ...
    }

    // Filtering.
    if (filterSettings) { (2)
        for (let i = 0; i < filterSettings.length; i++) { (3)
            const filter = filterSettings[i];
            if (i === 0) {
                eventsCollection.where(filter.property, filter.comparison, filter.value); (4)
            } else {
                eventsCollection.and(filter.property, filter.comparison, filter.value); (5)
            }
        }
    }

    eventsCollection.get().then( (6)
        (events) => {
            if (events.length === 0) {
                ...
                // Code to handle what happens when there are no events.
                ...
            } else {
                events.forEach((event) => {
                    printEvent(event);
                });
            }
        }
    );
};
1 The JS SDK call to create a collection request object, which in turn is used to prepare a GET request for a list of events (belonging to the current user) from your Datastore service.
The final collection() method in this line prepares this collection request on the events collection, of the current user’s user document within the users collection defined in the modified API specification in tutorial 5.
The collection request object is assigned to the eventsCollection const.
2 If the filterSettings array is not empty, then …​
3 Iterate through filterSettings's filter objects, and if the end of the array has not yet been reached, then …​
4 If the first filter object in the array is being processed, call the JS SDK where() filtering method on the collection request object (eventsCollection) with the first filter object’s values, consisting of the property, comparison and value values.
5 Otherwise, if the second or subsequent filter object in the array is being processed, call the JS SDK and() filtering method on the collection request object (eventsCollection) with the next filter object’s values instead.
6 The remainder of the JS SDK call to get the list of events.
The get() method is finally called on the collection request object (eventsCollection) to send a GET request to your Datastore service that retrieves the events collection’s array of individual event document objects, which have been filtered.
This method uses the API’s get definition within /users/{userid}/events, defined in the modified API specification in tutorial 5, to retrieve the requested collection of event documents (associated with your app’s currentUser.userid value) from your Datastore service.

How JS SDK calls resolve to API requests

The JS SDK where(), and(), and or() filtering methods actually resolve to a URL query string parameter, where:

  • where() resolves to the parameter’s initial field name filters,

  • and() resolves to a semicolon (;) as part of the parameter’s value, and

  • or() resolves to a comma (,) as part of the value too,

in the API GET request made to your Datastore service.

To examine how these JS SDK calls are resolved into API requests:

  1. Log in through your event app as one of the users you created above (e.g. jsmith).

  2. Access your browser’s Network feature:

    • In Google Chrome, you can access this through More tools  Developer tools  Network.

    • In Firefox, this would be through Web Developer  Network.

  3. Under Filter Events, choose Yes from the Do you have a ticket? field (representing a JS SDK method call of where('haveTicket','===',true)), click Update list, and notice the GET request sent, which should be similar to this:

    http://0.0.0.0:8001/abcd1234/users/jsmith/events?filters=document.haveTicket===true

    where the parameter value document.haveTicket===true refers to filtering documents from the events collection whose values for the haveTicket property (defined in the API spec’s JSON schema), is true.

    If you are using Google Chrome, certain characters in the URL’s query string may be escaped, such as equal signs (=) and spaces.

  4. Clear the Do you have a ticket? field, choose Live Music from the Category field (representing a JS SDK method call of where('category','===','Live Music')), click Update list, and notice the GET request sent, which should be similar to this:

    http://0.0.0.0:8001/abcd1234/users/jsmith/events?filters=document.category===Live Music

    where the parameter value document.category===Live Music refers to filtering documents from the events collection whose values for the category property (defined in the API spec’s JSON schema), is Live Music.

    Filtering on number parameter values can utilize exact match comparisons, like those above with the equals comparison operator (===). However, parameter number values can also utilize greater than-/less than-based comparison operators, as well as the and() method to filter according to number ranges.

  5. Clear the Category field, choose $0-50 from the Price range field (representing the JS SDK method call of where('cost','>=',0).and('cost','<=',50)), click Update list, and notice the GET request sent, which should be similar to this:

    http://0.0.0.0:8001/abcd1234/users/jsmith/events?filters=document.cost>=0;document.cost<=50

    where the parameter value document.cost>=0;document.cost<=50 refers to filtering documents from the events collection, whose values for the cost property (defined in the API spec’s JSON schema), is greater than or equal to 0, and (i.e. denoted by ;) less than or equal to 50.

    Last, the sortBy() method (described in tutorial 5) and filtering methods described in this tutorial, can be combined to both filter and sort collections retrieved from your Datastore service.

  6. Clear the Price range field, choose Yes from the Do you have a ticket? field, and choose Price - Low to high from the dropdown list at the top. This combination represents the JS SDK method call where('haveTicket','===',true).sortBy('cost') on the collection request object.

  7. Click Update list and notice the GET request sent, which should be similar to this:

    http://0.0.0.0:8001/abcd1234/users/jsmith/events?filters=document.haveTicket===true&sort=document.cost

    where the:

    • filters parameter value document.haveTicket===true refers to filtering documents from the events collection whose values for the haveTicket property (defined in the API spec’s JSON schema), is true, and

    • sort parameter value document.cost refers to sorting the filtered documents from the events collection according to their cost property (defined in the API spec’s JSON schema), in ascending order.

    When filtering and sorting are combined in a collection request to the Datastore service, filtering is always performed before sorting.

Now that you understand how to implement functionality to utilize your Datastore service’s server-side document filtering features for collection retrieval, you can now extend the functionality of your event app to paginate event documents retrieved from your Datastore service.