Help Center

Menu

Update administration interface progress message from a custom gatherer

Background

This article shows how to provide update messaging (such as the number of documents that have been stored) from a custom gatherer.

Details

Custom gatherers provide a way of gathering content from an arbitrary datasource using custom gather logic. When gathering content it is best practice to provide useful status messages that are displayed in the administration interface so that an administrator has visibility over what the gatherer is doing and constant enough feedback so that there is no doubt whether or not the gatherer may be stuck.

The following function calls can be made from within the custom gatherer to enable useful messaging to be returned:

  • config.isUpdateStopped(): Returns a boolean indicating if a stop has been requested (e.g. by clicking on the stop update button) for the collection. This allows you to gracefully exit the update in a timely manner. If you don’t check periodically for this flag then the update will run until the end of the custom gather process before the update halts.

  • config.setProgressMessage(string message): Sets the progress message that is displayed in the administration interface. This should be used to provide regular feedback to an administrator such as the number of documents that have been stored. If the progress message is not set it will display a generic gathering message for the duration of the gather process. The gather process may take a long time to complete depending on what is being gathered and this can lead to confusion about whether or not the update is actually doing anything.

Example: Check for a stopped update and provide a status message for every 100 records processed

The following code should be used within the main gathering loop of the custom gatherer (the part of the code that fetches and stores the content).

custom_gather.groovy
// store and skip counter, used for progress messages
def storeCounter = 0
def skipCounter = 0

while (item = getNextRecord())
{
    // Check for a user stop every 100 records processed and update the admin UI status message
    if (((storeCounter+skipCounter) % 100) == 0)
    {
        // Check to see if the update has been stopped
        if (config.isUpdateStopped()) {
            store.close()
            throw new RuntimeException("Update stop requested by user.");
        }
        // Set the collection update progress message
        config.setProgressMessage("Stored "+storeCounter+" records, skipped "+skipCounter+" records.");
        println("Progress: Stored "+storeCounter+" records, skipped "+skipCounter+" records.")
    }


    // Do any record processing as required

    ...

    if (<RECORD SHOULD BE STORED>) {

        // Add the item and associated to the document store
        store.add(record,metadata)

        // Increment the store counter
        storeCounter++
    }
    else {
        // Increment the skip counter
        skipCounter++

        println("Skipping record: "+recordUrl+" due to exclude pattern.")
    }
}