URLStore reported no documents stored

Description

This error occurs if the web crawler is unable to access any of the configured start URLs.

Error message

Displayed in the crawler.log file.

Coordinator: WARNING - URLStore reported no documents stored!

Cause

This means the web crawler couldn’t access any of the specified start URLs.

Resolution

  1. Confirm that the data source’s start URLs are valid and return a status 200 (OK). e.g. attempt to access the failed URL using your web browser.

  2. Examine entries for the start URLs in the data source crawl logs (crawl.log.*, url_errors.log).

    Common issues include:

    • The URL was temporarily unavailable

    • robots.txt file on webserver blocking access

    • The start URL redirects to another URL that doesn’t match any include patterns

    • The start URL requires authentication

  3. Correct the error and then update the data source