Crawler: Optimize name range crawling

17334
Reporter: trobertson
Type: Improvement
Summary: Crawler: Optimize name range crawling
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2015-03-02 11:28:15.017
Updated: 2015-03-02 13:43:06.579
Resolved: 2015-03-02 13:43:06.554
        
Description: Some protocols provide a declared count of records.  For those endpoints that assert they have a small number of records, we should use a more aggressive name range, rather than Aa-Ab, Ab-Ac etc.

Suggest we consider >1000 records in a single A-Z request, and <10,000 records in A-B,B-A etc.
]]>
    

Author: trobertson@gbif.org
Comment: https://github.com/gbif/crawler/commit/9742711c487a5c5ff3cca713394e6a7fd7bfe178
Created: 2015-03-02 13:43:06.577
Updated: 2015-03-02 13:43:06.577