Make sure that the Tapir Crawler supports greater page sizes and automatically handle the case where we request pages too large

12530
Reporter: lfrancke
Assignee: lfrancke
Type: Improvement
Summary: Make sure that the Tapir Crawler supports greater page sizes and automatically handle the case where we request pages too large
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2012-12-12 12:17:19.311
Updated: 2013-12-17 15:46:38.795
Resolved: 2013-03-26 11:21:44.738

Author: lfrancke@gbif.org
Created: 2013-03-26 11:20:28.837
Updated: 2013-03-26 11:20:28.837
        
I'll set the page size for TAPIR back to 1000.

Currently we advance either by the number of records we got in the last request or by the default limit (which is now 1000). This works as long as a record count is provided. If that is wrong or not provided at all and we request 1000 while the limit is set to 200 we'll skip records.

The crawler needs to be more intelligent at handling these kind of responses but that's out of scope for now. I've added a comment in the code.