Improve performance of the HBase based counters by removing reliance on the column increment value

13040
Reporter: trobertson
Assignee: omeyn
Type: Improvement
Summary: Improve performance of the HBase based counters by removing reliance on the column increment value
Priority: Critical
Resolution: Fixed
Status: Closed
Created: 2013-03-20 11:04:26.572
Updated: 2013-12-17 15:17:23.875
Resolved: 2013-03-20 12:58:09.465
        
Description: HBase column increment counter is limited to a few thousand ops per second.  This is a known limitation to the fragment persister on a cold start, or on a large DwC-A import.
Reserving batches of IDs would help eliminate this.]]>
    

Author: omeyn@gbif.org
Comment: By batching calls to hbase id generation goes to ~80k/sec with 100 id batch, 380k/sec with 1000 id batch, and 1M/sec with 10000 id batch. Slightly altered version of patch incorporated.
Created: 2013-03-20 12:55:06.974
Updated: 2013-03-20 12:55:06.974

Author: trobertson@gbif.org
Comment: Can you post link to commit please, just for a peer review (has potential for off by errors etc, which were caught in first review by Lars)
Created: 2013-03-20 13:10:13.295
Updated: 2013-03-20 13:10:13.295

Author: omeyn@gbif.org
Comment: reviewed in CR-OCC-18
Created: 2013-03-21 11:47:31.115
Updated: 2013-03-21 11:47:31.115

Author: omeyn@gbif.org
Comment: commit up for review at http://dev.gbif.org/code/cru/CR-OCC-19
Created: 2013-03-21 13:46:38.85
Updated: 2013-03-21 13:46:38.85