Make Fragment Processor faster

13030
Reporter: omeyn
Assignee: omeyn
Type: Improvement
Summary: Make Fragment Processor faster
Description: currently looks like process time is limited by (and 95% comprised of) generating the id for new records (~50ms for the total new frag persist). Make that faster - even 10ms makes a big difference.
Priority: Critical
Resolution: Fixed
Status: Closed
Created: 2013-03-18 16:42:35.127
Updated: 2013-12-17 15:17:24.284
Resolved: 2013-04-18 12:09:12.316

Author: omeyn@gbif.org
Comment: Initial investigation reveal a throughput limit of ~3600 ids/sec on the HBase incrementColumnValue used to generate new ids. This effectively limits the Fragment Processor to a maximum of 3600 fragments processed / sec when all records are new. When all records are updates the persistence step of the processing has a throughput of ~8k/sec.
Created: 2013-03-20 09:08:38.547
Updated: 2013-03-20 09:08:38.547

Author: omeyn@gbif.org
Comment: it's now faster than the interpreter, which means it's not a bottleneck
Created: 2013-04-18 12:09:12.353
Updated: 2013-04-18 12:09:12.353