Modify any persistence (write) helpers to batch their puts to increase throughput

12907
Reporter: omeyn
Assignee: omeyn
Type: Improvement
Summary: Modify any persistence (write) helpers to batch their puts to increase throughput
Description: Right now each put gets flushed immediately meaning there are more roundtrips to hbase than needed. HBaseStore is the primary culprit here.
Priority: Major
Resolution: WontFix
Status: Closed
Created: 2013-03-04 16:40:54.507
Updated: 2013-12-17 15:17:11.413
Resolved: 2013-10-03 15:50:37.396

Author: omeyn@gbif.org
Comment: Except that frag and verb processors need to write immediately because they send only occurrence keys in their downstream messages meaning the next step will immediately lookup the key. Interp processor sends the whole occurrence so could batch puts.
Created: 2013-03-15 10:24:17.435
Updated: 2013-03-15 10:24:17.435

Author: omeyn@gbif.org
Comment: The only place this would help is on the final interp occ write. If that turns up as a bottleneck (unlikely, as more limited by throughput of maps, cubes and index updaters) this issue can be reopened.
Created: 2013-10-03 15:50:37.426
Updated: 2013-10-03 15:50:37.426