Modify any persistence (write) helpers to batch their puts to increase throughput
Summary: Modify any persistence (write) helpers to batch their puts to increase throughput
Description: Right now each put gets flushed immediately meaning there are more roundtrips to hbase than needed. HBaseStore is the primary culprit here.
Created: 2013-03-04 16:40:54.507
Updated: 2013-12-17 15:17:11.413
Resolved: 2013-10-03 15:50:37.396
Comment: Except that frag and verb processors need to write immediately because they send only occurrence keys in their downstream messages meaning the next step will immediately lookup the key. Interp processor sends the whole occurrence so could batch puts.
Created: 2013-03-15 10:24:17.435
Updated: 2013-03-15 10:24:17.435
Comment: The only place this would help is on the final interp occ write. If that turns up as a bottleneck (unlikely, as more limited by throughput of maps, cubes and index updaters) this issue can be reopened.
Created: 2013-10-03 15:50:37.426
Updated: 2013-10-03 15:50:37.426