Issue 11639
HBase ScannerTimeoutException when creating SolrIndex
11639
Reporter: fmendez
Assignee: omeyn
Type: Bug
Summary: HBase ScannerTimeoutException when creating SolrIndex
Priority: Major
Resolution: Invalid
Status: Closed
Created: 2012-08-01 09:16:46.466
Updated: 2013-12-17 15:16:46.151
Resolved: 2012-08-07 09:52:23.956
Description: The following exception was thrown after reading 16,772,800 rows:
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hbase.client.ScannerTimeoutException: 61332ms passed since the last invocation, timeout is currently set to 60000
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1278)
at org.gbif.occurrence.index.hbase.SequentialOccurrenceIndexBuilder.build(SequentialOccurrenceIndexBuilder.java:130)
at org.gbif.occurrence.index.hbase.SequentialOccurrenceIndexBuilder.main(SequentialOccurrenceIndexBuilder.java:65)
Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 61332ms passed since the last invocation, timeout is currently set to 60000
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1179)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1275)
... 2 more
Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -4418127785014574014
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1861)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:83)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:38)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1226)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1170)
... 3 more
The Indexer is: http://code.google.com/p/gbif-occurrencestore/source/browse/occurrence/trunk/occurrence-index-builder/src/main/java/org/gbif/occurrence/index/hbase/SequentialOccurrenceIndexBuilder.java]]>
Author: omeyn@gbif.org
Created: 2012-08-01 09:32:19.357
Updated: 2012-08-01 09:32:19.357
Not sure if conf.set("hbase.client.scanner.caching", "200"); is doing what it should. Try doing it the api way with scan.setCaching(200);
The problem seems to be that after awhile adding the document takes a really long time, so you don't call next() on the scanner for the timeout period (60000ms). So try timing each document add (the iterator loop). Also try lowering the cache size.