Big downloads step sometimes fails with hdfs error (triggers cdh alert)

14605
Reporter: omeyn
Type: Bug
Summary: Big downloads step sometimes fails with hdfs error (triggers cdh alert)
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2014-01-09 15:02:28.103
Updated: 2015-03-10 11:03:02.197
Resolved: 2015-03-10 11:03:02.176
        
Description: The error log looks like this:

java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://c1n8.gbif.org:8020/tmp/hive-occurrence-download/hive_2014-01-09_14-41-08_965_46694990421724538/_task_tmp.-ext-10002/_tmp.000030_0 to: hdfs://c1n8.gbif.org:8020/tmp/hive-occurrence-download/hive_2014-01-09_14-41-08_965_46694990421724538/_tmp.-ext-10002/000030_0
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:191)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:103)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:747)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)

and may be related to:

http://mail-archives.apache.org/mod_mbox/hive-user/201306.mbox/%3CCANsfGRkR0jy5W3eY3z8aWTWpypHGu5YeDicybVJBnwR_o_52Vg@mail.gmail.com%3E

and

https://issues.apache.org/jira/browse/HIVE-4605]]>
    

Author: omeyn@gbif.org
Comment: cdh4, haven't seen since moving to cdh5
Created: 2015-03-10 11:03:02.195
Updated: 2015-03-10 11:03:02.195