Occurrence downloads fail silently

17649
Reporter: fmendez
Assignee: fmendez
Type: Feedback
Summary: Occurrence downloads fail silently
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2015-06-25 21:44:58.721
Updated: 2015-06-25 22:18:59.26
Resolved: 2015-06-25 21:47:51.805
        
Description: Several users reported that occurrence DWCA downloads failed unexpectedly, the Hive query that produces the intermediate tables silently failed with this error:
2015-06-25 06:38:39	Starting to launch local task to process map join;	maximum memory = 1005060096
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-5

Logs:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [2]


]]>
    

Author: fmendez@gbif.org
Created: 2015-06-25 21:47:51.825
Updated: 2015-06-25 22:18:59.257
        
Hive query was failing silently with OOM errors, this bug has been previuosly reported here https://issues.apache.org/jira/browse/HIVE-2601. To avoid this error hive.auto.convert.join was disable when creating the multimedia table:
SET hive.auto.convert.join=false;
CREATE TABLE ${multimediaTable}
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
AS SELECT m.*
FROM
  occurrence_multimedia m
  JOIN  ${interpretedTable} i ON m.gbifId = i.gbifId;
SET hive.auto.convert.join=true;

https://github.com/gbif/occurrence/commit/213e431cc2b59e41826ced72722c4226468e8ba3
https://github.com/gbif/occurrence/commit/65e75296e0aa2dd7463941cb31867c741ff598a5

Related doc:
http://dmtolpeko.com/2014/10/14/map-join-limitations-out-of-memory-in-local-task/
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+JoinOptimization