Reduce logging of the DataNode service

12042
Reporter: trobertson
Assignee: omeyn
Type: Improvement
Summary: Reduce logging of the DataNode service
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2012-10-17 16:16:51.044
Updated: 2013-12-05 11:15:05.709
Resolved: 2012-11-13 14:08:37.836
        
Description: HBase can spawn a lot of DN logs unnecessarily, e.g.:

bq.2012-10-15 23:59:59,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /130.226.238.182:50010, dest: /130.226.238.182:36641, bytes: 66564, op: HDFS_READ, cliID: DFSClient_hb_rs_c4n2.gbif.org,60020,1347355114333_1347355132883, offset: 55002624, srvID: DS-595496612-130.226.238.182-50010-1335949372838, blockid: blk_772796476940862069_1681802, duration: 133204
bq.2012-10-15 23:59:59,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /130.226.238.182:50010, dest: /130.226.238.182:36638, bytes: 66564, op: HDFS_READ, cliID: DFSClient_hb_rs_c4n2.gbif.org,60020,1347355114333_1347355132883, offset: 37387776, srvID: DS-595496612-130.226.238.182-50010-1335949372838, blockid: blk_5051173314344499974_1681270, duration: 175894

I propose this gets added to the DN log4j.properties:
  bq.log4j.logger.org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace=WARN

However, the voodoo on this machine makes it very difficult to understand where the DN is actually getting it's config from (seemingly from a jar file?), but there are many levels of indirection through sym linking I can't be sure.

I *think* if we add a log4j properties to puppet in the modules/cdh/templates/hdfs/log4j.properties.erb then puppet will handle this, but I am not totally sure.]]>