Issue 17526

Investigate feasibility of NFS over HDFS

17526
Reporter: omeyn
Assignee: omeyn
Type: SubTask
Summary: Investigate feasibility of NFS over HDFS
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2015-03-25 10:56:43.491
Updated: 2015-03-31 13:47:01.733
Resolved: 2015-03-27 15:11:47.16

Attachment Screen Shot 2015-03-31 at 13.46.42.png



Author: omeyn@gbif.org
Created: 2015-03-26 09:40:11.766
Updated: 2015-03-26 09:40:11.766
        
The hard part is using an export of hdfs that is below the root (ie /). To do that you have to add this property to the NFS Gateway's safety valve:


dfs.nfs3.export.point
/occurrence-download/dev-downloads


And then mount from your client (with appropriate permissions) as:

mount -t nfs -o vers=3,proto=tcp,nolock c1n1.gbif.org:/occurrence-download/dev-downloads  /mnt/testnfs
    


Author: omeyn@gbif.org
Comment: Needs performance testing.
Created: 2015-03-26 09:54:03.759
Updated: 2015-03-26 09:54:03.759


Author: omeyn@gbif.org
Comment: From a single client machine the transfer rate seems to be limited to about 50-60 Mb (megabytes) per second, so about half the theoretical max of a single gigabit connection. This seems sufficient for the purposes of reading downloads to serve to users. We should use the java api to do our writes though.
Created: 2015-03-27 15:11:42.504
Updated: 2015-03-27 15:11:42.504