Issue 17527
Investigate feasibility of HTTP over HDFS
17527
Reporter: omeyn
Assignee: omeyn
Type: SubTask
Summary: Investigate feasibility of HTTP over HDFS
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2015-03-25 10:56:58.362
Updated: 2015-03-31 14:52:36.07
Resolved: 2015-03-31 14:52:36.035
Attachment Screen Shot 2015-03-31 at 11.53.53.png
Author: omeyn@gbif.org
Created: 2015-03-31 11:39:55.527
Updated: 2015-03-31 11:39:55.527
Testing with 6 concurrent wget on apps.gbif-uat.org to httpfs running on prodmaster1-vh I saw close to 2gig speed. Once 3 had finished (all different files, with different sizes) the speed was capped at 1gig, presumably because the lacp hashing algorithm had assigned the remaining 3 to the same single port of the two bonded ports on the master. (Master had 2x1gig bond, apps on a blade so had 4x1gig).
Security is a problem - entire filesystem is accessible, with full api (ie read and write) access. Can be limited to specific users, which should make it reasonably safe, but still not as safe as the nfs option.