Hi folks,
I research at the moment a connection over fuse and samba for
hadoop-cluster. Its my private playground, but I have some issues I can't
figure out why they happen.
I use RHEL5.7, packetlist:
rpm -qa|grep samba
samba-3.0.33-3.29.el5_7.4.x86_64
samba-common-3.0.33-3.29.el5_7.4.x86_64
2 servers, one provides samba-shares and one I use as client.
Samba-Server (HOST2):
FuseFS:
hadoop-fuse-dfs#dfs://HOSTNAME:9000 /123/hdfs fuse usetrash,rw 0 0
mount:
fuse on /123/hdfs type fuse
(rw,nosuid,nodev,allow_other,default_permissions)
the mount /123 is accessible over ssh, cp, rsnyc from any host. Now I was
using NFS for populating, but the performance was not as I expected. So I
switched to samba and get real good results for reading.
Cifs-Client(HOST1):
mount -t cifs -o username=BLUBB,domain=hdfs,rw,uid=104 //lHOST2/hdfs /test
config:
[global]
workgroup = MYGROUP
server string = Samba Server Version %v
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
security = user
smb passwd file = /etc/samba/smbpasswd
#passdb backend = tdbsam
[hdfs]
comment = hhh
path = /123
public = yes
writable = yes
printable = no
create mask = 0744
force user = hdfs
force group = nobody
I want only hdfs exported ;)
read-performance with rsync:
rsync -av /test/hdfs/user/flume/weblogs/2011-12-07/ .
sent 20478888644 bytes received 92606 bytes 17377158.46 bytes/sec
total size is 20475835998
(19GB, 16 MB/s)
find . -type f |wc -l
4665
plain over rsync -essh .... I get around 20mb/s, so I can live with. But
now the part I spent days without a solution:
I perform a lot of write tests. I have a 10GB plain file (dd if of), thats
I use for all write tests. Now, if I try to copy the file from host1 (cifs
client), which hangs without socket options at 128MB. The interest part
there ist - 129MB are the blocksize in my hadoop-cluster. I played with
socket options and now the copy stops at 728MB. I get a nmb-error in
messages:
Dec 8 14:38:42 HOST2 nmbd[12609]: [2011/12/08 14:38:42, 0]
nmbd/nmbd.c:terminate(58)
Dec 8 14:38:42 HOST2 nmbd[12609]: Got SIGTERM: going down...
Dec 8 14:42:46 HOST2 kernel: INFO: task ls:12663 blocked for more than 120
seconds.
after the system freezes, HOST1 going into load 41 up. vmstat und iostat
say nothing, cp is going into state Z (zombie). Has anybody a advice? Could
be that the process is overloaded and die?
A problem with fuse I could imagine, because if I copy the same file over
scp from HOST1 => HOST2 into hdfs I get a speed of 41 MB/s without any
errors.
thanks,
- Alex
--
Alexander Lorenz
http://mapredit.blogspot.com
*P **Think of the environment: please don't print this email unless you
really need to.*
Hi folks,
I research at the moment a connection over fuse and samba for
hadoop-cluster. Its my private playground, but I have some issues I can't
figure out why they happen.
I use RHEL5.7, packetlist:
rpm -qa|grep samba
samba-3.0.33-3.29.el5_7.4.x86_64
samba-common-3.0.33-3.29.el5_7.4.x86_64
2 servers, one provides samba-shares and one I use as client.
Samba-Server (HOST2):
FuseFS:
hadoop-fuse-dfs#dfs://HOSTNAME:9000 /123/hdfs fuse usetrash,rw 0 0
mount:
fuse on /123/hdfs type fuse
(rw,nosuid,nodev,allow_other,default_permissions)
the mount /123 is accessible over ssh, cp, rsnyc from any host. Now I was
using NFS for populating, but the performance was not as I expected. So I
switched to samba and get real good results for reading.
Cifs-Client(HOST1):
mount -t cifs -o username=BLUBB,domain=hdfs,rw,uid=104 //lHOST2/hdfs /test
config:
[global]
workgroup = MYGROUP
server string = Samba Server Version %v
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
security = user
smb passwd file = /etc/samba/smbpasswd
#passdb backend = tdbsam
[hdfs]
comment = hhh
path = /123
public = yes
writable = yes
printable = no
create mask = 0744
force user = hdfs
force group = nobody
I want only hdfs exported ;)
read-performance with rsync:
rsync -av /test/hdfs/user/flume/weblogs/2011-12-07/ .
sent 20478888644 bytes received 92606 bytes 17377158.46 bytes/sec
total size is 20475835998
(19GB, 16 MB/s)
find . -type f |wc -l
4665
plain over rsync -essh .... I get around 20mb/s, so I can live with. But
now the part I spent days without a solution:
I perform a lot of write tests. I have a 10GB plain file (dd if of), thats
I use for all write tests. Now, if I try to copy the file from host1 (cifs
client), which hangs without socket options at 128MB. The interest part
there ist - 129MB are the blocksize in my hadoop-cluster. I played with
socket options and now the copy stops at 728MB. I get a nmb-error in
messages:
Dec 8 14:38:42 HOST2 nmbd[12609]: [2011/12/08 14:38:42, 0]
nmbd/nmbd.c:terminate(58)
Dec 8 14:38:42 HOST2 nmbd[12609]: Got SIGTERM: going down...
Dec 8 14:42:46 HOST2 kernel: INFO: task ls:12663 blocked for more than 120
seconds.
after the system freezes, HOST1 going into load 41 up. vmstat und iostat
say nothing, cp is going into state Z (zombie). Has anybody a advice? Could
be that the process is overloaded and die?
A problem with fuse I could imagine, because if I copy the same file over
scp from HOST1 => HOST2 into hdfs I get a speed of 41 MB/s without any
errors.
thanks,
- Alex
--
Alexander Lorenz
http://mapredit.blogspot.com
*P **Think of the environment: please don't print this email unless you
really need to.*