thr3ads.net - Gluster users - [Gluster-users] gluster 3.0 read hangs [Dec 2009]

If this information is useful, please help other people find it:
Share via:

Nick Birkett

2009-Dec-23 09:34 UTC

[Gluster-users] gluster 3.0 read hangs

I ran some benchmarks last week using 2.0.8. Single server with 8 Intel
e1000e bonded mode=balance-alb

All worked fine and I got some good results using 8 clients. All Gigabit.

The benchmarks did 2 passes of IOZONE in network mode using 1-8 threads
per client and using 1 - 8 clients. Each client used 32Gbyte files.

All jobs completed successfully. This takes about 32 hours to run
through all cases.

Yesterday I updated to 3.0.0 (server and clients) and re-configured the
server and client vol files using glusterfs-volgen (renamed some of the
vol names).

RedHat EL5 binary packages from Glusterfs site installed
glusterfs-server-3.0.0-1.x86_64
glusterfs-common-3.0.0-1.x86_64
glusterfs-client-3.0.0-1.x86_64

All works mainly ok, except every so often the IOZONE job just stops.
The network IO drops to zero.
This is always happens during either a read or re-read test. It happes
just as the IOZONE read
test starts. It doesnt happen every time and it may run
for several hours without incident. This has happened 6 times on
different test cases (thread/clients).

Anyone else noticed this ? Perhaps I have done something wrong ?

vol files attached - I know I dont need to distribute 1 remote vol -
part of larger test with multiple vols.

Attached sample outputs. 4 clients 4 files per client ran fine. 4
clients 8 files per client hung at re-read
on 2nd pass of IOZONE. All jobs with 5 clients and 8 clients ran to
completion.

Thanks,

Nick

This e-mail message may contain confidential and/or privileged information. If
you are not an addressee or otherwise authorized to receive this message, you
should not use, copy, disclose or take any action based on this e-mail or any
information contained in the message.
If you have received this material in error, please advise the sender
immediately by reply e-mail and delete this message. Thank you.
Streamline Computing is a trading division of Concurrent Thinking Limited:
Registered in England and Wales No: 03913912
Registered Address: The Innovation Centre, Warwick Technology Park, Gallows
Hill, Warwick, CV34 6UW, United Kingdom
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: data2.vol
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20091223/0f9bc45d/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: glusterfsd.vol
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20091223/0f9bc45d/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: clients=4.threads=4.gfs3_10_nics2.o556
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20091223/0f9bc45d/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: clients=4.threads=8.gfs3_10_nics2.o557
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20091223/0f9bc45d/attachment-0003.ksh>

Vikas Gorur

2009-Dec-23 09:38 UTC

head link

[Gluster-users] gluster 3.0 read hangs

Nick Birkett wrote:> Yesterday I updated to 3.0.0 (server and clients) and re-configured 
> the server and client vol files using glusterfs-volgen (renamed some 
> of the vol names).Nick,

Thanks for testing. If you still have the hung processes, can you get 
the statedump
for us (for both servers and clients). You can get the state dump of a 
glusterfs
process by

# kill -USR1 <pid of server or client>

You'll find the dump file /tmp/glusterdump.<pid> Please send us all
the
dump files. This will help us debug the hang.

Vikas

Tejas N. Bhise

2009-Dec-23 09:42 UTC

head link

[Gluster-users] gluster 3.0 read hangs

Hi Nick,

Thank you for using Gluster and sending us such detailed description of the
problem you are seeing. We will try a run with exactly the same switches and
config as you mention and see if we can reproduce this inhouse to make debugging
easier.

Regards,
Tejas.

----- Original Message -----
From: "Nick Birkett" <nick at streamline-computing.com>
To: gluster-users at gluster.org
Sent: Wednesday, December 23, 2009 3:04:43 PM GMT +05:30 Chennai, Kolkata,
Mumbai, New Delhi
Subject: [Gluster-users] gluster 3.0 read hangs

I ran some benchmarks last week using 2.0.8. Single server with 8 Intel 
e1000e  bonded  mode=balance-alb

All worked fine and I got some good results using 8 clients. All Gigabit.

The benchmarks did 2 passes of IOZONE  in network mode using 1-8 threads 
per client and using 1 - 8 clients. Each client used 32Gbyte files.

All jobs completed successfully. This takes about 32 hours to run 
through all cases.

Yesterday I updated to 3.0.0 (server and clients) and re-configured the 
server and client vol files using glusterfs-volgen (renamed some of the 
vol names).

RedHat EL5 binary packages from Glusterfs site installed
glusterfs-server-3.0.0-1.x86_64
glusterfs-common-3.0.0-1.x86_64
glusterfs-client-3.0.0-1.x86_64


All works mainly ok, except every so often the IOZONE job just stops. 
The network IO drops to zero.
This is always happens during either a read or re-read test. It happes 
just as the IOZONE read
test starts. It doesnt happen every time and it may run
for several hours without incident. This has happened 6 times on 
different test cases (thread/clients).



Anyone else noticed this ? Perhaps I have done something wrong ?

vol files attached  - I know I dont need to distribute 1 remote vol - 
part of larger test with multiple vols.

Attached sample outputs. 4 clients 4 files per client ran fine. 4 
clients 8 files per client hung at re-read
on 2nd pass of IOZONE. All jobs with 5 clients and 8 clients ran to 
completion.

Thanks,

Nick

 




This e-mail message may contain confidential and/or privileged information. If
you are not an addressee or otherwise authorized to receive this message, you
should not use, copy, disclose or take any action based on this e-mail or any
information contained in the message.
If you have received this material in error, please advise the sender
immediately by reply e-mail and delete this message. Thank you.
Streamline Computing is a trading division of Concurrent Thinking Limited:
Registered in England and Wales No: 03913912
Registered Address: The Innovation Centre, Warwick Technology Park, Gallows
Hill, Warwick, CV34 6UW, United Kingdom

 volume brick00.server-e
   type protocol/client
   option transport-type tcp
   option transport.socket.nodelay on
   option transport.remote-port 6996
   option remote-host 192.168.100.200 # can be IP or hostname
   option remote-subvolume brick00
 end-volume

 volume distribute
   type cluster/distribute
   subvolumes  brick00.server-e
 end-volume


volume writebehind
    type performance/write-behind
    option cache-size 4MB
    subvolumes distribute
end-volume

volume readahead
    type performance/read-ahead
    option page-count 4
    subvolumes writebehind
end-volume

volume iocache
    type performance/io-cache
    option cache-size 1GB
    option cache-timeout 1
    subvolumes readahead
end-volume

volume quickread
    type performance/quick-read
    option cache-timeout 1
    option max-file-size 64kB
    subvolumes iocache
end-volume

volume statprefetch
    type performance/stat-prefetch
    subvolumes quickread
end-volume

#glusterfsd_keep=0
 volume posix00
   type storage/posix
   option directory /data/data00
 end-volume

 volume locks00
   type features/locks
   subvolumes posix00
 end-volume

 volume brick00
   type performance/io-threads
   option thread-count 8
   subvolumes locks00
 end-volume

 volume server
   type protocol/server
   option transport-type tcp
   option transport.socket.listen-port 6996
   option transport.socket.nodelay on
   option auth.addr.brick00.allow *
   subvolumes  brick00
 end-volume

=========================================================Cluster name         :
Delldemo
Arch                 : x86_64 
SGE job submitted    : Tue Dec 22 22:21:38 GMT 2009
Number of CPUS 8 
Running Parallel IOZONE on ral03
Creating files in /data2/sccomp
NTHREADS=4
Total data size = 48196 MBytes
Running loop 1 of 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Tue Dec 22 22:21:38 2009

	Network distribution mode enabled.
	File size set to 12338176 KB
	Command line used: /opt/iozone/bin/iozone -+m hosts.556 -s 12049m -S 8192 -T -i
0 -i 1 -t 16 -F /data2/sccomp/BIG.0.comp03.streamline
/data2/sccomp/BIG.1.comp03.streamline /data2/sccomp/BIG.2.comp03.streamline
/data2/sccomp/BIG.3.comp03.streamline /data2/sccomp/BIG.0.ral02.streamline
/data2/sccomp/BIG.1.ral02.streamline /data2/sccomp/BIG.2.ral02.streamline
/data2/sccomp/BIG.3.ral02.streamline /data2/sccomp/BIG.0.ral03.streamline
/data2/sccomp/BIG.1.ral03.streamline /data2/sccomp/BIG.2.ral03.streamline
/data2/sccomp/BIG.3.ral03.streamline /data2/sccomp/BIG.0.ral04.streamline
/data2/sccomp/BIG.1.ral04.streamline /data2/sccomp/BIG.2.ral04.streamline
/data2/sccomp/BIG.3.ral04.streamline
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 8192 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 16 threads
	Each thread writes a 12338176 Kbyte file in 4 Kbyte records

	Test running:
	Children see throughput for 16 initial writers 	=  424003.62 KB/sec
	Min throughput per thread 			=   26480.14 KB/sec 
	Max throughput per thread 			=   26517.04 KB/sec
	Avg throughput per thread 			=   26500.23 KB/sec
	Min xfer 					= 12321928.00 KB

	Test running:
	Children see throughput for 16 rewriters 	=  424109.61 KB/sec
	Min throughput per thread 			=   26483.30 KB/sec 
	Max throughput per thread 			=   26530.66 KB/sec
	Avg throughput per thread 			=   26506.85 KB/sec
	Min xfer 					= 12316680.00 KB

	Test running:
	Children see throughput for 16 readers 		=  454358.62 KB/sec
	Min throughput per thread 			=   28298.30 KB/sec 
	Max throughput per thread 			=   28592.02 KB/sec
	Avg throughput per thread 			=   28397.41 KB/sec
	Min xfer 					= 12211568.00 KB

	Test running:
	Children see throughput for 16 re-readers 	=  459262.06 KB/sec
	Min throughput per thread 			=   28600.55 KB/sec 
	Max throughput per thread 			=   28892.20 KB/sec
	Avg throughput per thread 			=   28703.88 KB/sec
	Min xfer 					= 12219504.00 KB

	Test cleanup:


iozone test complete.
Running loop 2 of 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Tue Dec 22 22:53:33 2009

	Network distribution mode enabled.
	File size set to 12338176 KB
	Command line used: /opt/iozone/bin/iozone -+m hosts.556 -s 12049m -S 8192 -T -i
0 -i 1 -t 16 -F /data2/sccomp/BIG.0.comp03.streamline
/data2/sccomp/BIG.1.comp03.streamline /data2/sccomp/BIG.2.comp03.streamline
/data2/sccomp/BIG.3.comp03.streamline /data2/sccomp/BIG.0.ral02.streamline
/data2/sccomp/BIG.1.ral02.streamline /data2/sccomp/BIG.2.ral02.streamline
/data2/sccomp/BIG.3.ral02.streamline /data2/sccomp/BIG.0.ral03.streamline
/data2/sccomp/BIG.1.ral03.streamline /data2/sccomp/BIG.2.ral03.streamline
/data2/sccomp/BIG.3.ral03.streamline /data2/sccomp/BIG.0.ral04.streamline
/data2/sccomp/BIG.1.ral04.streamline /data2/sccomp/BIG.2.ral04.streamline
/data2/sccomp/BIG.3.ral04.streamline
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 8192 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 16 threads
	Each thread writes a 12338176 Kbyte file in 4 Kbyte records

	Test running:
	Children see throughput for 16 initial writers 	=  425851.12 KB/sec
	Min throughput per thread 			=   26593.95 KB/sec 
	Max throughput per thread 			=   26634.84 KB/sec
	Avg throughput per thread 			=   26615.70 KB/sec
	Min xfer 					= 12319368.00 KB

	Test running:
	Children see throughput for 16 rewriters 	=  424954.77 KB/sec
	Min throughput per thread 			=   26459.38 KB/sec 
	Max throughput per thread 			=   26656.61 KB/sec
	Avg throughput per thread 			=   26559.67 KB/sec
	Min xfer 					= 12247176.00 KB

	Test running:
	Children see throughput for 16 readers 		=  459433.33 KB/sec
	Min throughput per thread 			=   28449.77 KB/sec 
	Max throughput per thread 			=   28964.50 KB/sec
	Avg throughput per thread 			=   28714.58 KB/sec
	Min xfer 					= 12119024.00 KB

	Test running:
	Children see throughput for 16 re-readers 	=  458413.46 KB/sec
	Min throughput per thread 			=   28457.53 KB/sec 
	Max throughput per thread 			=   28831.23 KB/sec
	Avg throughput per thread 			=   28650.84 KB/sec
	Min xfer 					= 12178288.00 KB

	Test cleanup:


iozone test complete.
echo
echo ---------------
echo Job output ends
echo ========================================================echo SGE job:
finished   date = Tue Dec 22 23:25:20 GMT 2009
echo Total run time : 1 Hours 3 Minutes 42 Seconds
echo Time in seconds: 3822 Seconds
echo ========================================================
=========================================================Cluster name         :
Delldemo
Arch                 : x86_64 
SGE job submitted    : Tue Dec 22 23:25:30 GMT 2009
Number of CPUS 8 
Running Parallel IOZONE on comp01
Creating files in /data2/sccomp
NTHREADS=8
Total data size = 32240 MBytes
Running loop 1 of 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Tue Dec 22 23:25:30 2009

	Network distribution mode enabled.
	File size set to 4126720 KB
	Command line used: /opt/iozone/bin/iozone -+m hosts.557 -s 4030m -S 512 -T -i 0
-i 1 -t 32 -F /data2/sccomp/BIG.0.comp00.streamline
/data2/sccomp/BIG.1.comp00.streamline /data2/sccomp/BIG.2.comp00.streamline
/data2/sccomp/BIG.3.comp00.streamline /data2/sccomp/BIG.4.comp00.streamline
/data2/sccomp/BIG.5.comp00.streamline /data2/sccomp/BIG.6.comp00.streamline
/data2/sccomp/BIG.7.comp00.streamline /data2/sccomp/BIG.0.comp01.streamline
/data2/sccomp/BIG.1.comp01.streamline /data2/sccomp/BIG.2.comp01.streamline
/data2/sccomp/BIG.3.comp01.streamline /data2/sccomp/BIG.4.comp01.streamline
/data2/sccomp/BIG.5.comp01.streamline /data2/sccomp/BIG.6.comp01.streamline
/data2/sccomp/BIG.7.comp01.streamline /data2/sccomp/BIG.0.comp02.streamline
/data2/sccomp/BIG.1.comp02.streamline /data2/sccomp/BIG.2.comp02.streamline
/data2/sccomp/BIG.3.comp02.streamline /data2/sccomp/BIG.4.comp02.streamline
/data2/sccomp/BIG.5.comp02.streamline /data2/sccomp/BIG.6.comp02.streamline
/data2/sccomp/BIG.7.comp02.streamline /data2/sccomp/BIG.0.ral01.streamline
/data2/sccomp/BIG.1.ral01.streamlineCommand line too long to save completely.

	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 512 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 32 threads
	Each thread writes a 4126720 Kbyte file in 4 Kbyte records

	Test running:
	Children see throughput for 32 initial writers 	=  431608.71 KB/sec
	Min throughput per thread 			=   13462.86 KB/sec 
	Max throughput per thread 			=   13516.74 KB/sec
	Avg throughput per thread 			=   13487.77 KB/sec
	Min xfer 					= 4110728.00 KB

	Test running:
	Children see throughput for 32 rewriters 	=  433205.56 KB/sec
	Min throughput per thread 			=   13512.67 KB/sec 
	Max throughput per thread 			=   13550.23 KB/sec
	Avg throughput per thread 			=   13537.67 KB/sec
	Min xfer 					= 4116360.00 KB

	Test running:
	Children see throughput for 32 readers 		=  458239.61 KB/sec
	Min throughput per thread 			=   13983.61 KB/sec 
	Max throughput per thread 			=   14699.36 KB/sec
	Avg throughput per thread 			=   14319.99 KB/sec
	Min xfer 					= 3925872.00 KB

	Test running:
	Children see throughput for 32 re-readers 	=  457589.70 KB/sec
	Min throughput per thread 			=   13990.14 KB/sec 
	Max throughput per thread 			=   14654.56 KB/sec
	Avg throughput per thread 			=   14299.68 KB/sec
	Min xfer 					= 3939696.00 KB

	Test cleanup:


iozone test complete.
Running loop 2 of 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Tue Dec 22 23:48:31 2009

	Network distribution mode enabled.
	File size set to 4126720 KB
	Command line used: /opt/iozone/bin/iozone -+m hosts.557 -s 4030m -S 512 -T -i 0
-i 1 -t 32 -F /data2/sccomp/BIG.0.comp00.streamline
/data2/sccomp/BIG.1.comp00.streamline /data2/sccomp/BIG.2.comp00.streamline
/data2/sccomp/BIG.3.comp00.streamline /data2/sccomp/BIG.4.comp00.streamline
/data2/sccomp/BIG.5.comp00.streamline /data2/sccomp/BIG.6.comp00.streamline
/data2/sccomp/BIG.7.comp00.streamline /data2/sccomp/BIG.0.comp01.streamline
/data2/sccomp/BIG.1.comp01.streamline /data2/sccomp/BIG.2.comp01.streamline
/data2/sccomp/BIG.3.comp01.streamline /data2/sccomp/BIG.4.comp01.streamline
/data2/sccomp/BIG.5.comp01.streamline /data2/sccomp/BIG.6.comp01.streamline
/data2/sccomp/BIG.7.comp01.streamline /data2/sccomp/BIG.0.comp02.streamline
/data2/sccomp/BIG.1.comp02.streamline /data2/sccomp/BIG.2.comp02.streamline
/data2/sccomp/BIG.3.comp02.streamline /data2/sccomp/BIG.4.comp02.streamline
/data2/sccomp/BIG.5.comp02.streamline /data2/sccomp/BIG.6.comp02.streamline
/data2/sccomp/BIG.7.comp02.streamline /data2/sccomp/BIG.0.ral01.streamline
/data2/sccomp/BIG.1.ral01.streamlineCommand line too long to save completely.

	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 512 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 32 threads
	Each thread writes a 4126720 Kbyte file in 4 Kbyte records

	Test running:
	Children see throughput for 32 initial writers 	=  432863.52 KB/sec
	Min throughput per thread 			=   13489.46 KB/sec 
	Max throughput per thread 			=   13564.23 KB/sec
	Avg throughput per thread 			=   13526.99 KB/sec
	Min xfer 					= 4104456.00 KB

	Test running:
	Children see throughput for 32 rewriters 	=  433386.73 KB/sec
	Min throughput per thread 			=   13525.65 KB/sec 
	Max throughput per thread 			=   13553.97 KB/sec
	Avg throughput per thread 			=   13543.34 KB/sec
	Min xfer 					= 4118280.00 KB

	Test running:
	Children see throughput for 32 readers 		=  458043.86 KB/sec
	Min throughput per thread 			=   13969.76 KB/sec 
	Max throughput per thread 			=   14944.34 KB/sec
	Avg throughput per thread 			=   14313.87 KB/sec
	Min xfer 					= 3857648.00 KB

	Test running:
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Gluster users - Dec 2009 - gluster 3.0 read hangs

[Gluster-users] gluster 3.0 read hangs

[Gluster-users] gluster 3.0 read hangs

[Gluster-users] gluster 3.0 read hangs