Hi I was wondering if you ever got better performance out of your Gigabit/IDE/Fc2? I am facing a similar situation. I am running FC2 with Samba 3.x My problem lies in not that I am limited to 10 MBytes per second sustained. I think it's related to this pdflush and how it's buffers are setup. (I have been doing some research and before 2.6 kernels bdflush was the method that was used and it was tweakable. Have yet to find anything on HOW to tweak pdflush. My issue is that I can copy over the network at 15+ MBytes per second but then after a few minutes it will drop to 4-8 MBytes per second. Yet a drive to drive copy on the linux box itself can sustain 14+ MBytes per second on the same size file. (4+ gigs) While I watch a Top when the network copying is going on PDFlush is bouncing around with Samba, not sure what is going on. Was curious if you had improved your situation, and if you did what you did. BTW here are some tweaks for network stack related stuff for Gig. sysctl -w net.core.rmem_max=8388608 sysctl -w net.core.wmem_max=8388608 sysctl -w net.core.rmem_default=65536 sysctl -w net.core.wmem_default=65536 sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608' sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608' sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608' sysctl -w net.ipv4.route.flush=1 Brian M. Duncan Katten Muchin Rosenman LLP 525 West Monroe Street Chicago IL 60661-3693 312-577-8045 brian.duncan@kattenlaw.com ========================================================== Important: This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies. NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability partnership that has elected to be governed by the Illinois Uniform Partnership Act (1997). ===========================================================
Greetings, Can you post more info on your setup, i.e - configuration of machine, etc.... I am also looking into Linux network stack tweaks. I have seen my tweaks posted for GigE, but you have to be careful. You may actually get a lower throughput. For example: sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608' This tell your kernel it can use 8388608 pages ( a page is 4K on the Intel platform) for all your TCP currently in use. so (8388608 * 4096)/1048576 = 32768MB Take a look at: http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html On Fri, May 13, 2005 at 12:06:29PM -0500, Duncan, Brian M. wrote:> > Hi I was wondering if you ever got better performance out of your > Gigabit/IDE/Fc2? > > I am facing a similar situation. I am running FC2 with Samba 3.x > > My problem lies in not that I am limited to 10 MBytes per second > sustained. I think it's related to this pdflush and how it's buffers > are setup. (I have been doing some research and before 2.6 kernels > bdflush was the method that was used and it was tweakable. Have yet to > find anything on HOW to tweak pdflush. > > My issue is that I can copy over the network at 15+ MBytes per second > but then after a few minutes it will drop to 4-8 MBytes per second. Yet > a drive to drive copy on the linux box itself can sustain 14+ MBytes per > second on the same size file. (4+ gigs) While I watch a Top when the > network copying is going on PDFlush is bouncing around with Samba, not > sure what is going on. > > Was curious if you had improved your situation, and if you did what you > did. > > BTW here are some tweaks for network stack related stuff for Gig. > > sysctl -w net.core.rmem_max=8388608 > sysctl -w net.core.wmem_max=8388608 > sysctl -w net.core.rmem_default=65536 > sysctl -w net.core.wmem_default=65536 > sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608' > sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608' > sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608' > sysctl -w net.ipv4.route.flush=1 > > > > Brian M. Duncan > Katten Muchin Rosenman LLP > 525 West Monroe Street > Chicago IL 60661-3693 > 312-577-8045 > > brian.duncan@kattenlaw.com > > > > > > ==========================================================> > Important: > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, privileged, > confidential and/or exempt from disclosure under applicable law. If you > are not the intended recipient, you are hereby notified that any viewing, > copying, disclosure or distribution of this information may be subject to > legal restriction or sanction. Please notify the sender, by electronic > mail or telephone, of any unintended recipients and delete the original > message without making any copies. > > NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability > partnership that has elected to be governed by the Illinois Uniform > Partnership Act (1997). > > ==========================================================> -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/listinfo/samba
512 Megs of ram in server
Pentium 4 2.0 Ghz
2 Ultra 133 controllers - 8 drives total - all 300 gig drives running at
max UDMA support with write back cache turned off on each drive.
(Clients connected are all at 1 Gig full duplex)
FC2, with Samba 3.0.10-1
Any tweaks I try I test before and after and have only left in place
what tweaks seemed to improve performance.
I am just running into a wall with Linux's manner that it handles
caching of the files I think before it writes them to disk. I have seen
my transfers start out as high as 50 Megabytes per second, but then they
slowly go down (seen it go as low as 1 Megabyte per sec) My guess is if
I added more memory to the server that time for it to slow down would be
increased a bit. (I was going to confirm that this weekend)
I just verified since this entire time I have been ONLY focused on
copying to the Linux server. My numbers all level off to almost exactly
10 MegaBytes per sec if I am copying FROM the linux box down to a
workstation. (Test copy of 4 gig file, starts off at 30-40 - till XP I
assume fills its buffers and starts spooling to disk, then drops and
holds at 10)
Also in your case if you NEVER see your rates change. (They just stay at
around 10Mbytes per sec I recall?) Do you have the ability to try jumbo
frames? >1500 MTU on all devices? My current switch does not support
jumbo frames so I can't.
I believe it's tied to this pdflush. (My issue)
I found this regarding bdflush, and I think it would help me if I could
tweak pdflush:
Hi all,
I recently switched to FC2 with the CCRMA 2.6.8 kernel. I was
wondering if there is a way to tune pdflush in the 2.6 kernel like you
could bdflush in the 2.4 kernel? I have seen advice on this list to set
bdflush by adding *"echo* "10 64 0 0 2000 15000 90 5 0" >
/proc/sys/vm/*bdflush" to the rc.local file, thus causing bdflush to
write smaller chunks more often to disk instead of waiting for the
"dirty buffer" to accumulate alot of stuff to write to disk all at
once,
possibly causing dropouts. Is there a way to do this with pdflush or is
it even necessary with the 2.6 kernel? Thanks in advance.
-----Original Message-----
From: Paul Griffith [mailto:paulg@cs.yorku.ca]
Sent: Friday, May 13, 2005 12:23 PM
To: Duncan, Brian M.
Cc: samba@lists.samba.org
Subject: Re: [Samba] Gigabit Throughput too low
Greetings,
Can you post more info on your setup, i.e - configuration of machine,
etc....
I am also looking into Linux network stack tweaks. I have seen my tweaks
posted for GigE, but you have to be careful. You may actually get a
lower throughput.
For example:
sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
This tell your kernel it can use
8388608 pages ( a page is 4K on the Intel platform) for all your TCP
currently in use.
so (8388608 * 4096)/1048576 = 32768MB
Take a look at:
http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html
On Fri, May 13, 2005 at 12:06:29PM -0500, Duncan, Brian M.
wrote:>
> Hi I was wondering if you ever got better performance out of your
> Gigabit/IDE/Fc2?
>
> I am facing a similar situation. I am running FC2 with Samba 3.x
>
> My problem lies in not that I am limited to 10 MBytes per second
> sustained. I think it's related to this pdflush and how it's
buffers
> are setup. (I have been doing some research and before 2.6 kernels
> bdflush was the method that was used and it was tweakable. Have yet
> to find anything on HOW to tweak pdflush.
>
> My issue is that I can copy over the network at 15+ MBytes per second
> but then after a few minutes it will drop to 4-8 MBytes per second.
> Yet a drive to drive copy on the linux box itself can sustain 14+
> MBytes per second on the same size file. (4+ gigs) While I watch a Top
> when the network copying is going on PDFlush is bouncing around with
> Samba, not sure what is going on.
>
> Was curious if you had improved your situation, and if you did what
> you did.
>
> BTW here are some tweaks for network stack related stuff for Gig.
>
> sysctl -w net.core.rmem_max=8388608
> sysctl -w net.core.wmem_max=8388608
> sysctl -w net.core.rmem_default=65536
> sysctl -w net.core.wmem_default=65536
> sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'
> sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'
> sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
> sysctl -w net.ipv4.route.flush=1
>
>
>
> Brian M. Duncan
> Katten Muchin Rosenman LLP
> 525 West Monroe Street
> Chicago IL 60661-3693
> 312-577-8045
>
> brian.duncan@kattenlaw.com
>
>
>
>
>
> ==========================================================>
> Important:
> This electronic mail message and any attached files contain
> information intended for the exclusive use of the individual or entity
> to whom it is addressed and may contain information that is
> proprietary, privileged, confidential and/or exempt from disclosure
> under applicable law. If you are not the intended recipient, you are
> hereby notified that any viewing, copying, disclosure or distribution
> of this information may be subject to legal restriction or sanction.
> Please notify the sender, by electronic mail or telephone, of any
> unintended recipients and delete the original message without making
> any copies.
>
> NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited
> liability partnership that has elected to be governed by the Illinois
> Uniform Partnership Act (1997).
>
> ==========================================================> --
> To unsubscribe from this list go to the following URL and read the
> instructions: https://lists.samba.org/mailman/listinfo/samba
==========================================================
Important:
This electronic mail message and any attached files contain information
intended for the exclusive use of the individual or entity to whom it is
addressed and may contain information that is proprietary, privileged,
confidential and/or exempt from disclosure under applicable law. If you
are not the intended recipient, you are hereby notified that any viewing,
copying, disclosure or distribution of this information may be subject to
legal restriction or sanction. Please notify the sender, by electronic
mail or telephone, of any unintended recipients and delete the original
message without making any copies.
NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability
partnership that has elected to be governed by the Illinois Uniform
Partnership Act (1997).
===========================================================
Oops.. did not realize I was responding on the list before. Excuse me, I am not even sure this belongs here. I know I would increase my possible rates going raid, I am using JFS BTW. This is not a production box in a company. Its actually my video server at home, that normally has 1 or 2 clients hitting it at a time. (I need (or want) max bandwidth at those times) What I am trying to obtain is speed across the wire - comparable to what I can get copying from 1 drive on 1 channel to another drive on another channel from the server (through the whole transfer) over the wire Which I currently cannot obtain.) If I can currently take a 4 gig file on drive 1 controller one, and copy it to drive 1 on controller 2 at a constant 20 Megabytes per second locally. You would think I should be able to copy a file from the 1 Gigabit network to drive 1 controller 2 at at least the same speed? (The client is capable of delivering the file at the speed) I am just trying to figure out in my Linux config what exactly is causing this bottleneck for me. I don't know if it's Samba, I am starting to think it's a caching issue. (Dirty cache) BTW thanks everyone for the feedback. -----Original Message----- From: Greg Freemyer [mailto:greg.freemyer@gmail.com] Sent: Friday, May 13, 2005 2:53 PM To: Duncan, Brian M. Cc: Paul Griffith; samba@lists.samba.org Subject: Re: [Samba] Gigabit Throughput too low On 5/13/05, Duncan, Brian M. <brian.duncan@kattenlaw.com> wrote:> > 512 Megs of ram in server > Pentium 4 2.0 Ghz > 2 Ultra 133 controllers - 8 drives total - all 300 gig drives running > at max UDMA support with write back cache turned off on each drive. > (Clients connected are all at 1 Gig full duplex) > > FC2, with Samba 3.0.10-1 > > Any tweaks I try I test before and after and have only left in place > what tweaks seemed to improve performance. > > I am just running into a wall with Linux's manner that it handles > caching of the files I think before it writes them to disk. I have > seen my transfers start out as high as 50 Megabytes per second, but > then they slowly go down (seen it go as low as 1 Megabyte per sec) My > guess is if I added more memory to the server that time for it to slow> down would be increased a bit. (I was going to confirm that this > weekend) >You do know that 10 MB/sec is not horrible for what you describe above. ie. You have a very low cost ide-controller structure. You have multiple drives per ide channel (in use at the same time? I hope not due to master/slave contention). You don't describe any raid. raid-10 is typically the fastest way to go, but uses more drives. A good 8-drive raid-10 is theoretically 4x faster than no raid on writes and 8x faster on reads. (Admittedly, that is only in theory, but it should still be faster.) You don't mention the filesystem, but I'm guessing ext3, which is also not a great speed daemon. I'm guessing you have the default journaling setup. Asssuming a journalling FS, you want to put the journal on some dedicated spindles, not the same drives as the FS. Basically, I would optimize your disk-subsystem speed before I started worrying about your 1Gb/sec. LAN. Personally, I would consider 3ware parallel IDE controllers, raid 5 at a minimum (raid 10 if you can), xfs filesystem, dedicated journal drive. 3ware has a white-paper describing a high-performance Linux setup. You might want to look at it. With a $1000 3ware controller and lots of reconfiguration, you can probably get your disk sub-system up to 100MB/sec with no probem. Even 200 or 300 should be achievable. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century ========================================================== Important: This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies. NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability partnership that has elected to be governed by the Illinois Uniform Partnership Act (1997). ===========================================================
Following up to my own issues, I have determined that it def seems like Samba is some how introducing my issues. For the heck of it I tried FTPing tonight to this same box that I described my issues with below. Transferred 4, 4 gig files and managed to sustain somewhere between 15-21 Megabytes per second. Never dropped to a crawl like it does copying the files via Samba. There were some pauses while the server must have been clearing it's dirty cache, etc.. But very small, not 6-8 seconds I would see with Samba, with FTP it would drop from 20 MegaBytes per sec to like 16 then go right back up to 18 then 20 in a second.. So I guess I will look further into tweaking Samba. After finding this out, I searched the lists and I see there are others that can't get Samba to work at it's peak efficiency. Very similar reports to what I have seen. (unexplainable pauses in LARGE file copies, people blaming their network settings, DNS settings, hubs, etc) If there is anyone out there that has Samba running on a Linux box that has a config already tweaked that has NO issues copying large (2+ gig files) over their SMB shares with no long pauses and it compares to what they can get FTP'ing to the same box, can you please email it to me? Thanks -----Original Message----- From: Duncan, Brian M. Sent: Friday, May 13, 2005 3:11 PM To: 'Greg Freemyer' Cc: samba@lists.samba.org Subject: RE: [Samba] Gigabit Throughput too low Oops.. did not realize I was responding on the list before. Excuse me, I am not even sure this belongs here. I know I would increase my possible rates going raid, I am using JFS BTW. This is not a production box in a company. Its actually my video server at home, that normally has 1 or 2 clients hitting it at a time. (I need (or want) max bandwidth at those times) What I am trying to obtain is speed across the wire - comparable to what I can get copying from 1 drive on 1 channel to another drive on another channel from the server (through the whole transfer) over the wire Which I currently cannot obtain.) If I can currently take a 4 gig file on drive 1 controller one, and copy it to drive 1 on controller 2 at a constant 20 Megabytes per second locally. You would think I should be able to copy a file from the 1 Gigabit network to drive 1 controller 2 at at least the same speed? (The client is capable of delivering the file at the speed) I am just trying to figure out in my Linux config what exactly is causing this bottleneck for me. I don't know if it's Samba, I am starting to think it's a caching issue. (Dirty cache) BTW thanks everyone for the feedback. -----Original Message----- From: Greg Freemyer [mailto:greg.freemyer@gmail.com] Sent: Friday, May 13, 2005 2:53 PM To: Duncan, Brian M. Cc: Paul Griffith; samba@lists.samba.org Subject: Re: [Samba] Gigabit Throughput too low On 5/13/05, Duncan, Brian M. <brian.duncan@kattenlaw.com> wrote:> > 512 Megs of ram in server > Pentium 4 2.0 Ghz > 2 Ultra 133 controllers - 8 drives total - all 300 gig drives running > at max UDMA support with write back cache turned off on each drive. > (Clients connected are all at 1 Gig full duplex) > > FC2, with Samba 3.0.10-1 > > Any tweaks I try I test before and after and have only left in place > what tweaks seemed to improve performance. > > I am just running into a wall with Linux's manner that it handles > caching of the files I think before it writes them to disk. I have > seen my transfers start out as high as 50 Megabytes per second, but > then they slowly go down (seen it go as low as 1 Megabyte per sec) My > guess is if I added more memory to the server that time for it to slow> down would be increased a bit. (I was going to confirm that this > weekend) >You do know that 10 MB/sec is not horrible for what you describe above. ie. You have a very low cost ide-controller structure. You have multiple drives per ide channel (in use at the same time? I hope not due to master/slave contention). You don't describe any raid. raid-10 is typically the fastest way to go, but uses more drives. A good 8-drive raid-10 is theoretically 4x faster than no raid on writes and 8x faster on reads. (Admittedly, that is only in theory, but it should still be faster.) You don't mention the filesystem, but I'm guessing ext3, which is also not a great speed daemon. I'm guessing you have the default journaling setup. Asssuming a journalling FS, you want to put the journal on some dedicated spindles, not the same drives as the FS. Basically, I would optimize your disk-subsystem speed before I started worrying about your 1Gb/sec. LAN. Personally, I would consider 3ware parallel IDE controllers, raid 5 at a minimum (raid 10 if you can), xfs filesystem, dedicated journal drive. 3ware has a white-paper describing a high-performance Linux setup. You might want to look at it. With a $1000 3ware controller and lots of reconfiguration, you can probably get your disk sub-system up to 100MB/sec with no probem. Even 200 or 300 should be achievable. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century ========================================================== Important: This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies. NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability partnership that has elected to be governed by the Illinois Uniform Partnership Act (1997). ===========================================================