Hi folks.
I'm posting this to both the Fedora as well as the CentOS lists in
hopes that somewhere, someone can help me figure out what's going on. I
have a dual Xeon 3GHz server that's performing rather slow when it comes
to disk activities.
The machine is configured with a single 160 GiB OS drive (with
CentOS 5.0) and 4x500 GiB drives setup in a RAID-5 configuration. All
drives are setup for 3.0 GiB SATA link, and the motherboard also
supports that. Looking in dmesg when the system comes up, I see that
reflected as well:
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-7, max UDMA/133, 312581808 sectors: LBA48 NCQ (depth 0/32)
ata1.00: configured for UDMA/133
scsi1 : ahci
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: configured for UDMA/133
scsi2 : ahci
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32)
ata3.00: configured for UDMA/133
scsi3 : ahci
ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata4.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/133
scsi4 : ahci
ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata5.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32)
ata5.00: configured for UDMA/133
scsi5 : ahci
Now, I don't know what performance numbers *should* be, but on a 1.8
GiB copy on the RAID (cp from one location to another on the RAID), it
gets done in just under 50 seconds. If I try to delete the folder
afterwards (rm -rf FOLDER) it takes a few seconds to do so, however if I
delete the CONTENTS of the folder, it does so within a fraction of a
second (but then 'sync' takes a few seconds to catch up.)
That same folder that I'm copying contains 452 jpeg files in it,
ranging from 2.5 to 6.2 MiB. Doing some image processing on them is
where it takes a long time. At the moment I'm doing a simple thumbnail
creation with the ImageMagick suite (convert FILE -thumbnail
"200x200>'
`basename FILE .jpg`.th.jpg) and it takes upwards of 8 minutes to
complete. The whole time it's running, 'top' reports the server
load as
follows: load average: 1.06, 1.00, 0.81 And the CPU usage is around 9%.
Interestingly, if I run the same command and have it create .png
instead, it takes longer, but I won't go there just yet. My question
is, is this the expected performance on something like this, or should I
be able to get better results? Is there something I should or could do
to speed up disk based processes?
Or is this something where it's more memory intensive and I need to
look at adding more (right now it has 2 GiB of memory.)
This problem is causing one of our web sites to time out because
it's trying to process hundreds of image files and generate thumbnails,
and it's taking forever to do that. So I'm starting at the bottom of
the pile here, hardware. If it turns out the hardware is fine, and
there's nothing else that can be done to speed it up, then I'll move
forward to other possible culprits, such as the routines within the site
scripts themselves...
--
W | It's not a bug - it's an undocumented feature.
+--------------------------------------------------------------------
Ashley M. Kirchner <mailto:ashley at pcraft.com> . 303.442.6410 x130
IT Director / SysAdmin / Websmith . 800.441.3873 x130
Photo Craft Imaging . 3550 Arapahoe Ave. #6
http://www.pcraft.com ..... . . . Boulder, CO 80303, U.S.A.
On Mon, April 30, 2007 1:58 pm, Ashley M. Kirchner wrote:> > Hi folks. > > I'm posting this to both the Fedora as well as the CentOS lists in > hopes that somewhere, someone can help me figure out what's going on. I > have a dual Xeon 3GHz server that's performing rather slow when it comes > to disk activities. > > The machine is configured with a single 160 GiB OS drive (with > CentOS 5.0) and 4x500 GiB drives setup in a RAID-5 configuration. All > drives are setup for 3.0 GiB SATA link, and the motherboard also > supports that. Looking in dmesg when the system comes up, I see that > reflected as well: > > Now, I don't know what performance numbers *should* be, but on a 1.8 > GiB copy on the RAID (cp from one location to another on the RAID), it > gets done in just under 50 seconds. If I try to delete the folder > afterwards (rm -rf FOLDER) it takes a few seconds to do so, however if I > delete the CONTENTS of the folder, it does so within a fraction of a > second (but then 'sync' takes a few seconds to catch up.) > > That same folder that I'm copying contains 452 jpeg files in it, > ranging from 2.5 to 6.2 MiB. Doing some image processing on them is > where it takes a long time. At the moment I'm doing a simple thumbnail > creation with the ImageMagick suite (convert FILE -thumbnail "200x200>' > `basename FILE .jpg`.th.jpg) and it takes upwards of 8 minutes to > complete. The whole time it's running, 'top' reports the server load as > follows: load average: 1.06, 1.00, 0.81 And the CPU usage is around 9%. > > Interestingly, if I run the same command and have it create .png > instead, it takes longer, but I won't go there just yet. My question > is, is this the expected performance on something like this, or should I > be able to get better results? Is there something I should or could do > to speed up disk based processes? > > Or is this something where it's more memory intensive and I need to > look at adding more (right now it has 2 GiB of memory.) > > > This problem is causing one of our web sites to time out because > it's trying to process hundreds of image files and generate thumbnails, > and it's taking forever to do that. So I'm starting at the bottom of > the pile here, hardware. If it turns out the hardware is fine, and > there's nothing else that can be done to speed it up, then I'll move > forward to other possible culprits, such as the routines within the site > scripts themselves...Try running "hdparm -Tt /dev/sda" replace /dev/sda with the device name for your hard disks. After running that post back here with the results. You can also install sysstat and use the sar command to see if your system is indeed spening a lot of time waiting on disk reads and writes. -- Matt Martz CentOS Mirror Admin mdmartz at gflug.net
On 4/30/07, Ashley M. Kirchner <ashley at pcraft.com> wrote:> I'm posting this to both the Fedora as well as the CentOS lists in > hopes that somewhere, someone can help me figure out what's going on. I > have a dual Xeon 3GHz server that's performing rather slow when it comes > to disk activities. > > The machine is configured with a single 160 GiB OS drive (with > CentOS 5.0) and 4x500 GiB drives setup in a RAID-5 configuration. All > drives are setup for 3.0 GiB SATA link, and the motherboard also > supports that. Looking in dmesg when the system comes up, I see that > reflected as well:What raid controller are you using? Have you read the disk tuning guide on the wiki (http://wiki.centos.org/HowTos/Disk_Optimization)? Also note that raid 5 is not the fastest option around. If you're really looking for speed, you should look at using raid 10. -- During times of universal deceit, telling the truth becomes a revolutionary act. George Orwell
Ashley M. Kirchner wrote:> > Hi folks. > > I'm posting this to both the Fedora as well as the CentOS lists in > hopes that somewhere, someone can help me figure out what's going on. I > have a dual Xeon 3GHz server that's performing rather slow when it comes > to disk activities. > > The machine is configured with a single 160 GiB OS drive (with CentOS > 5.0) and 4x500 GiB drives setup in a RAID-5 configuration. All drives > are setup for 3.0 GiB SATA link, and the motherboard also supports > that. Looking in dmesg when the system comes up, I see that reflected > as well: > > ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) > ata1.00: ATA-7, max UDMA/133, 312581808 sectors: LBA48 NCQ (depth 0/32) > ata1.00: configured for UDMA/133 > scsi1 : ahci > ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) > ata2.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32) > ata2.00: configured for UDMA/133 > scsi2 : ahci > ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) > ata3.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32) > ata3.00: configured for UDMA/133 > scsi3 : ahci > ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) > ata4.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32) > ata4.00: configured for UDMA/133 > scsi4 : ahci > ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) > ata5.00: ATA-7, max UDMA/133, 976773168 sectors: LBA48 NCQ (depth 0/32) > ata5.00: configured for UDMA/133 > scsi5 : ahci > > > Now, I don't know what performance numbers *should* be, but on a 1.8 > GiB copy on the RAID (cp from one location to another on the RAID), it > gets done in just under 50 seconds. If I try to delete the folder > afterwards (rm -rf FOLDER) it takes a few seconds to do so, however if I > delete the CONTENTS of the folder, it does so within a fraction of a > second (but then 'sync' takes a few seconds to catch up.) > > That same folder that I'm copying contains 452 jpeg files in it, > ranging from 2.5 to 6.2 MiB. Doing some image processing on them is > where it takes a long time. At the moment I'm doing a simple thumbnail > creation with the ImageMagick suite (convert FILE -thumbnail "200x200>' > `basename FILE .jpg`.th.jpg) and it takes upwards of 8 minutes to > complete. The whole time it's running, 'top' reports the server load as > follows: load average: 1.06, 1.00, 0.81 And the CPU usage is around 9%. > > Interestingly, if I run the same command and have it create .png > instead, it takes longer, but I won't go there just yet. My question > is, is this the expected performance on something like this, or should I > be able to get better results? Is there something I should or could do > to speed up disk based processes? > > Or is this something where it's more memory intensive and I need to > look at adding more (right now it has 2 GiB of memory.) > > > This problem is causing one of our web sites to time out because it's > trying to process hundreds of image files and generate thumbnails, and > it's taking forever to do that. So I'm starting at the bottom of the > pile here, hardware. If it turns out the hardware is fine, and there's > nothing else that can be done to speed it up, then I'll move forward to > other possible culprits, such as the routines within the site scripts > themselves...At first I would suggest against cross-posting. The two OSes are not exactly the same. Also you should specify your exact OS. Regarding the 2 CPUs, do you have the service irqbalance running? Also if not running CentOS 5.0, have you installed the SMP kernel? Regarding parallel processing, I do not know much about GNU/Linux parallelization, however I assume a given script is running on one CPU of the two. Obviously there are others here with more insight about this than me.