Is there anyway to see how busy a SAS/Sata controller is vs disks? I have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid 10 ZFS. When firing off a dd I (bs=1G count=10) seems like the disks never go above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW or if its something else (like terrible dd options). my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a dedicated 8 drive board inside the front of the R720. Basically i'm just saying its not a single SAS cable to 14 drives. Don't have cpu info hand.. zeon something. DDR3-1600 (128GB) Both controllers are in 8x slots running PCIe gen 3. BTW i'm sure this has been asked a million times but what would be some decent benchmark tests while i'm at it?
On 9/24/19 8:45 AM, John Fleming wrote:> Is there anyway to see how busy a SAS/Sata controller is vs disks? I > have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid > 10 ZFS. > > When firing off a dd I (bs=1G count=10) seems like the disks never go > above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW > or if its something else (like terrible dd options). > > my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a > dedicated 8 drive board inside the front of the R720. Basically i'm > just saying its not a single SAS cable to 14 drives. > > Don't have cpu info hand.. zeon something. DDR3-1600 (128GB) > > Both controllers are in 8x slots running PCIe gen 3.might want to take a look at sysutils/intel-pcm (https://github.com/opcm/pcm).? I *think* this should give you metrics on PCIe bus utilization among other useful status. Also, lookup the bandwidth for the PCIe bus and see if your aggregate disk throughput on one of the PCIe lanes is saturating the bus (pcm should also help here).? You can also run "zpool iostat -v 2" to see per disk i/o metrics to help determine if this is an issue.> BTW i'm sure this has been asked a million times but what would be > some decent benchmark tests while i'm at it?I generally run several tests and then compare results, for example bonnie++, iozone, iperf (writing over the wire and to disk) as well as some more realistic scripts based on the use-case i'm building a solution for.? hope that helps. -pete -- Pete Wright pete at nomadlogic.org @nomadlogicLA
On Tue, Sep 24, 2019 at 5:46 PM John Fleming <john at spikefishsolutions.com> wrote:> Is there anyway to see how busy a SAS/Sata controller is vs disks? I > have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid > 10 ZFS. > > When firing off a dd I (bs=1G count=10) seems like the disks never go > above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW > or if its something else (like terrible dd options). >Two points to consider here. First, NVMe has lots of queues and needs lots of concurrent transactions to saturate, so the 50% busy means you are no where close to saturating the drives. Schedule more I/O too fix that. It's better to do lots and lots of concurrent DD to different parts of the drive, or to use fio with the aio kernel option and posixaio I/O scheduling method. I use the following script, but often need to increase the number of threads / jobs to saturate. ; SSD testing: 128k I/O 64 jobs 32 deep queue [global] direct=1 rw=randread refill_buffers norandommap randrepeat=0 bs=128k ioengine=posixaio iodepth=32 numjobs=64 runtime=60 group_reporting thread [ssd128k] Second, the system's % busy statistics are misleading. They are the %of the time that a command is outstanding on the drive. 100% busy can be a tiny percentage of the total bandwidth you can get from the drive.> my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a > dedicated 8 drive board inside the front of the R720. Basically i'm > just saying its not a single SAS cable to 14 drives. > > Don't have cpu info hand.. zeon something. DDR3-1600 (128GB) > > Both controllers are in 8x slots running PCIe gen 3. > > BTW i'm sure this has been asked a million times but what would be > some decent benchmark tests while i'm at it? >See above... :) Warner