zfs discuss - Dec 2008 - ZFS and AVS replication performance issues

Hi,

I have setup AVS replication between two zvols on two opensolaris-2008.11
nodes. I have been seeing BIG performance issues, so I tried to setup the
system to be as fast as possible using a couple of tricks. The detailed
setup and performance data are below:

    *  A 100G zvol has been setup on each node of an AVS replicating pair
    * A "ramdisk" has been setup on each node using the following
command.
This functions as a very fast logging disk!

      ramdiskadm -a ram1 10m

    * The replication relationship has been setup using

      sndradm -E pri /dev/zvol/rdsk/gold/myzvol /dev/rramdisk/ram1 sec
/dev/zvol/rdsk/gold/myzvol /dev/rramdisk/ram1 ip async

    * The AVS driver was configured to *not* log the disk bitmap to disk,
rather to keep it in kernel memory and write it to disk only upon machine
shutdown. This is configured as such

      # grep bitmap_mode /usr/kernel/drv/rdc.conf
      rdc_bitmap_mode=2;

    * The replication was configured to be in logging mode (To avoid any
possible network bottlenecks)

      #sndradm -P
      /dev/zvol/rdsk/gold/myzvol      <-      pri:/dev/zvol/rdsk/gold/myzvol
      autosync: off, max q writes: 4096, max q fbas: 16384, async threads:
2, mode: async, state: logging

=== Testing ==
All tests were performed using the following command line

# dd if=/dev/zero of=/dev/zvol/rdsk/gold/xxVolNamexx oflag=dsync
bs=256M count=10

I usually ran a couple of runs initially to avoid caching effects.

=== Results ==
The results

The following Results were reported after initial couple of runs to avoid
cache effects

Run#    dd count=N     Native Vol Throughput               Replicated Vol
Throughput (logging mode)
1         4                     42.2 MB/s
   4.9 MB/s
2         4                     52.8 MB/s
        5.5 MB/s
3         10                    50.9 MB/s
        4.6 MB/s

As you can see the performance is almost 10 times slower!! Although no disk
logging is done to disk (only ram disk if not driver memory), and no network
traffic ! Seems to me that the AVS kernel drivers slow the system A LOT for
simply hooking any write operation and flipping a kernel memory bit per 32k
written disk space?!!

Any suggestions as to why this is happening is most appreciated
Best Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081221/ec3dbd6b/attachment.html>