This is due to the NFS read caching. Use io-cache to have the
equivalent of the functionality in glusterfs enabled.
Avati
On Wed, Apr 1, 2009 at 6:31 PM, Greg <greg at easyflirt.com>
wrote:> Hi,
>
> I have a production setup, with clients that are both NFS and GlusterFS
> clients. I've made a _very_ quick benchmark (repeated ten times), to
compare
> read performance between local hardrives, a NFS mount and a GlusterFS
mount.
> - a 17k file :
> cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> - a 50M file : users0.new.spi
> - /nfs is a NFS mounted point, server with SATA drives
> - /gluster is a GlusterFS mounted point, 2x servers with SAS drives
>
> php-14:~# for f in
>
/var/www/www/cache/cache_c/cache_cf/cache_cfb/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
/nfs/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> /gluster/sphinx/users0.new.spi /nfs/tmp/users0.new.spi /tmp/users0.new.spi;
> do echo $f; time cat $f >/dev/null; done
>
/var/www/www/cache/cache_c/cache_cf/cache_cfb/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real ? ?0m0.011s
> user ? ?0m0.000s
> sys ? ? 0m0.000s
>
/nfs/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real ? ?0m0.002s
> user ? ?0m0.000s
> sys ? ? 0m0.000s
>
/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real ? ?0m0.001s
> user ? ?0m0.000s
> sys ? ? 0m0.000s
> /gluster/sphinx/users0.new.spi
>
> real ? ?0m0.081s
> user ? ?0m0.000s
> sys ? ? 0m0.068s
> /nfs/tmp/users0.new.spi
>
> real ? ?0m0.037s
> user ? ?0m0.000s
> sys ? ? 0m0.036s
> /tmp/users0.new.spi
>
> real ? ?0m0.028s
> user ? ?0m0.000s
> sys ? ? 0m0.028s
>
>
> Configuration:
> # file: /etc/glusterfs/glusterfsd.vol
>
> #
> # Volumes
> #
> volume media-small
> ? ? ? type storage/posix
> ? ? ? option directory /var/local/glusterfs/media_small
> end-volume
>
> volume media-medium
> ? ? ? type storage/posix
> ? ? ? option directory /var/local/glusterfs/media_medium
> end-volume
>
> # Lock posix
> volume media-small-locks
> ?type features/posix-locks
> ?option mandatory-locks on
> ?subvolumes media-small
> # subvolumes trash # enable this if you need trash can support (NOTE: not
> present in 1.3.0-pre5+ releases)
> end-volume
>
> volume media-medium-locks
> ?type features/posix-locks
> ?option mandatory-locks on
> ?subvolumes media-medium
> # subvolumes trash # enable this if you need trash can support (NOTE: not
> present in 1.3.0-pre5+ releases)
> end-volume
>
>
> #
> # Performance
> #
> volume media-small-iot
> type performance/io-threads
> subvolumes media-small-locks
> option thread-count 4 # default value is 1
> end-volume
>
> volume media-small-wb
> type performance/write-behind
> option flush-behind on # default is off
> subvolumes media-small-iot
> end-volume
>
> volume media-medium-iot
> type performance/io-threads
> subvolumes media-medium-locks
> option thread-count 4 # default value is 1
> end-volume
>
> volume media-medium-wb
> type performance/write-behind
> option flush-behind on # default is off
> subvolumes media-medium-iot
> end-volume
>
>
>
> #
> # Serveur
> #
> volume server
> ? ? ? type protocol/server
> ? ? ? option transport-type tcp/server
> ? ? ? option auth.addr.media-small-wb.allow 10.0.*.*
> ? ? ? option auth.addr.media-medium-wb.allow 10.0.*.*
> ? ? ? # Autoconfiguration, e.g. :
> ? ? ? # glusterfs -l /tmp/glusterfs.log --server=filer-04 ./Cache
> ? ? ? option client-volume-filename /etc/glusterfs/glusterfs.vol
> ? ? ? subvolumes media-small-wb media-medium-wb # volumes export?s
> end-volume
>
> # file: /etc/glusterfs/glusterfs.vol
>
>
> #
> # Clients
> #
> volume media-small-filer-04
> ? ? ? type protocol/client
> ? ? ? option transport-type tcp/client
> ? ? ? option remote-host filer-04.local
> ? ? ? option remote-subvolume media-small-wb
> end-volume
>
> volume media-small-filer-05
> ? ? ? type protocol/client
> ? ? ? option transport-type tcp/client
> ? ? ? option remote-host filer-05.local
> ? ? ? option remote-subvolume media-small-wb
> end-volume
>
> volume media-medium-filer-04
> ? ? ? type protocol/client
> ? ? ? option transport-type tcp/client
> ? ? ? option remote-host filer-04.local
> ? ? ? option remote-subvolume media-medium-wb
> end-volume
>
> volume media-medium-filer-05
> ? ? ? type protocol/client
> ? ? ? option transport-type tcp/client
> ? ? ? option remote-host filer-05.local
> ? ? ? option remote-subvolume media-medium-wb
> end-volume
>
>
> #
> # Volume principal
> #
> volume afr-small
> ? ? ? # AFR has been renamed to "Replicate" for simplicity.
> ? ? ? type cluster/replicate
> ? ? ? # Il faut mettre le serveur avec le moins d'espace disque en 1er
:
> ? ? ? # "When doing a "df -h" on a client, the AVAILABLE
disk space will
> display the maximum disk space of the first AFR sub volume defined in the
> spec file. So if you have two servers with 50 gigs and 100 gigs of free
disk
> space, and the server with 100 gigs is listed first, then you will see 100
> gigs available even though one server only has 50 gigs free. "
> ? ? ? subvolumes media-small-filer-04 media-small-filer-05
> end-volume
>
> volume afr-medium
> ? ? ? # AFR has been renamed to "Replicate" for simplicity.
> ? ? ? type cluster/replicate
> ? ? ? subvolumes media-medium-filer-04 media-medium-filer-05
> end-volume
>
> #
> # Performance
> #
> volume iot-small
> ? ? ? type performance/io-threads
> ? ? ? subvolumes afr-small
> end-volume
>
> volume readahead-small
> ? ? ? type performance/read-ahead
> ? ? ? subvolumes iot-small
> ? ? ? option page-size 1MB ? ? ? ? # default is 256KB
> ? ? ? option page-count 4 ? ? ? ? ?# default is 2 - cache per file >
(page-count x page-size)
> ? ? ? option force-atime-update no # defalut is 'no'
> end-volume
>
> volume iocache-small
> ? ? ? type performance/io-cache
> ? ? ? option cache-size 64MB ? ? ? ? ?# default is 32MB
> ? ? ? option page-size 256KB ? ? ? ? ?# default is 128KB
> ? ? ? subvolumes readahead-small
> end-volume
>
> volume writeback-small
> ? ? ? type performance/write-behind
> ? ? ? option block-size 1MB # max 4MB
> ? ? ? option flush-behind on # default is off
> ? ? ? subvolumes iocache-small
> end-volume
>
> volume iot-medium
> ? ? ? type performance/io-threads
> ? ? ? subvolumes afr-medium
> end-volume
>
> volume readahead-medium
> ? ? ? type performance/read-ahead
> ? ? ? subvolumes iot-medium
> ? ? ? option page-size 1MB ? ? ? ? # default is 256KB
> ? ? ? option page-count 4 ? ? ? ? ?# default is 2 - cache per file >
(page-count x page-size)
> ? ? ? option force-atime-update no # defalut is 'no'
> end-volume
>
> volume iocache-medium
> ? ? ? type performance/io-cache
> ? ? ? option cache-size 64MB ? ? ? ? ?# default is 32MB
> ? ? ? option page-size 256KB ? ? ? ? ?# default is 128KB
> ? ? ? option page-count 2
> ? ? ? subvolumes readahead-medium
> end-volume
>
> volume writeback-medium
> ? ? ? type performance/write-behind
> ? ? ? option block-size 1MB # max 4MB
> ? ? ? option flush-behind on # default is off
> ? ? ? subvolumes iocache-medium
> end-volume
>
> Regards,
>
> --
> Greg
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>