I am trying to set up a HA type system involving two identical boxes and
have gone through the following to set up the systems:
Slave server:
ggated -R 196608 -S 196608
(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072
Master server:
ggatec create -u 0 -R 196608 -S 196608 -o rw [slaveip] /dev/amrd1
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 65536
#gmirror status
Name Status Components
mirror/gm0 COMPLETE amrd1s1
ggate0s1
the two servers are connected to each other via their 2ndary physical
gigE interfaces using cat6 crossover cable. (Netperf shows 890 Mbps at
95% confidence).
softupdates are enable on gm0 (though this does not affect the results).
The results:
/usr/bin/time -h cp testfile64M /data1
28.62s real 0.00s user 0.16s sys
and this is very consistent ... about 3 MB/s over repeated runs
dd if=/dev/zero of=/data1/testfile32M2 bs=32k count=1024
1024+0 records in
1024+0 records out
33554432 bytes transferred in 16.122641 secs (2081199 bytes/sec)
What else can I tune here to make this functional? If I increase
recvspace and sendspace much beyond those numbers, ggated will not start
claiming to not have enough buffer space.
Sven
On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:> I am trying to set up a HA type system involving two identical boxes and > have gone through the following to set up the systems: > > Slave server: > ggated -R 196608 -S 196608 > (exporting /dev/amrd1 ) > net.inet.tcp.sendspace: 65536 > net.inet.tcp.recvspace: 131072Try net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 Also, try increase this sysctls with net.inet.tcp.rfc1323=1 I use it on FreeBSD 5.x with: net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 ggated -R 1048576 -S 1048576 ggatec -R 1048576 -S 1048576 WBR. Dmitriy
Dmitriy Kirhlarov wrote:> On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: >> I am trying to set up a HA type system involving two identical boxes and >> have gone through the following to set up the systems: >> >> Slave server: >> ggated -R 196608 -S 196608 >> (exporting /dev/amrd1 ) >> net.inet.tcp.sendspace: 65536 >> net.inet.tcp.recvspace: 131072 > > Try > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > > Also, try increase this sysctls with > net.inet.tcp.rfc1323=1 > > I use it on FreeBSD 5.x with: > net.inet.tcp.sendspace=131072 > net.inet.tcp.recvspace=131072 > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > > ggated -R 1048576 -S 1048576 > ggatec -R 1048576 -S 1048576 > > WBR. > Dmitriy > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"I have seen sustained writes of 30Mb/s using the following configuration: cat /boot/loader.conf kern.ipc.nmbclusters="32768" cat /etc/sysctl.conf net.inet.tcp.sendspace=1048576 net.inet.tcp.recvspace=1048576 Server: /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports Client: /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 /dev/amrd0s2 The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with adaptive read ahead and write back caching. Tom
On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:> Dmitriy Kirhlarov wrote: > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > >> I am trying to set up a HA type system involving two identical boxes and > >> have gone through the following to set up the systems: > >> > >> Slave server: > >> ggated -R 196608 -S 196608 > >> (exporting /dev/amrd1 ) > >> net.inet.tcp.sendspace: 65536 > >> net.inet.tcp.recvspace: 131072 > > > > Try > > net.local.stream.recvspace=65535 > > net.local.stream.sendspace=65535 > > > > Also, try increase this sysctls with > > net.inet.tcp.rfc1323=1 > > > > I use it on FreeBSD 5.x with: > > net.inet.tcp.sendspace=131072 > > net.inet.tcp.recvspace=131072 > > net.local.stream.recvspace=65535 > > net.local.stream.sendspace=65535 > > > > ggated -R 1048576 -S 1048576 > > ggatec -R 1048576 -S 1048576 > > > > WBR. > > Dmitriy > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > I have seen sustained writes of 30Mb/s using the following configuration: > > cat /boot/loader.conf > kern.ipc.nmbclusters="32768" > > cat /etc/sysctl.conf > net.inet.tcp.sendspace=1048576 > net.inet.tcp.recvspace=1048576 > > Server: > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > Client: > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > /dev/amrd0s2 > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > adaptive read ahead and write back caching. > > TomI have tried both the settings ideas suggested above but I cannot even get out of the gate with those. Setting net.inet.tcp.{send,recv}space to anything higher that 131072 results in ggated bailing with the error: # ggated -v -a 10.10.0.19 info: Reading exports file (/etc/gg.exports). debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. info: Exporting 2 object(s). error: Cannot open stream socket: No buffer space available. error: Exiting. setting net.inet.tcp.{send,recv}space to 131072 allows me to start ggated with the default R and S values of 131072; anything higher results in "no buffer space" errors. At 131072 ggated starts but then I cannot even open a new connection (like ssh) to the server as the ssh client bails with "no buffer space available". more information: # netstat -m 514/641/1155 mbufs in use (current/cache/total) 512/284/796/32768 mbuf clusters in use (current/cache/total/max) 512/256 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1152K/728K/1880K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA Raid using LSiMegaRaid. The odd thing is that even after I set the send and recvspace down to values like 65536, I continue to get the no buffer error when trying to connect to it remotely again. Sven
Sven Willenberger wrote:> On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: >> Dmitriy Kirhlarov wrote: >>> On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: >>>> I am trying to set up a HA type system involving two identical boxes and >>>> have gone through the following to set up the systems: >>>> >>>> Slave server: >>>> ggated -R 196608 -S 196608 >>>> (exporting /dev/amrd1 ) >>>> net.inet.tcp.sendspace: 65536 >>>> net.inet.tcp.recvspace: 131072 >>> Try >>> net.local.stream.recvspace=65535 >>> net.local.stream.sendspace=65535 >>> >>> Also, try increase this sysctls with >>> net.inet.tcp.rfc1323=1 >>> >>> I use it on FreeBSD 5.x with: >>> net.inet.tcp.sendspace=131072 >>> net.inet.tcp.recvspace=131072 >>> net.local.stream.recvspace=65535 >>> net.local.stream.sendspace=65535 >>> >>> ggated -R 1048576 -S 1048576 >>> ggatec -R 1048576 -S 1048576 >>> >>> WBR. >>> Dmitriy >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >> I have seen sustained writes of 30Mb/s using the following configuration: >> >> cat /boot/loader.conf >> kern.ipc.nmbclusters="32768" >> >> cat /etc/sysctl.conf >> net.inet.tcp.sendspace=1048576 >> net.inet.tcp.recvspace=1048576 >> >> Server: >> /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports >> >> Client: >> /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 >> /dev/amrd0s2 >> >> The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with >> adaptive read ahead and write back caching. >> >> Tom > > I have tried both the settings ideas suggested above but I cannot even > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > anything higher that 131072 results in ggated bailing with the error: > # ggated -v -a 10.10.0.19 > info: Reading exports file (/etc/gg.exports). > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > info: Exporting 2 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. > > setting net.inet.tcp.{send,recv}space to 131072 allows me to start > ggated with the default R and S values of 131072; anything higher > results in "no buffer space" errors. At 131072 ggated starts but then I > cannot even open a new connection (like ssh) to the server as the ssh > client bails with "no buffer space available".Did you also set kern.ipc.nmbclusters="32768" in /boot/loader.conf and reboot? It sounds like you did not as this is the exact same problem I came across before adjusting that value. <<SNIP>>> This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA > Raid using LSiMegaRaid.Do you have the cache BBU fitted (Batery Backup Unit) and the array caching set to write back? Also have you tested writing to the array locally without ggate to test the write speed?> > The odd thing is that even after I set the send and recvspace down to > values like 65536, I continue to get the no buffer error when trying to > connect to it remotely again. >I found that the easyest way to fix this was to reboot the system with good values for net.inet.tcp.{send,recv}space.
On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:> On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > > Dmitriy Kirhlarov wrote: > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > > >> I am trying to set up a HA type system involving two identical boxes and > > >> have gone through the following to set up the systems: > > >> > > >> Slave server: > > >> ggated -R 196608 -S 196608 > > >> (exporting /dev/amrd1 ) > > >> net.inet.tcp.sendspace: 65536 > > >> net.inet.tcp.recvspace: 131072 > > > > > > Try > > > net.local.stream.recvspace=65535 > > > net.local.stream.sendspace=65535 > > > > > > Also, try increase this sysctls with > > > net.inet.tcp.rfc1323=1 > > > > > > I use it on FreeBSD 5.x with: > > > net.inet.tcp.sendspace=131072 > > > net.inet.tcp.recvspace=131072 > > > net.local.stream.recvspace=65535 > > > net.local.stream.sendspace=65535 > > > > > > ggated -R 1048576 -S 1048576 > > > ggatec -R 1048576 -S 1048576 > > > > > > WBR. > > > Dmitriy > > > _______________________________________________ > > > freebsd-stable@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > > > I have seen sustained writes of 30Mb/s using the following configuration: > > > > cat /boot/loader.conf > > kern.ipc.nmbclusters="32768" > > > > cat /etc/sysctl.conf > > net.inet.tcp.sendspace=1048576 > > net.inet.tcp.recvspace=1048576 > > > > Server: > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > > > Client: > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > > /dev/amrd0s2 > > > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > > adaptive read ahead and write back caching. > > > > Tom > > I have tried both the settings ideas suggested above but I cannot even > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > anything higher that 131072 results in ggated bailing with the error: > # ggated -v -a 10.10.0.19 > info: Reading exports file (/etc/gg.exports). > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > info: Exporting 2 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting.For values of net.inet.tcp.{send,recv}space more than 524288 you also need to adjust kern.ipc.maxsockbuf Try this configuration for example: kern.ipc.maxsockbuf=2049152 net.inet.tcp.recvspace=1024576 net.inet.tcp.sendspace=1024576> > setting net.inet.tcp.{send,recv}space to 131072 allows me to start > ggated with the default R and S values of 131072; anything higher > results in "no buffer space" errors. At 131072 ggated starts but then I > cannot even open a new connection (like ssh) to the server as the ssh > client bails with "no buffer space available". > > more information: > # netstat -m > 514/641/1155 mbufs in use (current/cache/total) > 512/284/796/32768 mbuf clusters in use (current/cache/total/max) > 512/256 mbuf+clusters out of packet secondary zone in use > (current/cache) > 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) > 1152K/728K/1880K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/4/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA > Raid using LSiMegaRaid. > > The odd thing is that even after I set the send and recvspace down to > values like 65536, I continue to get the no buffer error when trying to > connect to it remotely again. > > Sven > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"-- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================
On Fri, 2007-04-06 at 16:18 +0300, Nikolay Pavlov wrote:> On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote: > > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > > > Dmitriy Kirhlarov wrote: > > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > > > >> I am trying to set up a HA type system involving two identical boxes and > > > >> have gone through the following to set up the systems: > > > >> > > > >> Slave server: > > > >> ggated -R 196608 -S 196608 > > > >> (exporting /dev/amrd1 ) > > > >> net.inet.tcp.sendspace: 65536 > > > >> net.inet.tcp.recvspace: 131072 > > > > > > > > Try > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > Also, try increase this sysctls with > > > > net.inet.tcp.rfc1323=1 > > > > > > > > I use it on FreeBSD 5.x with: > > > > net.inet.tcp.sendspace=131072 > > > > net.inet.tcp.recvspace=131072 > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > ggated -R 1048576 -S 1048576 > > > > ggatec -R 1048576 -S 1048576 > > > > > > > > WBR. > > > > Dmitriy > > > > _______________________________________________ > > > > freebsd-stable@freebsd.org mailing list > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > > > > > > I have seen sustained writes of 30Mb/s using the following configuration: > > > > > > cat /boot/loader.conf > > > kern.ipc.nmbclusters="32768" > > > > > > cat /etc/sysctl.conf > > > net.inet.tcp.sendspace=1048576 > > > net.inet.tcp.recvspace=1048576 > > > > > > Server: > > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > > > > > Client: > > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > > > /dev/amrd0s2 > > > > > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > > > adaptive read ahead and write back caching. > > > > > > Tom > > > > I have tried both the settings ideas suggested above but I cannot even > > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > > anything higher that 131072 results in ggated bailing with the error: > > # ggated -v -a 10.10.0.19 > > info: Reading exports file (/etc/gg.exports). > > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > > info: Exporting 2 object(s). > > error: Cannot open stream socket: No buffer space available. > > error: Exiting. > > For values of net.inet.tcp.{send,recv}space more than > 524288 you also need to adjust kern.ipc.maxsockbuf > > Try this configuration for example: > kern.ipc.maxsockbuf=2049152 > net.inet.tcp.recvspace=1024576 > net.inet.tcp.sendspace=1024576 >kern.ipc.maxsockbuf was the issue here; I increased its value and now I no longer get the buffer space error. Furthermore, the write speed issue was also tied to a hardware raid controller issue. After fixing that issue, and with just the following: kern.ipc.maxsockbuf=1048576 net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 I can start ggated with -R 262144 -S 262144 as well as the ggatec and see write speeds of 60+MB/s. I may play around with the settings more (and see if any further speed improvements occur), but this is quite acceptable at this point. (For the record nmbclusters is set to 32768). The next part of the project will be writing the freevrrp failover scripts to deal with I/O locking issues that will happen if the ggated server fails, etc.
Hi! On Mon, Apr 09, 2007 at 12:55:43PM -0400, Sven Willenberger wrote:> The next part of the project will be writing the freevrrp failover > scripts to deal with I/O locking issues that will happen if the ggated > server fails, etc.check carp(4) and ports/net/ifstated. You can use identical advbase configuration on both nodes and prevent service migration back from second node to first, if first node will reboot. also, take a look one{start,stop,restart} options for rcNG designed initscripts. WBR Dmitriy