nixiaoke
2011-Oct-12 03:40 UTC
[Gluster-users] Problem with VM images when one node goes online
When you use gluster volume as VM datastore, it will generate a big file such as *.vmdk according to your VM machine configure. When gluster doing self-heal, I am not sure gluster will replicate the whole file or only part of the file. If the whole file is replicated, it will be a heavy burden to the system. Maybe you can consider other arrangement to use gluster. From: Peter Linder <peter.linder at fiberdirekt.se> Subject: Re: [Gluster-users] Problem with VM images when one node goes online (self-healing) on a 2 node replication gluster for VMware datastore To: gluster-users at gluster.org Message-ID: <4E9406E0.8000803 at fiberdirekt.se> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" With 3.2.4 during self-heal, no operation on the file being healed is allowed so your VM's will stall and time out if the self-heal isn't finished quick enough. gluster 3.3 will fix this, but I don't know when it will be released. There are betas to try out though :). Perhaps somebody else can say how stable 3.3-beta2 is compared to 3.2.4? On 10/11/2011 10:57 AM, keith wrote:> Hi all > > I am testing gluster-3.2.4 on a 2 nodes storage with replication as > our VMware datastore. > > The setup is running replication on 2 nodes with ucarp and mount it on > WMware using NFS to gluster as a datastore. > >> Volume Name: GLVOL1 >> Type: Replicate >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: t4-01.store:/EXPORT/GLVOL1 >> Brick2: t4-03.store:/EXPORT/GLVOL1 >> Options Reconfigured: >> performance.cache-size: 4096MB > > High-availability testing goes on smoothly without any problem or > data-corruption, that is when any node is down, all VM guests runs > normally without any problem. > > The problem arises when I bring up the failed node and the node start > doing self-healing. All my VM guests get kernel error messages and > finally the VM guests ended up with "EXT3-fs error: > ext3_journal_start_sb: detected aborted journal" remount filesystem > (root) as read-only. > > Below are some of the VM guests kernel error generated when I bring up > the failed gluster node for self-healing: > >> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >> ffff8100221c90c0 >> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >> ffff8100221c9240 >> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >> ffff8100221c93c0 >> Oct 11 15:58:34 testvm3 kernel: INFO: task kjournald:2081 blocked for >> more than 120 seconds. >> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Oct 11 15:58:34 testvm3 kernel: kjournald D ffff810001736420 >> 0 2081 14 2494 2060 (L-TLB) >> Oct 11 15:58:34 testvm3 kernel: ffff81003c087cf0 0000000000000046 >> ffff810030ef2288 ffff81003f5d6048 >> Oct 11 15:58:34 testvm3 kernel: 00000000037685c8 000000000000000a >> ffff810037c53820 ffffffff80314b60 >> Oct 11 15:58:34 testvm3 kernel: 00001883cb68d47d 0000000000002c4e >> ffff810037c53a08 000000003f5128b8 >> Oct 11 15:58:34 testvm3 kernel: Call Trace: >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8006ec8f>] >> do_gettimeofday+0x40/0x90 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >> sync_buffer+0x0/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800637ce>] >> io_schedule+0x3f/0x67 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8001560e>] >> sync_buffer+0x3b/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800639fa>] >> __wait_on_bit+0x40/0x6e >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >> sync_buffer+0x0/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80063a94>] >> out_of_line_wait_on_bit+0x6c/0x78 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2e2b>] >> wake_bit_function+0x0/0x23 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88033a41>] >> :jbd:journal_commit_transaction+0x553/0x10aa >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8003d85b>] >> lock_timer_base+0x1b/0x3c >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8004ad98>] >> try_to_del_timer_sync+0x7f/0x88 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88037662>] >> :jbd:kjournald+0xc1/0x213 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2dfd>] >> autoremove_wake_function+0x0/0x2e >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >> keventd_create_kthread+0x0/0xc4 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff880375a1>] >> :jbd:kjournald+0x0/0x213 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >> keventd_create_kthread+0x0/0xc4 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80032722>] kthread+0xfe/0x132 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >> keventd_create_kthread+0x0/0xc4 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80032624>] kthread+0x0/0x132 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 >> Oct 11 15:58:34 testvm3 kernel: >> Oct 11 15:58:34 testvm3 kernel: INFO: task crond:3418 blocked for >> more than 120 seconds. >> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Oct 11 15:58:34 testvm3 kernel: crond D ffff810001736420 >> 0 3418 1 3436 3405 (NOTLB) >> Oct 11 15:58:34 testvm3 kernel: ffff810036c55ca8 0000000000000086 >> 0000000000000000 ffffffff80019e3e >> Oct 11 15:58:34 testvm3 kernel: 0000000000065bf2 0000000000000007 >> ffff81003ce4b080 ffffffff80314b60 >> Oct 11 15:58:34 testvm3 kernel: 000018899ae16270 0000000000023110 >> ffff81003ce4b268 000000008804ec00 >> Oct 11 15:58:34 testvm3 kernel: Call Trace: >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80019e3e>] __getblk+0x25/0x22c >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8006ec8f>] >> do_gettimeofday+0x40/0x90 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >> sync_buffer+0x0/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800637ce>] >> io_schedule+0x3f/0x67 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8001560e>] >> sync_buffer+0x3b/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80063912>] >> __wait_on_bit_lock+0x36/0x66 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >> sync_buffer+0x0/0x3f >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800639ae>] >> out_of_line_wait_on_bit_lock+0x6c/0x78 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2e2b>] >> wake_bit_function+0x0/0x23 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8803181e>] >> :jbd:do_get_write_access+0x54/0x522 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80019e3e>] __getblk+0x25/0x22c >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88031d0e>] >> :jbd:journal_get_write_access+0x22/0x33 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8804dd37>] >> :ext3:ext3_reserve_inode_write+0x38/0x90 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8804ddb0>] >> :ext3:ext3_mark_inode_dirty+0x21/0x3c >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88050d35>] >> :ext3:ext3_dirty_inode+0x63/0x7b >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80013d98>] >> __mark_inode_dirty+0x29/0x16e >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80025a49>] filldir+0x0/0xb7 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8003516b>] >> vfs_readdir+0x8c/0xa9 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800389db>] >> sys_getdents+0x75/0xbd >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 >> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 >> Oct 11 15:58:34 testvm3 kernel: >> Oct 11 15:58:34 testvm3 kernel: INFO: task httpd:3452 blocked for >> more than 120 seconds. >> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Oct 11 15:58:34 testvm3 kernel: httpd D ffff810001736420 >> 0 3452 3405 3453 (NOTLB) >> Oct 11 15:58:34 testvm3 kernel: ffff810035ea9dc8 0000000000000086 >> 0000000000000000 ffffffff80009a1c >> Oct 11 15:58:34 testvm3 kernel: ffff810035ea9e28 0000000000000009 >> ffff810037e52080 ffffffff80314b60 >> Oct 11 15:58:34 testvm3 kernel: 000018839f75405c 000000000003363d >> ffff810037e52268 000000003f5e7150 > > Please note that although I am using ucarp for IP failover and by > default ucarp will alway have a preferred master, I have added codes > to make sure that the ucarp master will always become slave when it > goes down and come up again. This will ensure that WMware will not > connect back to the failed node when it comes back up. > > However this does not prevent the problem I describe above. > > There are a lot of logs generated during self-healing process. It > doesn't make any sense to me. I am attaching it. It's over 900k. So I > zip them up. Hopefully the mailling list allow attachment. > > Is there any best practices to setup/run gluster with replication as a > datastore to VMware that make sure VM guests run smoothly even when > one node goes into self-healing? > > Any advise is appreciated. > > Keith > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Peter Linder
2011-Oct-12 06:23 UTC
[Gluster-users] Problem with VM images when one node goes online
Gluster has some way to determine if it is best to replace the whole file (typically good for small files), or transfer a diff or some sort (usually good for large files that are similar). It does have to compare the whole file though, that means reading it and comparing checksums (I think thats how its done). During that time, with 3.2 and below, the entire file is locked to ensure its integrity after the self-heal is complete. 3.3 introduces locking only the part being worked on at the moment, so that the VM can continue reading/writing any part of the file any time it wants to. Even if it happened to hit a locked portion of the file, that part will be unlocked very soon as the self-heal process moves on. Thats how I understand it, at least. On 10/12/2011 5:40 AM, nixiaoke wrote:> > When you use gluster volume as VM datastore, it will generate a big > file such as *.vmdk according to your VM machine configure. > > When gluster doing self-heal, I am not sure gluster will replicate the > whole file or only part of the file. > > If the whole file is replicated, it will be a heavy burden to the system. > > Maybe you can consider other arrangement to use gluster. > > > From: Peter Linder <peter.linder at fiberdirekt.se> > Subject: Re: [Gluster-users] Problem with VM images when one node goes > online (self-healing) on a 2 node replication gluster for VMware > datastore > To: gluster-users at gluster.org > Message-ID: <4E9406E0.8000803 at fiberdirekt.se> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > With 3.2.4 during self-heal, no operation on the file being healed is > allowed so your VM's will stall and time out if the self-heal isn't > finished quick enough. gluster 3.3 will fix this, but I don't know when > it will be released. There are betas to try out though :). Perhaps > somebody else can say how stable 3.3-beta2 is compared to 3.2.4? > > On 10/11/2011 10:57 AM, keith wrote: >> Hi all >> >> I am testing gluster-3.2.4 on a 2 nodes storage with replication as >> our VMware datastore. >> >> The setup is running replication on 2 nodes with ucarp and mount it on >> WMware using NFS to gluster as a datastore. >> >>> Volume Name: GLVOL1 >>> Type: Replicate >>> Status: Started >>> Number of Bricks: 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: t4-01.store:/EXPORT/GLVOL1 >>> Brick2: t4-03.store:/EXPORT/GLVOL1 >>> Options Reconfigured: >>> performance.cache-size: 4096MB >> >> High-availability testing goes on smoothly without any problem or >> data-corruption, that is when any node is down, all VM guests runs >> normally without any problem. >> >> The problem arises when I bring up the failed node and the node start >> doing self-healing. All my VM guests get kernel error messages and >> finally the VM guests ended up with "EXT3-fs error: >> ext3_journal_start_sb: detected aborted journal" remount filesystem >> (root) as read-only. >> >> Below are some of the VM guests kernel error generated when I bring up >> the failed gluster node for self-healing: >> >>> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >>> ffff8100221c90c0 >>> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >>> ffff8100221c9240 >>> Oct 11 15:57:58 testvm3 kernel: pvscsi: task abort on host 1, >>> ffff8100221c93c0 >>> Oct 11 15:58:34 testvm3 kernel: INFO: task kjournald:2081 blocked for >>> more than 120 seconds. >>> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> Oct 11 15:58:34 testvm3 kernel: kjournald D ffff810001736420 >>> 0 2081 14 2494 2060 (L-TLB) >>> Oct 11 15:58:34 testvm3 kernel: ffff81003c087cf0 0000000000000046 >>> ffff810030ef2288 ffff81003f5d6048 >>> Oct 11 15:58:34 testvm3 kernel: 00000000037685c8 000000000000000a >>> ffff810037c53820 ffffffff80314b60 >>> Oct 11 15:58:34 testvm3 kernel: 00001883cb68d47d 0000000000002c4e >>> ffff810037c53a08 000000003f5128b8 >>> Oct 11 15:58:34 testvm3 kernel: Call Trace: >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8006ec8f>] >>> do_gettimeofday+0x40/0x90 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >>> sync_buffer+0x0/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800637ce>] >>> io_schedule+0x3f/0x67 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8001560e>] >>> sync_buffer+0x3b/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800639fa>] >>> __wait_on_bit+0x40/0x6e >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >>> sync_buffer+0x0/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80063a94>] >>> out_of_line_wait_on_bit+0x6c/0x78 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2e2b>] >>> wake_bit_function+0x0/0x23 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88033a41>] >>> :jbd:journal_commit_transaction+0x553/0x10aa >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8003d85b>] >>> lock_timer_base+0x1b/0x3c >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8004ad98>] >>> try_to_del_timer_sync+0x7f/0x88 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88037662>] >>> :jbd:kjournald+0xc1/0x213 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2dfd>] >>> autoremove_wake_function+0x0/0x2e >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >>> keventd_create_kthread+0x0/0xc4 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff880375a1>] >>> :jbd:kjournald+0x0/0x213 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >>> keventd_create_kthread+0x0/0xc4 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80032722>] kthread+0xfe/0x132 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2be5>] >>> keventd_create_kthread+0x0/0xc4 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80032624>] kthread+0x0/0x132 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 >>> Oct 11 15:58:34 testvm3 kernel: >>> Oct 11 15:58:34 testvm3 kernel: INFO: task crond:3418 blocked for >>> more than 120 seconds. >>> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> Oct 11 15:58:34 testvm3 kernel: crond D ffff810001736420 >>> 0 3418 1 3436 3405 (NOTLB) >>> Oct 11 15:58:34 testvm3 kernel: ffff810036c55ca8 0000000000000086 >>> 0000000000000000 ffffffff80019e3e >>> Oct 11 15:58:34 testvm3 kernel: 0000000000065bf2 0000000000000007 >>> ffff81003ce4b080 ffffffff80314b60 >>> Oct 11 15:58:34 testvm3 kernel: 000018899ae16270 0000000000023110 >>> ffff81003ce4b268 000000008804ec00 >>> Oct 11 15:58:34 testvm3 kernel: Call Trace: >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80019e3e>] >>> __getblk+0x25/0x22c >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8006ec8f>] >>> do_gettimeofday+0x40/0x90 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >>> sync_buffer+0x0/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800637ce>] >>> io_schedule+0x3f/0x67 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8001560e>] >>> sync_buffer+0x3b/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80063912>] >>> __wait_on_bit_lock+0x36/0x66 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800155d3>] >>> sync_buffer+0x0/0x3f >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800639ae>] >>> out_of_line_wait_on_bit_lock+0x6c/0x78 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800a2e2b>] >>> wake_bit_function+0x0/0x23 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8803181e>] >>> :jbd:do_get_write_access+0x54/0x522 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80019e3e>] >>> __getblk+0x25/0x22c >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88031d0e>] >>> :jbd:journal_get_write_access+0x22/0x33 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8804dd37>] >>> :ext3:ext3_reserve_inode_write+0x38/0x90 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8804ddb0>] >>> :ext3:ext3_mark_inode_dirty+0x21/0x3c >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff88050d35>] >>> :ext3:ext3_dirty_inode+0x63/0x7b >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80013d98>] >>> __mark_inode_dirty+0x29/0x16e >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff80025a49>] filldir+0x0/0xb7 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8003516b>] >>> vfs_readdir+0x8c/0xa9 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff800389db>] >>> sys_getdents+0x75/0xbd >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 >>> Oct 11 15:58:34 testvm3 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 >>> Oct 11 15:58:34 testvm3 kernel: >>> Oct 11 15:58:34 testvm3 kernel: INFO: task httpd:3452 blocked for >>> more than 120 seconds. >>> Oct 11 15:58:34 testvm3 kernel: "echo 0 > >>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> Oct 11 15:58:34 testvm3 kernel: httpd D ffff810001736420 >>> 0 3452 3405 3453 (NOTLB) >>> Oct 11 15:58:34 testvm3 kernel: ffff810035ea9dc8 0000000000000086 >>> 0000000000000000 ffffffff80009a1c >>> Oct 11 15:58:34 testvm3 kernel: ffff810035ea9e28 0000000000000009 >>> ffff810037e52080 ffffffff80314b60 >>> Oct 11 15:58:34 testvm3 kernel: 000018839f75405c 000000000003363d >>> ffff810037e52268 000000003f5e7150 >> >> Please note that although I am using ucarp for IP failover and by >> default ucarp will alway have a preferred master, I have added codes >> to make sure that the ucarp master will always become slave when it >> goes down and come up again. This will ensure that WMware will not >> connect back to the failed node when it comes back up. >> >> However this does not prevent the problem I describe above. >> >> There are a lot of logs generated during self-healing process. It >> doesn't make any sense to me. I am attaching it. It's over 900k. So I >> zip them up. Hopefully the mailling list allow attachment. >> >> Is there any best practices to setup/run gluster with replication as a >> datastore to VMware that make sure VM guests run smoothly even when >> one node goes into self-healing? >> >> Any advise is appreciated. >> >> Keith >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users