Hello Gluster users, As you all aware that glusterfs-6 is out, we would like to inform you that, we have spent a significant amount of time in testing glusterfs-6 in upgrade scenarios. We have done upgrade testing to glusterfs-6 from various releases like 3.12, 4.1 and 5.3. As glusterfs-6 has got in a lot of changes, we wanted to test those portions. There were xlators (and respective options to enable/disable them) added and deprecated in glusterfs-6 from various versions [1]. We had to check the following upgrade scenarios for all such options Identified in [1]: 1) option never enabled and upgraded 2) option enabled and then upgraded 3) option enabled and then disabled and then upgraded We weren't manually able to check all the combinations for all the options. So the options involving enabling and disabling xlators were prioritized. The below are the result of the ones tested. Never enabled and upgraded: checked from 3.12, 4.1, 5.3 to 6 the upgrade works. Enabled and upgraded: Tested for tier which is deprecated, It is not a recommended upgrade. As expected the volume won't be consumable and will have a few more issues as well. Tested with 3.12, 4.1 and 5.3 to 6 upgrade. Enabled, disabled before upgrade. Tested for tier with 3.12 and the upgrade went fine. There is one common issue to note in every upgrade. The node being upgraded is going into disconnected state. You have to flush the iptables and the restart glusterd on all nodes to fix this. The testing for enabling new options is still pending. The new options won't cause as much issues as the deprecated ones so this was put at the end of the priority list. It would be nice to get contributions for this. For the disable testing, tier was used as it covers most of the xlator that was removed. And all of these tests were done on a replica 3 volume. Note: This is only for upgrade testing of the newly added and removed xlators. Does not involve the normal tests for the xlator. If you have any questions, please feel free to reach us. [1] https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing Regards, Hari and Sanju.
Hi, Have added a few more info that was missed earlier. The disconnect issue being minor we are working on it with a lower priority. But yes, it will be fixed soon. The bug to track this is: https://bugzilla.redhat.com/show_bug.cgi?id=1694010 The workaround to get over this if it happens is to, upgrade the nodes one after other to the latest version. Once the upgrade is done, 1) kill the glusterd process alone in all the nodes using the command "pkill glusterd" 2) then do a "iptables -F" to flush the iptables. 3) start glusterd using "glusterd" Note: users can use systemctl stop/start glusterd.service command as well instead of the above to kill and start glusterd. On Fri, Mar 29, 2019 at 11:42 AM Hari Gowtham <hgowtham at redhat.com> wrote:> > Hello Gluster users, > > As you all aware that glusterfs-6 is out, we would like to inform you > that, we have spent a significant amount of time in testing > glusterfs-6 in upgrade scenarios. We have done upgrade testing to > glusterfs-6 from various releases like 3.12, 4.1 and 5.3. > > As glusterfs-6 has got in a lot of changes, we wanted to test those portions. > There were xlators (and respective options to enable/disable them) > added and deprecated in glusterfs-6 from various versions [1]. > > We had to check the following upgrade scenarios for all such options > Identified in [1]: > 1) option never enabled and upgraded > 2) option enabled and then upgraded > 3) option enabled and then disabled and then upgraded > > We weren't manually able to check all the combinations for all the options. > So the options involving enabling and disabling xlators were prioritized. > The below are the result of the ones tested. > > Never enabled and upgraded: > checked from 3.12, 4.1, 5.3 to 6 the upgrade works. > > Enabled and upgraded: > Tested for tier which is deprecated, It is not a recommended upgrade. > As expected the volume won't be consumable and will have a few more > issues as well. > Tested with 3.12, 4.1 and 5.3 to 6 upgrade. > > Enabled, disabled before upgrade. > Tested for tier with 3.12 and the upgrade went fine. > > There is one common issue to note in every upgrade. The node being > upgraded is going into disconnected state. You have to flush the iptables > and the restart glusterd on all nodes to fix this. > > The testing for enabling new options is still pending. The new options > won't cause as much issues as the deprecated ones so this was put at > the end of the priority list. It would be nice to get contributions > for this. > > For the disable testing, tier was used as it covers most of the xlator > that was removed. And all of these tests were done on a replica 3 volume. > > Note: This is only for upgrade testing of the newly added and removed > xlators. Does not involve the normal tests for the xlator. > > If you have any questions, please feel free to reach us. > > [1] https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing > > Regards, > Hari and Sanju.-- Regards, Hari Gowtham.
Sankarshan Mukhopadhyay
2019-Apr-01 00:23 UTC
[Gluster-users] [Gluster-devel] Upgrade testing to gluster 6
Quite a considerable amount of detail here. Thank you! On Fri, Mar 29, 2019 at 11:42 AM Hari Gowtham <hgowtham at redhat.com> wrote:> > Hello Gluster users, > > As you all aware that glusterfs-6 is out, we would like to inform you > that, we have spent a significant amount of time in testing > glusterfs-6 in upgrade scenarios. We have done upgrade testing to > glusterfs-6 from various releases like 3.12, 4.1 and 5.3. > > As glusterfs-6 has got in a lot of changes, we wanted to test those portions. > There were xlators (and respective options to enable/disable them) > added and deprecated in glusterfs-6 from various versions [1]. > > We had to check the following upgrade scenarios for all such options > Identified in [1]: > 1) option never enabled and upgraded > 2) option enabled and then upgraded > 3) option enabled and then disabled and then upgraded > > We weren't manually able to check all the combinations for all the options. > So the options involving enabling and disabling xlators were prioritized. > The below are the result of the ones tested. > > Never enabled and upgraded: > checked from 3.12, 4.1, 5.3 to 6 the upgrade works. > > Enabled and upgraded: > Tested for tier which is deprecated, It is not a recommended upgrade. > As expected the volume won't be consumable and will have a few more > issues as well. > Tested with 3.12, 4.1 and 5.3 to 6 upgrade. > > Enabled, disabled before upgrade. > Tested for tier with 3.12 and the upgrade went fine. > > There is one common issue to note in every upgrade. The node being > upgraded is going into disconnected state. You have to flush the iptables > and the restart glusterd on all nodes to fix this. >Is this something that is written in the upgrade notes? I do not seem to recall, if not, I'll send a PR> The testing for enabling new options is still pending. The new options > won't cause as much issues as the deprecated ones so this was put at > the end of the priority list. It would be nice to get contributions > for this. >Did the range of tests lead to any new issues?> For the disable testing, tier was used as it covers most of the xlator > that was removed. And all of these tests were done on a replica 3 volume. >I'm not sure if the Glusto team is reading this, but it would be pertinent to understand if the approach you have taken can be converted into a form of automated testing pre-release.> Note: This is only for upgrade testing of the newly added and removed > xlators. Does not involve the normal tests for the xlator. > > If you have any questions, please feel free to reach us. > > [1] https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing > > Regards, > Hari and Sanju.