Jeevan Patnaik
2018-Sep-24 15:51 UTC
[Gluster-users] Gluster tier in progress after attach
Hi, Yes, I did. The status shows all as completed. Regards, Jeevan. On Mon 24 Sep, 2018, 9:20 PM Nithya Balachandran, <nbalacha at redhat.com> wrote:> > > On 24 September 2018 at 21:01, Jeevan Patnaik <g1patnaik at gmail.com> wrote: > >> Hi, >> >> What do you mean by expanding volume? >> >> I mean adding new bricks to extend storage. It is mentioned that >> detaching tier is must for adding or removing bricks. >> >> What command did you use to detach the tier? And how did you check that >> the files still exist on the hot tier? >> >> gluster volume tier volume_name detach start >> >> I check the files directly from backend filesystem on disk and there are >> still left over files on all hot tier disks. >> >> Anyhow I proceeded with detach commit to see what would happen..so after >> commit, I can't access those files from client filesystems..so they're gone >> with the hot tier and were not moved to cold tier. >> > > Did you check that the detach tier operation had completed before running > detach commit? > >> >> And even if I try to attach the hot tier bricks again to bring back the >> files, attach won't work with existing files. >> >> So, it seems we shouldn't detach hot tier without making sure all files >> are moved to cold tier..but in my case, some files just won't move, even >> though they aren't being accessed anywhere. >> > > > >> >> There's a statement saying POSIX locks can cause files to be still left >> in hot tier and in that case, either the application causing posix lock has >> to be closed or files should be moved manually (but where?)... however, I >> don't think in my case there's any posix locks as the files are not >> accessed by any application except gluster itself. >> >> Regards, >> Jeevan. >> >> >> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, <nbalacha at redhat.com> >> wrote: >> >>> It has been a while since I worked on tier but here is what I remember: >>> >>> >>> On 24 September 2018 at 19:15, Jeevan Patnaik <g1patnaik at gmail.com> >>> wrote: >>> >>>> Hi Nithya, >>>> >>>> No..the files are not being accessed. And tiering mode is in cache mode >>>> and what I understood is in cache mode every file is moved to hot tier >>>> until it reaches low watermark, then it only promotes highly accessed files. >>>> >>>> Not quite . All files will initially exist on the cold tier. Any new >>> files will be created on the hot tier. Any old files that are accessed will >>> be promoted ot the hot tier. >>> >>> >>> And do you mean everytime we expand volume, we are virtually flushing >>>> the tier and it regenerates only when a files are accessed? >>>> >>>> What do you mean by expanding the volume? >>> >>> >>>> Also I see one more issue now, I hav generated another 1000 files after >>>> tiering is started and I see all 1000 files are now created on hot tier. >>>> Now, I detached again..in detach status it shows completed..but when I >>>> checked, around 260 files are still located on hot tier..those files are >>>> not being used anywhere..ca you tell why are they still not moved to cold >>>> tier and why the detach status is showing a wrong status. >>>> >>> >>> What command did you use to detach the tier? And how did you check that >>> the files still exist on the hot tier? >>> >>> Regards, >>> Nithya >>> >>>> >>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 >>>> hosts also. >>>> >>>> Regards, >>>> Jeevan. >>>> >>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <nbalacha at redhat.com> >>>> wrote: >>>> >>>>> Are those files being accessed? Tiering will only promote those that >>>>> have been accessed recently. >>>>> >>>>> Regards, >>>>> Nithya >>>>> >>>>> On 24 September 2018 at 18:32, Jeevan Patnaik <g1patnaik at gmail.com> >>>>> wrote: >>>>> >>>>>> I see it still not promoting any files. Do we need to run any command >>>>>> to force the movement? >>>>>> >>>>>> This would be an issue if we need to expand disk as then we also need >>>>>> to detach existing tier and attach again and expect the data to be promoted >>>>>> to the hot tier. >>>>>> >>>>>> Regards, >>>>>> Jeevan. >>>>>> >>>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, <g1patnaik at gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have created a 18 disk replica 3 volume and created 1000 random >>>>>>> files each of 10M. >>>>>>> >>>>>>> Later I have attached 15 disk replica 3 tier with each hot tier disk >>>>>>> coming from cold tier host, except on 3 (as we don't have hot tier disk) >>>>>>> >>>>>>> After attach, I see no files are promoted at even after 15 minutes. >>>>>>> Status is showing as In progress. >>>>>>> >>>>>>> Is it always this much slow? For 10G data, 15 minutes is too much >>>>>>> Also will the data try to be promoted to a hot tier coming from the >>>>>>> same host to reduce the time taken to move the files? >>>>>>> >>>>>>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe >>>>>>> SSDs. >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Jeevan. >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>> >>>>> >>> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180924/e15be0d3/attachment.html>
Nithya Balachandran
2018-Sep-24 16:06 UTC
[Gluster-users] Gluster tier in progress after attach
Please check the rebalance log to see why those files were not migrated. Regards, Nithya On 24 September 2018 at 21:21, Jeevan Patnaik <g1patnaik at gmail.com> wrote:> Hi, > > Yes, I did. The status shows all as completed. > > Regards, > Jeevan. > > On Mon 24 Sep, 2018, 9:20 PM Nithya Balachandran, <nbalacha at redhat.com> > wrote: > >> >> >> On 24 September 2018 at 21:01, Jeevan Patnaik <g1patnaik at gmail.com> >> wrote: >> >>> Hi, >>> >>> What do you mean by expanding volume? >>> >>> I mean adding new bricks to extend storage. It is mentioned that >>> detaching tier is must for adding or removing bricks. >>> >>> What command did you use to detach the tier? And how did you check that >>> the files still exist on the hot tier? >>> >>> gluster volume tier volume_name detach start >>> >>> I check the files directly from backend filesystem on disk and there are >>> still left over files on all hot tier disks. >>> >>> Anyhow I proceeded with detach commit to see what would happen..so after >>> commit, I can't access those files from client filesystems..so they're gone >>> with the hot tier and were not moved to cold tier. >>> >> >> Did you check that the detach tier operation had completed before running >> detach commit? >> >>> >>> And even if I try to attach the hot tier bricks again to bring back the >>> files, attach won't work with existing files. >>> >>> So, it seems we shouldn't detach hot tier without making sure all files >>> are moved to cold tier..but in my case, some files just won't move, even >>> though they aren't being accessed anywhere. >>> >> >> >> >>> >>> There's a statement saying POSIX locks can cause files to be still left >>> in hot tier and in that case, either the application causing posix lock has >>> to be closed or files should be moved manually (but where?)... however, I >>> don't think in my case there's any posix locks as the files are not >>> accessed by any application except gluster itself. >>> >>> Regards, >>> Jeevan. >>> >>> >>> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, <nbalacha at redhat.com> >>> wrote: >>> >>>> It has been a while since I worked on tier but here is what I remember: >>>> >>>> >>>> On 24 September 2018 at 19:15, Jeevan Patnaik <g1patnaik at gmail.com> >>>> wrote: >>>> >>>>> Hi Nithya, >>>>> >>>>> No..the files are not being accessed. And tiering mode is in cache >>>>> mode and what I understood is in cache mode every file is moved to hot tier >>>>> until it reaches low watermark, then it only promotes highly accessed files. >>>>> >>>>> Not quite . All files will initially exist on the cold tier. Any new >>>> files will be created on the hot tier. Any old files that are accessed will >>>> be promoted ot the hot tier. >>>> >>>> >>>> And do you mean everytime we expand volume, we are virtually flushing >>>>> the tier and it regenerates only when a files are accessed? >>>>> >>>>> What do you mean by expanding the volume? >>>> >>>> >>>>> Also I see one more issue now, I hav generated another 1000 files >>>>> after tiering is started and I see all 1000 files are now created on hot >>>>> tier. Now, I detached again..in detach status it shows completed..but when >>>>> I checked, around 260 files are still located on hot tier..those files are >>>>> not being used anywhere..ca you tell why are they still not moved to cold >>>>> tier and why the detach status is showing a wrong status. >>>>> >>>> >>>> What command did you use to detach the tier? And how did you check that >>>> the files still exist on the hot tier? >>>> >>>> Regards, >>>> Nithya >>>> >>>>> >>>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 >>>>> hosts also. >>>>> >>>>> Regards, >>>>> Jeevan. >>>>> >>>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <nbalacha at redhat.com> >>>>> wrote: >>>>> >>>>>> Are those files being accessed? Tiering will only promote those that >>>>>> have been accessed recently. >>>>>> >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On 24 September 2018 at 18:32, Jeevan Patnaik <g1patnaik at gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I see it still not promoting any files. Do we need to run any >>>>>>> command to force the movement? >>>>>>> >>>>>>> This would be an issue if we need to expand disk as then we also >>>>>>> need to detach existing tier and attach again and expect the data to be >>>>>>> promoted to the hot tier. >>>>>>> >>>>>>> Regards, >>>>>>> Jeevan. >>>>>>> >>>>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, <g1patnaik at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have created a 18 disk replica 3 volume and created 1000 random >>>>>>>> files each of 10M. >>>>>>>> >>>>>>>> Later I have attached 15 disk replica 3 tier with each hot tier >>>>>>>> disk coming from cold tier host, except on 3 (as we don't have hot tier >>>>>>>> disk) >>>>>>>> >>>>>>>> After attach, I see no files are promoted at even after 15 minutes. >>>>>>>> Status is showing as In progress. >>>>>>>> >>>>>>>> Is it always this much slow? For 10G data, 15 minutes is too much >>>>>>>> Also will the data try to be promoted to a hot tier coming from the >>>>>>>> same host to reduce the time taken to move the files? >>>>>>>> >>>>>>>> Cold tier is made of RAID supported SSD and hot tier is made of >>>>>>>> NVMe SSDs. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Jeevan. >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180924/47f7da3b/attachment.html>