When we say client side heal or server side heal, we basically talking about the side which "triggers" heal of a file. 1 - server side heal - shd scans indices and triggers heal 2 - client side heal - a fop finds that file needs heal and it triggers heal for that file. Now, what happens when heal gets triggered. In both the cases following functions takes part - ec_heal => ec_heal_throttle=>ec_launch_heal Now ec_launch_heal just creates heal tasks (with ec_synctask_heal_wrap which calls ec_heal_do ) and put it into a queue. This happens on server and "syncenv" infrastructure which is nothing but a set of workers pick these tasks and execute it. That is when actual read/write for heal happens. ----- Original Message ----- From: "Serkan ?oban" <cobanserkan at gmail.com> To: "Ashish Pandey" <aspandey at redhat.com> Cc: "Gluster Users" <gluster-users at gluster.org> Sent: Monday, May 29, 2017 6:44:50 PM Subject: Re: [Gluster-users] Heal operation detail of EC volumes>>Healing could be triggered by client side (access of file) or server side (shd). >>However, in both the cases actual heal starts from "ec_heal_do" function.If I do a recursive getfattr operation from clients, then all heal operation is done on clients right? Client read the chunks, calculate and write the missing chunk. And If I don't access files from client then SHD daemons will start heal and read,calculate,write the missing chunks right? In first case EC calculations takes places in client fuse process, in second case EC calculations will be made in SHD process right? Does brick process has any role in EC calculations? On Mon, May 29, 2017 at 3:32 PM, Ashish Pandey <aspandey at redhat.com> wrote:> > > ________________________________ > From: "Serkan ?oban" <cobanserkan at gmail.com> > To: "Gluster Users" <gluster-users at gluster.org> > Sent: Monday, May 29, 2017 5:13:06 PM > Subject: [Gluster-users] Heal operation detail of EC volumes > > Hi, > > When a brick fails in EC, What is the healing read/write data path? > Which processes do the operations? > > Healing could be triggered by client side (access of file) or server side > (shd). > However, in both the cases actual heal starts from "ec_heal_do" function. > > > Assume a 2GB file is being healed in 16+4 EC configuration. I was > thinking that SHD deamon on failed brick host will read 2GB from > network and reconstruct its 100MB chunk and write it on to brick. Is > this right? > > You are correct about read/write. > The only point is that, SHD deamon on one of the good brick will pick the > index entry and heal it. > SHD deamon scans the .glusterfs/index directory and heals the entries. If > the brick went down while IO was going on, index will be present on killed > brick also. > However, if a brick was down and then you started writing on a file then in > this case index entry would not be present on killed brick. > So even after brick will be UP, sdh on that brick will not be able to find > it out this index. However, other bricks would have entries and shd on that > brick will heal it. > > Note: I am considering each brick on different node. > > Ashish > > > > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >_______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170530/4d92f1a7/attachment.html>
Ok I understand that heal operation takes place on server side. In this case I should see X KB out network traffic from 16 servers and 16X KB input traffic to the failed brick server right? So that process will get 16 chunks recalculate our chunk and write it to disk. The problem is I am not seeing such kind of traffic on servers. In my configuration (16+4 EC) I see 20 servers are all have 7-8MB outbound traffic and none of them has more than 10MB incoming traffic. Only heal operation is happening on cluster right now, no client/other traffic. I see constant 7-8MB write to healing brick disk. So where is the missing traffic? On Tue, May 30, 2017 at 10:25 AM, Ashish Pandey <aspandey at redhat.com> wrote:> > When we say client side heal or server side heal, we basically talking about > the side which "triggers" heal of a file. > > 1 - server side heal - shd scans indices and triggers heal > > 2 - client side heal - a fop finds that file needs heal and it triggers heal > for that file. > > Now, what happens when heal gets triggered. > In both the cases following functions takes part - > > ec_heal => ec_heal_throttle=>ec_launch_heal > > Now ec_launch_heal just creates heal tasks (with ec_synctask_heal_wrap which > calls ec_heal_do ) and put it into a queue. > This happens on server and "syncenv" infrastructure which is nothing but a > set of workers pick these tasks and execute it. That is when actual > read/write for > heal happens. > > > ________________________________ > From: "Serkan ?oban" <cobanserkan at gmail.com> > To: "Ashish Pandey" <aspandey at redhat.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Monday, May 29, 2017 6:44:50 PM > Subject: Re: [Gluster-users] Heal operation detail of EC volumes > > >>>Healing could be triggered by client side (access of file) or server side >>> (shd). >>>However, in both the cases actual heal starts from "ec_heal_do" function. > If I do a recursive getfattr operation from clients, then all heal > operation is done on clients right? Client read the chunks, calculate > and write the missing chunk. > And If I don't access files from client then SHD daemons will start > heal and read,calculate,write the missing chunks right? > > In first case EC calculations takes places in client fuse process, in > second case EC calculations will be made in SHD process right? > Does brick process has any role in EC calculations? > > On Mon, May 29, 2017 at 3:32 PM, Ashish Pandey <aspandey at redhat.com> wrote: >> >> >> ________________________________ >> From: "Serkan ?oban" <cobanserkan at gmail.com> >> To: "Gluster Users" <gluster-users at gluster.org> >> Sent: Monday, May 29, 2017 5:13:06 PM >> Subject: [Gluster-users] Heal operation detail of EC volumes >> >> Hi, >> >> When a brick fails in EC, What is the healing read/write data path? >> Which processes do the operations? >> >> Healing could be triggered by client side (access of file) or server side >> (shd). >> However, in both the cases actual heal starts from "ec_heal_do" function. >> >> >> Assume a 2GB file is being healed in 16+4 EC configuration. I was >> thinking that SHD deamon on failed brick host will read 2GB from >> network and reconstruct its 100MB chunk and write it on to brick. Is >> this right? >> >> You are correct about read/write. >> The only point is that, SHD deamon on one of the good brick will pick the >> index entry and heal it. >> SHD deamon scans the .glusterfs/index directory and heals the entries. If >> the brick went down while IO was going on, index will be present on killed >> brick also. >> However, if a brick was down and then you started writing on a file then >> in >> this case index entry would not be present on killed brick. >> So even after brick will be UP, sdh on that brick will not be able to >> find >> it out this index. However, other bricks would have entries and shd on >> that >> brick will heal it. >> >> Note: I am considering each brick on different node. >> >> Ashish >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >
Xavier Hernandez
2017-Jun-01 07:30 UTC
[Gluster-users] Heal operation detail of EC volumes
Hi Serkan, On 30/05/17 10:22, Serkan ?oban wrote:> Ok I understand that heal operation takes place on server side. In > this case I should see X KB > out network traffic from 16 servers and 16X KB input traffic to the > failed brick server right? So that process will get 16 chunks > recalculate our chunk and write it to disk.That should be the normal operation for a single heal.> The problem is I am not seeing such kind of traffic on servers. In my > configuration (16+4 EC) I see 20 servers are all have 7-8MB outbound > traffic and none of them has more than 10MB incoming traffic. > Only heal operation is happening on cluster right now, no client/other > traffic. I see constant 7-8MB write to healing brick disk. So where is > the missing traffic?Not sure about your configuration, but probably you are seeing the result of having the SHD of each server doing heals. That would explain the network traffic you have. Suppose that all SHD but the one on the damaged brick are working. In this case 19 servers will peek 16 fragments each. This gives 19 * 16 = 304 fragments to be requested. EC balances the reads among all available servers, and there's a chance (1/19) that a fragment is local to the server asking it. So we'll need a total of 304 - 304 / 19 = 288 network requests, 288 / 19 = 15.2 sent by each server. If we have a total of 288 requests, it means that each server will answer 288 / 19 = 15.2 requests. The net effect of all this is that each healthy server is sending 15.2*X bytes of data and each server is receiving 15.2*X bytes of data. Now we need to account for the writes to the damaged brick. We have 19 simultaneous heals. This means that the damaged brick will receive 19*X bytes of data, and each healthy server will send X additional bytes of data. So: A healthy server receives 15.2*X bytes of data A healthy server sends 16.2*X bytes of data A damaged server receives 19*X bytes of data A damaged server sends few bytes of data (communication and synchronization overhead basically) As you can see, in this configuration each server has almost the same amount of inbound and outbound traffic. Only big difference is the damaged brick, that should receive a little more of traffic, but it should send much less. Is it possible that this matches your observations ? There's one more thing to consider here, and it's the apparent low throughput of self-heal. One possible thing to check is the small size and random behavior of the requests. Assuming that each request has a size of ~128 / 16 = 8KB, at a rate of ~8 MB/s the servers are processing ~1000 IOPS. Since requests are going to 19 different files, even if each file is accessed sequentially, the real effect will be like random access (some read-ahead on the filesystem can improve reads a bit, but writes won't benefit so much). How many IOPS can handle your bricks ? Do you have a test environment where we could check all this ? if possible it would be interesting to have only a single SHD (kill all SHD from all servers but one). In this situation, without client accesses, we should see the 16/1 ratio of reads vs writes on the network. We should also see a similar of even a little better speed because all reads and writes will be sequential, optimizing available IOPS. There's a feature to allow to configure the self-heal block size to optimize these cases. The feature is available on 3.11. Best regards, Xavi> > On Tue, May 30, 2017 at 10:25 AM, Ashish Pandey <aspandey at redhat.com> wrote: >> >> When we say client side heal or server side heal, we basically talking about >> the side which "triggers" heal of a file. >> >> 1 - server side heal - shd scans indices and triggers heal >> >> 2 - client side heal - a fop finds that file needs heal and it triggers heal >> for that file. >> >> Now, what happens when heal gets triggered. >> In both the cases following functions takes part - >> >> ec_heal => ec_heal_throttle=>ec_launch_heal >> >> Now ec_launch_heal just creates heal tasks (with ec_synctask_heal_wrap which >> calls ec_heal_do ) and put it into a queue. >> This happens on server and "syncenv" infrastructure which is nothing but a >> set of workers pick these tasks and execute it. That is when actual >> read/write for >> heal happens. >> >> >> ________________________________ >> From: "Serkan ?oban" <cobanserkan at gmail.com> >> To: "Ashish Pandey" <aspandey at redhat.com> >> Cc: "Gluster Users" <gluster-users at gluster.org> >> Sent: Monday, May 29, 2017 6:44:50 PM >> Subject: Re: [Gluster-users] Heal operation detail of EC volumes >> >> >>>> Healing could be triggered by client side (access of file) or server side >>>> (shd). >>>> However, in both the cases actual heal starts from "ec_heal_do" function. >> If I do a recursive getfattr operation from clients, then all heal >> operation is done on clients right? Client read the chunks, calculate >> and write the missing chunk. >> And If I don't access files from client then SHD daemons will start >> heal and read,calculate,write the missing chunks right? >> >> In first case EC calculations takes places in client fuse process, in >> second case EC calculations will be made in SHD process right? >> Does brick process has any role in EC calculations? >> >> On Mon, May 29, 2017 at 3:32 PM, Ashish Pandey <aspandey at redhat.com> wrote: >>> >>> >>> ________________________________ >>> From: "Serkan ?oban" <cobanserkan at gmail.com> >>> To: "Gluster Users" <gluster-users at gluster.org> >>> Sent: Monday, May 29, 2017 5:13:06 PM >>> Subject: [Gluster-users] Heal operation detail of EC volumes >>> >>> Hi, >>> >>> When a brick fails in EC, What is the healing read/write data path? >>> Which processes do the operations? >>> >>> Healing could be triggered by client side (access of file) or server side >>> (shd). >>> However, in both the cases actual heal starts from "ec_heal_do" function. >>> >>> >>> Assume a 2GB file is being healed in 16+4 EC configuration. I was >>> thinking that SHD deamon on failed brick host will read 2GB from >>> network and reconstruct its 100MB chunk and write it on to brick. Is >>> this right? >>> >>> You are correct about read/write. >>> The only point is that, SHD deamon on one of the good brick will pick the >>> index entry and heal it. >>> SHD deamon scans the .glusterfs/index directory and heals the entries. If >>> the brick went down while IO was going on, index will be present on killed >>> brick also. >>> However, if a brick was down and then you started writing on a file then >>> in >>> this case index entry would not be present on killed brick. >>> So even after brick will be UP, sdh on that brick will not be able to >>> find >>> it out this index. However, other bricks would have entries and shd on >>> that >>> brick will heal it. >>> >>> Note: I am considering each brick on different node. >>> >>> Ashish >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >