Christian Rice
2016-Oct-27 18:52 UTC
[Gluster-users] trying to avoid a penalty for renaming every file
We have a gluster pool where our developers are writing large files with the suffix .COPYING, then removing that suffix when the copy is done (renaming the file), so consumers of the data know the file is safe to read. Back in 2014, Jeff Darcy wrote the below in a thread. I guess I am basically asking if this is still the supported/encouraged approach, and how do I implement this when I?m using gluster?s NFS? "That said, there's also a trick you can use to avoid creation of a linkfile. Other tools, such as rsync and our own object interface, use the same write-then-rename idiom. To serve them, there's an option called extra-hash-regex that can be used to place files on the "right" brick according to their final name even though they're created with another. Unfortunately, specifying that option via the command line doesn't seem to work (it creates a malformed volfile) so you have to mount a bit differently. For example: glusterfs --volfile-server=a_server --volfile-id=a_volume \ --xlator-option a_volume-dht.extra_hash_regex='(.*+)tmp' \ /a/mountpoint The important part is that second line. That causes any file with a "tmp" suffix to be hashed and placed as though only the part in the first parenthesized part of the regex (i.e. without the "tmp") was there. Therefore, creating "xxxtmp" and then renaming it to "xxx" is the same as just creating "xxx" in the first place as far as linkfiles etc. are concerned. Note that the excluded part can be anything that a regex can match, including a unique random number. If I recall, rsync uses temp files something like this: fubar = .fubar.NNNNNN (where NNNNNNN is a random number) I know this probably seems a little voodoo-ish, but with a little bit of experimentation to find the right regex you should be able to avoid those dreaded linkfiles altogether." -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161027/645f8fe0/attachment.html>
Joe Julian
2016-Oct-27 20:23 UTC
[Gluster-users] trying to avoid a penalty for renaming every file
The best slide of all from the Gluster Developer Summit https://twitter.com/JoeCyberGuru/status/784310321980674048 On 10/27/2016 11:52 AM, Christian Rice wrote:> We have a gluster pool where our developers are writing large files > with the suffix .COPYING, then removing that suffix when the copy is > done (renaming the file), so consumers of the data know the file is > safe to read. > > Back in 2014, Jeff Darcy wrote the below in a thread. I guess I am > basically asking if this is still the supported/encouraged approach, > and how do I implement this when I?m using gluster?s NFS? > > "That said, there's also a trick you can use to avoid creation of a > linkfile. Other tools, such as rsync and our own object interface, > use the same write-then-rename idiom. To serve them, there's an > option called extra-hash-regex that can be used to place files on the > "right" brick according to their final name even though they're created > with another. Unfortunately, specifying that option via the command line > doesn't seem to work (it creates a malformed volfile) so you have to > mount a bit differently. For example: > > glusterfs --volfile-server=a_server --volfile-id=a_volume \ > --xlator-option a_volume-dht.extra_hash_regex='(.*+)tmp' \ > /a/mountpoint > > The important part is that second line. That causes any file with a > "tmp" suffix to be hashed and placed as though only the part in the > first parenthesized part of the regex (i.e. without the "tmp") was > there. Therefore, creating "xxxtmp" and then renaming it to "xxx" is > the same as just creating "xxx" in the first place as far as linkfiles > etc. are concerned. Note that the excluded part can be anything that > a regex can match, including a unique random number. If I recall, > rsync uses temp files something like this: > > fubar = .fubar.NNNNNN (where NNNNNNN is a random number) > > I know this probably seems a little voodoo-ish, but with a little bit > of experimentation to find the right regex you should be able to avoid > those dreaded linkfiles altogether." > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161027/f36be574/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2016-10-27 13-22-04.png Type: image/png Size: 114132 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161027/f36be574/attachment.png>