I'm running 1.4rc7 (glusterfs--mainline--3.0--patch-814) and seeing some odd behavior when tailing a file that is being written to by a single process -- a log file in this case. The odd behaviors that I've noticed are that "tail -f" behaves like "tail" and doesn't show any updates. In addition, /usr/bin/less seems to show binary values (at least that's what I assume the "^@" characters are supposed to be) when the bottom of the file "G" accessed instead of the new data added to the file after less was started. Is this a known issue? Is there a work-around? Here's how I'm able to reproduce it. Run the script below and direct the output to a gluster-hosted file. Then attempt to "tail -f" or use /usr/bin/less on the file from another terminal. $> num=0; while [ 1 ]; do echo $((num++)); sleep 1; done > /mnt/ gluster/nums.txt The output from /usr/bin/less ends up looking like this: ... 301 302 303 304 305 306 307 308 309 310 311 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Gluster configs are very basic: ## Server volume brick type storage/posix option directory /glusterfs/export end-volume volume lock type features/posix-locks subvolumes brick end-volume volume export type performance/io-threads subvolumes lock option thread-count 4 # default value is 1 end-volume volume server type protocol/server option transport-type tcp subvolumes export option auth.addr.export.allow 10.10.10.* end-volume ## Client volume volume1 type protocol/client option transport-type tcp/client option remote-host 10.10.10.2 option remote-subvolume export end-volume volume volume2 type protocol/client option transport-type tcp/client option remote-host 10.10.10.3 option remote-subvolume export end-volume volume mirror1 type cluster/afr subvolumes volume1 volume2 end-volume -Bryan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090112/eb70cb6f/attachment.html>
this is interesting. I''m running glusterfs--mainline--3.0--patch-840 using AFR. I did your test on the local machine, running tail does exactly what you indicated... it acts like it was run without the -f. on the other replication server, lines show up 2 at a time. so it started at 9 or something then I got 10 & 11, 2 seconds later, 12, 13, etc... the same time tail -f on the server I was running the thing on sat there a while then produced some output. again, the afr machine updated more frequently, but the local machine just needed to get past some buffering what I saw was (you''ll notice this faster if you remove the sleep) it would show some numbers, then jump and show another batch of numbers, then pause then show another batch. here''s an output from tail -f Notice that at 63, there was some weirdness, like it was trying to print 1041 and 1860, I''m guessing then I got 1861 -.... and then I''d get a jump in numbers.. if I cat the file all the numbers are there. Also, in some of my tests I got input/output errors ---- I belive this was due to having tail -f running on the other afr server and this code you provided using > which truncates the file. seems AFR has a little bug there if a file is open for reading on the other server and is truncated. the io error went away when I killed the tail process on the other machine. 55 56 57 58 59 60 61 62 63 041 60 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 I also noticed At 02:25 PM 1/12/2009, Bryan Talbot wrote:>I''m running 1.4rc7 (glusterfs--mainline--3.0--patch-814) and seeing >some odd behavior when tailing a file that is being written to by a >single process -- a log file in this case. > >The odd behaviors that I''ve noticed are that "tail -f" behaves like >"tail" and doesn''t show any updates. In addition, /usr/bin/less >seems to show binary values (at least that''s what I assume the "^@" >characters are supposed to be) when the bottom of the file "G" >accessed instead of the new data added to the file after less was started. > >Is this a known issue? Is there a work-around? > >Here''s how I''m able to reproduce it. Run the script below and >direct the output to a gluster-hosted file. Then attempt to "tail >-f" or use /usr/bin/less on the file from another terminal. > > >$> num=0; while [ 1 ]; do echo $((num++)); sleep 1; done > >/mnt/gluster/nums.txt > > >The output from /usr/bin/less ends up looking like this: >... >301 >302 >303 >304 >305 >306 >307 >308 >309 >310 >311 >^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > > >Gluster configs are very basic: >## Server >volume brick > type storage/posix > option directory /glusterfs/export >end-volume > >volume lock > type features/posix-locks > subvolumes brick >end-volume > >volume export > type performance/io-threads > subvolumes lock > option thread-count 4 # default value is 1 >end-volume > >volume server > type protocol/server > option transport-type tcp > subvolumes export > option auth.addr.export.allow 10.10.10.* >end-volume > > > >## Client >volume volume1 > type protocol/client > option transport-type tcp/client > option remote-host 10.10.10.2 > option remote-subvolume export >end-volume > >volume volume2 > type protocol/client > option transport-type tcp/client > option remote-host 10.10.10.3 > option remote-subvolume export >end-volume > >volume mirror1 > type cluster/afr > subvolumes volume1 volume2 >end-volume > > > >-Bryan > > > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
this is interesting. I'm running glusterfs--mainline--3.0--patch-840 using AFR. I did your test on the local machine, running tail does exactly what you indicated... it acts like it was run without the -f. on the other replication server, lines show up 2 at a time. so it started at 9 or something then I got 10 & 11, 2 seconds later, 12, 13, etc... the same time tail -f on the server I was running the thing on sat there a while then produced some output. again, the afr machine updated more frequently, but the local machine just needed to get past some buffering what I saw was (you'll notice this faster if you remove the sleep) it would show some numbers, then jump and show another batch of numbers, then pause then show another batch. here's an output from tail -f Notice that at 63, there was some weirdness, like it was trying to print 1041 and 1860, I'm guessing then I got 1861 -.... and then I'd get a jump in numbers.. if I cat the file all the numbers are there. Also, in some of my tests I got input/output errors ---- I belive this was due to having tail -f running on the other afr server and this code you provided using > which truncates the file. seems AFR has a little bug there if a file is open for reading on the other server and is truncated. the io error went away when I killed the tail process on the other machine. 55 56 57 58 59 60 61 62 63 041 60 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 I also noticed At 02:25 PM 1/12/2009, Bryan Talbot wrote:>I'm running 1.4rc7 (glusterfs--mainline--3.0--patch-814) and seeing >some odd behavior when tailing a file that is being written to by a >single process -- a log file in this case. > >The odd behaviors that I've noticed are that "tail -f" behaves like >"tail" and doesn't show any updates. In addition, /usr/bin/less >seems to show binary values (at least that's what I assume the "^@" >characters are supposed to be) when the bottom of the file "G" >accessed instead of the new data added to the file after less was started. > >Is this a known issue? Is there a work-around? > >Here's how I'm able to reproduce it. Run the script below and >direct the output to a gluster-hosted file. Then attempt to "tail >-f" or use /usr/bin/less on the file from another terminal. > > >$> num=0; while [ 1 ]; do echo $((num++)); sleep 1; done > >/mnt/gluster/nums.txt > > >The output from /usr/bin/less ends up looking like this: >... >301 >302 >303 >304 >305 >306 >307 >308 >309 >310 >311 >^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > > >Gluster configs are very basic: >## Server >volume brick > type storage/posix > option directory /glusterfs/export >end-volume > >volume lock > type features/posix-locks > subvolumes brick >end-volume > >volume export > type performance/io-threads > subvolumes lock > option thread-count 4 # default value is 1 >end-volume > >volume server > type protocol/server > option transport-type tcp > subvolumes export > option auth.addr.export.allow 10.10.10.* >end-volume > > > >## Client >volume volume1 > type protocol/client > option transport-type tcp/client > option remote-host 10.10.10.2 > option remote-subvolume export >end-volume > >volume volume2 > type protocol/client > option transport-type tcp/client > option remote-host 10.10.10.3 > option remote-subvolume export >end-volume > >volume mirror1 > type cluster/afr > subvolumes volume1 volume2 >end-volume > > > >-Bryan > > > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
Same on 1.3.12: tail -f doesn't -f :
glusterfs 1.3.12
Repository revision: glusterfs--mainline--2.5--patch-797
Here's something interesting though -- it is actually working, it is just
reading nulls instead of actual data. Here is the transition from '8',
'9'
to '10', '11':
strace tail -f nums.txt # .......
nanosleep({1, 0}, NULL) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=422, ...}) = 0
read(3, "\0\0", 8192) = 2
read(3, "", 8192) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=422, ...}) = 0
write(1, "\0\0", 2) = 2
nanosleep({1, 0}, NULL) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=425, ...}) = 0
read(3, "\0\0\0", 8192) = 3
read(3, "", 8192) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=425, ...}) = 0
write(1, "\0\0\0", 3) = 3
nanosleep({1, 0}, NULL) = 0
&:-)
On Tuesday 13 January 2009 05:04:53 Keith Freedman
wrote:> this is interesting.
>
> I'm running glusterfs--mainline--3.0--patch-840
>
> using AFR.
> I did your test
> on the local machine, running tail does exactly what you indicated...
> it acts like it was run without the -f.
> on the other replication server, lines show up 2 at a time.
> so it started at 9 or something
> then I got 10 & 11, 2 seconds later, 12, 13, etc...
> the same time tail -f on the server I was running the thing on sat
> there a while then produced some output.
>
> again, the afr machine updated more frequently, but the local machine
> just needed to get past some buffering
>
> what I saw was (you'll notice this faster if you remove the sleep)
> it would show some numbers, then jump and show another batch of
> numbers, then pause then show another batch.
> here's an output from tail -f
> Notice that at 63, there was some weirdness, like it was trying to
> print 1041 and 1860, I'm guessing
> then I got 1861 -.... and then I'd get a jump in numbers.. if I cat
> the file all the numbers are there.
>
> Also, in some of my tests I got input/output errors ---- I belive
> this was due to having tail -f running on the other afr server and
> this code you provided using > which truncates the file. seems AFR
> has a little bug there if a file is open for reading on the other
> server and is truncated. the io error went away when I killed the
> tail process on the other machine.
>
> 55
> 56
> 57
> 58
> 59
> 60
> 61
> 62
> 63
> 041
> 60
> 1861
> 1862
> 1863
> 1864
> 1865
> 1866
> 1867
> 1868
> 1869
> 1870
> 1871
> 1872
>
> I also noticed
>
> At 02:25 PM 1/12/2009, Bryan Talbot wrote:
> >I'm running 1.4rc7 (glusterfs--mainline--3.0--patch-814) and seeing
> >some odd behavior when tailing a file that is being written to by a
> >single process -- a log file in this case.
> >
> >The odd behaviors that I've noticed are that "tail -f"
behaves like
> >"tail" and doesn't show any updates. In addition,
/usr/bin/less
> >seems to show binary values (at least that's what I assume the
"^@"
> >characters are supposed to be) when the bottom of the file
"G"
> >accessed instead of the new data added to the file after less was
started.
> >
> >Is this a known issue? Is there a work-around?
> >
> >Here's how I'm able to reproduce it. Run the script below and
> >direct the output to a gluster-hosted file. Then attempt to "tail
> >-f" or use /usr/bin/less on the file from another terminal.
> >
> >
> >$> num=0; while [ 1 ]; do echo $((num++)); sleep 1; done >
> >/mnt/gluster/nums.txt
> >
> >
> >The output from /usr/bin/less ends up looking like this:
> >...
> >301
> >302
> >303
> >304
> >305
> >306
> >307
> >308
> >309
> >310
> >311
> >^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
> >
> >
> >Gluster configs are very basic:
> >## Server
> >volume brick
> > type storage/posix
> > option directory /glusterfs/export
> >end-volume
> >
> >volume lock
> > type features/posix-locks
> > subvolumes brick
> >end-volume
> >
> >volume export
> > type performance/io-threads
> > subvolumes lock
> > option thread-count 4 # default value is 1
> >end-volume
> >
> >volume server
> > type protocol/server
> > option transport-type tcp
> > subvolumes export
> > option auth.addr.export.allow 10.10.10.*
> >end-volume
> >
> >
> >
> >## Client
> >volume volume1
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host 10.10.10.2
> > option remote-subvolume export
> >end-volume
> >
> >volume volume2
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host 10.10.10.3
> > option remote-subvolume export
> >end-volume
> >
> >volume mirror1
> > type cluster/afr
> > subvolumes volume1 volume2
> >end-volume
> >
> >
> >
> >-Bryan
> >
> >
> >
> >
> >_______________________________________________
> >Gluster-users mailing list
> >Gluster-users at gluster.org
> >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
--
Tel +27 11 234 5045
Fax +27 86 552 8833
Same on 1.3.12: tail -f doesn't -f :
glusterfs 1.3.12
Repository revision: glusterfs--mainline--2.5--patch-797
Here's something interesting though -- it is actually working, it is just
reading nulls instead of actual data. Here is the transition from '8',
'9'
to '10', '11':
strace tail -f nums.txt # .......
nanosleep({1, 0}, NULL) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=422, ...}) = 0
read(3, "\0\0", 8192) = 2
read(3, "", 8192) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=422, ...}) = 0
write(1, "\0\0", 2) = 2
nanosleep({1, 0}, NULL) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=425, ...}) = 0
read(3, "\0\0\0", 8192) = 3
read(3, "", 8192) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=425, ...}) = 0
write(1, "\0\0\0", 3) = 3
nanosleep({1, 0}, NULL) = 0
&:-)
On Tuesday 13 January 2009 05:04:53 Keith Freedman
wrote:> this is interesting.
>
> I'm running glusterfs--mainline--3.0--patch-840
>
> using AFR.
> I did your test
> on the local machine, running tail does exactly what you indicated...
> it acts like it was run without the -f.
> on the other replication server, lines show up 2 at a time.
> so it started at 9 or something
> then I got 10 & 11, 2 seconds later, 12, 13, etc...
> the same time tail -f on the server I was running the thing on sat
> there a while then produced some output.
>
> again, the afr machine updated more frequently, but the local machine
> just needed to get past some buffering
>
> what I saw was (you'll notice this faster if you remove the sleep)
> it would show some numbers, then jump and show another batch of
> numbers, then pause then show another batch.
> here's an output from tail -f
> Notice that at 63, there was some weirdness, like it was trying to
> print 1041 and 1860, I'm guessing
> then I got 1861 -.... and then I'd get a jump in numbers.. if I cat
> the file all the numbers are there.
>
> Also, in some of my tests I got input/output errors ---- I belive
> this was due to having tail -f running on the other afr server and
> this code you provided using > which truncates the file. seems AFR
> has a little bug there if a file is open for reading on the other
> server and is truncated. the io error went away when I killed the
> tail process on the other machine.
>
> 55
> 56
> 57
> 58
> 59
> 60
> 61
> 62
> 63
> 041
> 60
> 1861
> 1862
> 1863
> 1864
> 1865
> 1866
> 1867
> 1868
> 1869
> 1870
> 1871
> 1872
>
> I also noticed
>
> At 02:25 PM 1/12/2009, Bryan Talbot wrote:
> >I'm running 1.4rc7 (glusterfs--mainline--3.0--patch-814) and seeing
> >some odd behavior when tailing a file that is being written to by a
> >single process -- a log file in this case.
> >
> >The odd behaviors that I've noticed are that "tail -f"
behaves like
> >"tail" and doesn't show any updates. In addition,
/usr/bin/less
> >seems to show binary values (at least that's what I assume the
"^@"
> >characters are supposed to be) when the bottom of the file
"G"
> >accessed instead of the new data added to the file after less was
started.
> >
> >Is this a known issue? Is there a work-around?
> >
> >Here's how I'm able to reproduce it. Run the script below and
> >direct the output to a gluster-hosted file. Then attempt to "tail
> >-f" or use /usr/bin/less on the file from another terminal.
> >
> >
> >$> num=0; while [ 1 ]; do echo $((num++)); sleep 1; done >
> >/mnt/gluster/nums.txt
> >
> >
> >The output from /usr/bin/less ends up looking like this:
> >...
> >301
> >302
> >303
> >304
> >305
> >306
> >307
> >308
> >309
> >310
> >311
> >^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
> >
> >
> >Gluster configs are very basic:
> >## Server
> >volume brick
> > type storage/posix
> > option directory /glusterfs/export
> >end-volume
> >
> >volume lock
> > type features/posix-locks
> > subvolumes brick
> >end-volume
> >
> >volume export
> > type performance/io-threads
> > subvolumes lock
> > option thread-count 4 # default value is 1
> >end-volume
> >
> >volume server
> > type protocol/server
> > option transport-type tcp
> > subvolumes export
> > option auth.addr.export.allow 10.10.10.*
> >end-volume
> >
> >
> >
> >## Client
> >volume volume1
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host 10.10.10.2
> > option remote-subvolume export
> >end-volume
> >
> >volume volume2
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host 10.10.10.3
> > option remote-subvolume export
> >end-volume
> >
> >volume mirror1
> > type cluster/afr
> > subvolumes volume1 volume2
> >end-volume
> >
> >
> >
> >-Bryan
> >
> >
> >
> >
> >_______________________________________________
> >Gluster-users mailing list
> >Gluster-users at gluster.org
> >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
> Will there be a gluster optimized version of fuse 2.7.4 > or will we not need a special version in order to support posix locking?We will have glusterfs patches for 2.7.4 soon. It has been delayed for a while now. avati
At 11:50 PM 1/13/2009, Anand Avati wrote:> > Will there be a gluster optimized version of fuse 2.7.4 > > or will we not need a special version in order to support posix locking? > >We will have glusterfs patches for 2.7.4 soon. It has been delayed for >a while now.what do the gluster mods do? as I understood it, it added posix locking. is this still not supported by the default fuse? what are the other improvements? it''d be ideal if we could use the standard fuse, but if it doesn''t do what needs doing, then it''s obviously a problem. Keith
> what do the gluster mods do? > as I understood it, it added posix locking. is this still not supported by > the default fuse? > what are the other improvements?it adds flock() support, O_DIRECT open mode support and some performance enhancments.> it'd be ideal if we could use the standard fuse, but if it doesn't do what > needs doing, then it's obviously a problem.I plan to (from a long time, actually :) submit the patch in a more acceptable form to the mainstream fuse. There are a few hurdles, like, the flock() support is not 100% compliant (works for practically all the cases) and O_DIRECT mode support is only in open but not modifiable through fcntl(). I will try resubmiting the patch again since I was seeing O_DIRECT support request on the fuse-devel list by other users recently. avati