Gluster users - Jan 2009 - Missing files with unify 2.0.0rc1

Hi,

I'm trying to use unify to replace my NFS servers but i have some problems.
In my tests I also see about a quarter of the fiels that I see on NFS.
I also get the following errors on my glusterfsd.log:

Version      : glusterfs 2.0.0rc1 built on Jan 15 2009 00:02:28
TLA Revision : glusterfs--mainline--3.0--patch-844
Starting Time: 2009-01-15 13:50:01
Command line : glusterfsd
given volfile
+-----
  1: volume disk
  2:   type storage/posix
  3:   option directory /homes/davinci
  4: end-volume
  5:
  6: volume disk-rs
  7:   type features/filter
  8:   option root-squashing enable
  9:   subvolumes disk
 10: end-volume
 11:
 12: volume iot
 13:   type performance/io-threads
 14:   subvolumes disk-rs
 15:   option thread-count 4
 16: end-volume
 17:
 18: volume brick
 19:   type performance/write-behind
 20:   subvolumes iot
 21:   option window-size 2MB
 22:   option aggregate-size 1MB
 23: end-volume
 24:
 25: # Volume name is server
 26: volume server
 27:   type protocol/server
 28:   option transport-type tcp
 29:   option auth.addr.brick.allow *
 30:   subvolumes brick
 31: end-volume
+-----

2009-01-15 13:50:01 W [xlator.c:382:validate_xlator_volume_options]
brick: option 'aggregate-size' i
s deprecated, preferred is 'block-size', continuing with correction
2009-01-15 13:50:01 W [xlator.c:382:validate_xlator_volume_options]
brick: option 'window-size' is d
eprecated, preferred is 'cache-size', continuing with correction
2009-01-15 14:01:42 E [socket.c:104:__socket_rwv] server: readv failed
(Connection reset by peer)
2009-01-15 14:01:42 E [socket.c:566:socket_proto_state_machine]
server: socket read failed (Connecti
on reset by peer) in state 1 (192.168.1.235:1020)
2009-01-15 14:04:07 W [posix.c:1042:posix_link] disk: link
/filipe/.Xauthority-n to /filipe/.Xauthor
ity failed: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.dbus: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.dbus/session-bus: File exist
s
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Defaults: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker:
File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/B
ackgrounds: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/I
conSets: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/S
oundSets: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/P
ixmaps: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/Icons: File e
xists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/S
ounds: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/S
tyles: File exists
2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/GNUstep/Library/WindowMaker/T
hemes: File exists
2009-01-15 14:08:57 E [socket.c:104:__socket_rwv] server: readv failed
(Connection reset by peer)
2009-01-15 14:08:57 E [socket.c:566:socket_proto_state_machine]
server: socket read failed (Connecti
on reset by peer) in state 1 (192.168.1.235:1019)
2009-01-15 15:34:44 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.kde: File exists
2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.kde/share: File exists
2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.kde/share/config: File exist
s
2009-01-15 15:34:46 W [posix.c:928:posix_symlink] disk: symlink of
/filipe/.kde/socket-gauguin --> /
tmp/ksocket-filipe: File exists
2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.qt: File exists
2009-01-15 15:34:47 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.kde/share/apps: File exists
2009-01-15 15:35:02 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.mcop: File exists
2009-01-15 15:35:57 W [posix.c:796:posix_mkdir] disk: mkdir of
/filipe/.ssh: File exists
2009-01-15 15:37:58 E [socket.c:104:__socket_rwv] server: readv failed
(Connection reset by peer)
2009-01-15 15:37:58 E [socket.c:566:socket_proto_state_machine]
server: socket read failed (Connecti
on reset by peer) in state 1 (192.168.1.235:1016)
2009-01-15 16:04:32 E [socket.c:104:__socket_rwv] server: writev
failed (Connection reset by peer)
2009-01-15 16:05:16 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD
2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning EBADFD


I don't think I have any hardware problems as I can cat all the files
in my home directory without any problem.

I tried to reproduce the problem with a smaller setup without much
luck unfortunately.

Here is the client file:

volume tintoretto
  type protocol/client
  option transport-type tcp
  option remote-host tintoretto
  option remote-subvolume brick
end-volume

volume giotto
  type protocol/client
  option transport-type tcp
  option remote-host giotto
  option remote-subvolume brick
end-volume

volume michelangelo
  type protocol/client
  option transport-type tcp
  option remote-host michelangelo
  option remote-subvolume brick

volume donatello
  type protocol/client
  option transport-type tcp
  option remote-host donatello
  option remote-subvolume brick
end-volume

volume ns
  type protocol/client
  option transport-type tcp
  option remote-host tintoretto
  option remote-subvolume ns
end-volume

volume bricks
  type cluster/unify
  option namespace ns # this will not be storage child of unify.
  subvolumes tintoretto michelangelo giotto donatello
#  option self-heal foreground # foreground off # default is foreground
#  option self-heal background # foreground off # default is foreground
### ** Round Robin (RR) Scheduler **
  option scheduler rr
# A server is not used if it's free disk space drops below 15%.
  option scheduler.limits.min-free-disk 15% #%
end-volume

volume bricks-rs
  type features/filter
  option root-squashing enable
  subvolumes bricks
end-volume

volume iot
  type performance/io-threads
  subvolumes bricks-rs
  option thread-count 4
end-volume

volume wb
  type performance/write-behind
  subvolumes iot
  option flush-behind off    # default value is 'off'
  option window-size 2MB
  option aggregate-size 1MB # default value is 0
end-volume

### 'IO-Cache' translator is best used on client side when a filesystem
has file
# which are not modified frequently but read several times. For example, while
# compiling a kernel, *.h files are read while compiling every *.c
file, in these
# case, io-cache translator comes very handy, as it keeps the whole
file content in
# the cache, and serves from the cache.
# One can provide the priority of the cache too.

volume ioc
  type performance/io-cache
  subvolumes wb
  option page-size 1MB      # 128KB is default
  option cache-size 64MB    # 32MB is default
  option cache-timeout 5 # 1second is default
  option priority *.c:2,*.h:1 # default is *:0
end-volume


### 'Read-Ahead' translator is best utilized on client side, as it
prefetches
# the file contents when the first read() call is issued.
volume ra
  type performance/read-ahead
  subvolumes ioc
  option page-size 1MB         # default is 256KB
  option page-count 4          # default is 2
  option force-atime-update no # defalut is 'no'
end-volume


Filipe

Filipe,
  how did you populate data into your volumes? was it pre-exising data
or did you copy in all data into a freshly created empty volume?

avati

2009/1/16 Filipe Maia <filipe at xray.bmc.uu.se>:
> Hi,
>
> I'm trying to use unify to replace my NFS servers but i have some
problems.
> In my tests I also see about a quarter of the fiels that I see on NFS.
> I also get the following errors on my glusterfsd.log:
>
> Version      : glusterfs 2.0.0rc1 built on Jan 15 2009 00:02:28
> TLA Revision : glusterfs--mainline--3.0--patch-844
> Starting Time: 2009-01-15 13:50:01
> Command line : glusterfsd
> given volfile
> +-----
>  1: volume disk
>  2:   type storage/posix
>  3:   option directory /homes/davinci
>  4: end-volume
>  5:
>  6: volume disk-rs
>  7:   type features/filter
>  8:   option root-squashing enable
>  9:   subvolumes disk
>  10: end-volume
>  11:
>  12: volume iot
>  13:   type performance/io-threads
>  14:   subvolumes disk-rs
>  15:   option thread-count 4
>  16: end-volume
>  17:
>  18: volume brick
>  19:   type performance/write-behind
>  20:   subvolumes iot
>  21:   option window-size 2MB
>  22:   option aggregate-size 1MB
>  23: end-volume
>  24:
>  25: # Volume name is server
>  26: volume server
>  27:   type protocol/server
>  28:   option transport-type tcp
>  29:   option auth.addr.brick.allow *
>  30:   subvolumes brick
>  31: end-volume
> +-----
>
> 2009-01-15 13:50:01 W [xlator.c:382:validate_xlator_volume_options]
> brick: option 'aggregate-size' i
> s deprecated, preferred is 'block-size', continuing with correction
> 2009-01-15 13:50:01 W [xlator.c:382:validate_xlator_volume_options]
> brick: option 'window-size' is d
> eprecated, preferred is 'cache-size', continuing with correction
> 2009-01-15 14:01:42 E [socket.c:104:__socket_rwv] server: readv failed
> (Connection reset by peer)
> 2009-01-15 14:01:42 E [socket.c:566:socket_proto_state_machine]
> server: socket read failed (Connecti
> on reset by peer) in state 1 (192.168.1.235:1020)
> 2009-01-15 14:04:07 W [posix.c:1042:posix_link] disk: link
> /filipe/.Xauthority-n to /filipe/.Xauthor
> ity failed: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.dbus: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.dbus/session-bus: File exist
> s
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Defaults: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker:
> File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/B
> ackgrounds: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/I
> conSets: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/S
> oundSets: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/P
> ixmaps: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/Icons: File e
> xists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/S
> ounds: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/S
> tyles: File exists
> 2009-01-15 14:04:08 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/GNUstep/Library/WindowMaker/T
> hemes: File exists
> 2009-01-15 14:08:57 E [socket.c:104:__socket_rwv] server: readv failed
> (Connection reset by peer)
> 2009-01-15 14:08:57 E [socket.c:566:socket_proto_state_machine]
> server: socket read failed (Connecti
> on reset by peer) in state 1 (192.168.1.235:1019)
> 2009-01-15 15:34:44 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.kde: File exists
> 2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.kde/share: File exists
> 2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.kde/share/config: File exist
> s
> 2009-01-15 15:34:46 W [posix.c:928:posix_symlink] disk: symlink of
> /filipe/.kde/socket-gauguin --> /
> tmp/ksocket-filipe: File exists
> 2009-01-15 15:34:46 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.qt: File exists
> 2009-01-15 15:34:47 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.kde/share/apps: File exists
> 2009-01-15 15:35:02 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.mcop: File exists
> 2009-01-15 15:35:57 W [posix.c:796:posix_mkdir] disk: mkdir of
> /filipe/.ssh: File exists
> 2009-01-15 15:37:58 E [socket.c:104:__socket_rwv] server: readv failed
> (Connection reset by peer)
> 2009-01-15 15:37:58 E [socket.c:566:socket_proto_state_machine]
> server: socket read failed (Connecti
> on reset by peer) in state 1 (192.168.1.235:1016)
> 2009-01-15 16:04:32 E [socket.c:104:__socket_rwv] server: writev
> failed (Connection reset by peer)
> 2009-01-15 16:05:16 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
> 2009-01-15 16:05:17 E [write-behind.c:1150:wb_flush] brick: returning
EBADFD
>
>
> I don't think I have any hardware problems as I can cat all the files
> in my home directory without any problem.
>
> I tried to reproduce the problem with a smaller setup without much
> luck unfortunately.
>
> Here is the client file:
>
> volume tintoretto
>  type protocol/client
>  option transport-type tcp
>  option remote-host tintoretto
>  option remote-subvolume brick
> end-volume
>
> volume giotto
>  type protocol/client
>  option transport-type tcp
>  option remote-host giotto
>  option remote-subvolume brick
> end-volume
>
> volume michelangelo
>  type protocol/client
>  option transport-type tcp
>  option remote-host michelangelo
>  option remote-subvolume brick
>
> volume donatello
>  type protocol/client
>  option transport-type tcp
>  option remote-host donatello
>  option remote-subvolume brick
> end-volume
>
> volume ns
>  type protocol/client
>  option transport-type tcp
>  option remote-host tintoretto
>  option remote-subvolume ns
> end-volume
>
> volume bricks
>  type cluster/unify
>  option namespace ns # this will not be storage child of unify.
>  subvolumes tintoretto michelangelo giotto donatello
> #  option self-heal foreground # foreground off # default is foreground
> #  option self-heal background # foreground off # default is foreground
> ### ** Round Robin (RR) Scheduler **
>  option scheduler rr
> # A server is not used if it's free disk space drops below 15%.
>  option scheduler.limits.min-free-disk 15% #%
> end-volume
>
> volume bricks-rs
>  type features/filter
>  option root-squashing enable
>  subvolumes bricks
> end-volume
>
> volume iot
>  type performance/io-threads
>  subvolumes bricks-rs
>  option thread-count 4
> end-volume
>
> volume wb
>  type performance/write-behind
>  subvolumes iot
>  option flush-behind off    # default value is 'off'
>  option window-size 2MB
>  option aggregate-size 1MB # default value is 0
> end-volume
>
> ### 'IO-Cache' translator is best used on client side when a
filesystem has file
> # which are not modified frequently but read several times. For example,
while
> # compiling a kernel, *.h files are read while compiling every *.c
> file, in these
> # case, io-cache translator comes very handy, as it keeps the whole
> file content in
> # the cache, and serves from the cache.
> # One can provide the priority of the cache too.
>
> volume ioc
>  type performance/io-cache
>  subvolumes wb
>  option page-size 1MB      # 128KB is default
>  option cache-size 64MB    # 32MB is default
>  option cache-timeout 5 # 1second is default
>  option priority *.c:2,*.h:1 # default is *:0
> end-volume
>
>
> ### 'Read-Ahead' translator is best utilized on client side, as it
prefetches
> # the file contents when the first read() call is issued.
> volume ra
>  type performance/read-ahead
>  subvolumes ioc
>  option page-size 1MB         # default is 256KB
>  option page-count 4          # default is 2
>  option force-atime-update no # defalut is 'no'
> end-volume
>
>
> Filipe
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>

> I *think* (and I'm sure Anand will correct me if I'm wrong), in the
case of
> a single brick, gluster doesn't care much about it's extended
attributes,
> but once another brick/node is involved (in unify or replicate), then the
> extended attributes become important.
>
> pre-existing data is generally without Xattr's and so this causes
confusion.
> My guess is, adding the extended attributes (which happens when you copy
the
> file into the gluster mountpoint) would solve the problem.
This is generally true, but in this situation he is using unify (which
does not use xattrs on the backend) and most of the other cases, if
the xattr is missing, the appropriate self heal will atleast try to
set the most sensible xattr values when the file is first looked up.

Avati