Gluster users - Jan 2009 - Re : problem with running mysql on glusterfs

Hello. I would like to ask about having mysql data hosted on glusterfs.
Please see my issue below. I have posted DEBUG log information on pastebin.

I am using

GlusterFS version: glusterfs 1.4.0rc7

FUSE Version     :fuse-2.7.3glfs10

my ISSUE is:
When hosting mysql data on glusterfs I have an issue:

The first time I start the glusterfsd server using

[root at mohan ~]#  /usr/local/sbin/glusterfsd -f
/usr/local/etc/glusterfs/glusterfs-server.vol.sample

and mount the glusterfs client (/var/lib/mysql) mountpoint  using

[root at localhost ~]# /usr/local/sbin/glusterfs -f
/usr/local/etc/glusterfs/glusterfs-client.vol.sample /var/lib/mysql

Ok fine . then i am trying to login it through error

[root at localhost ~]# service mysqld status
mysqld (pid 1700) is running...

[root at localhost ~]# mysql -u root
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)

Then I am unmounted using

[root at localhost ~]# umount /var/lib/mysql/

Again mounted using

[root at localhost ~]# /usr/local/sbin/glusterfs -f
/usr/local/etc/glusterfs/glusterfs-client.vol.sample /var/lib/mysql


i am getting the same problem like

[root at localhost ~]# mysql -u root
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)


then I checked log ,there is no any error

[root at localhost ~]# cat /var/log/mysqld.log


I stoped and start mysqld in client machine using

[root at localhost ~]# service mysqld stop
Stopping MySQL:                                            [  OK  ]

[root at localhost ~]# service mysqld start
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL:                                            [FAILED]

Then i am trying to unmount ,it tells device id busy.but I am not using this
directory

[root at localhost ~]# umount /var/lib/mysql/
umount: /var/lib/mysql: device is busy
umount: /var/lib/mysql: device is busy

again tryed to login mysql ,still it gives the error

[root at localhost ~]# mysql -u root
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)


Then Again I am started mysqld . lost time this was failed .now it is
started

[root at localhost ~]# service mysqld start
Starting MySQL:                                            [  OK  ]

Then I tryed to login . Now i am able to login .everthing is fine .

[root at localhost ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.51a Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the
buffer.

mysql>


Now I am getting log file it has error like this .

[root at localhost ~]# cat /var/log/mysqld.log
090120 20:52:20  mysqld ended

090120 20:52:29  mysqld started
InnoDB: Unable to lock ./ibdata1, error: 38
090120 20:52:29  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
090120 20:53:42  mysqld started
InnoDB: Unable to lock ./ibdata1, error: 38
090120 20:53:42  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
090120 20:54:09  InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
090120 20:54:09  InnoDB: Operating system error number 38 in a file
operation.
InnoDB: Error number 38 means 'Function not implemented'.
InnoDB: Some operating system error numbers are described at
InnoDB:
http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
090120 20:54:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.51a'  socket: '/var/lib/mysql/mysql.sock'  port:
3306  Source
distribution
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38



NOTE : I am using load posix-locks translator in my server vol file like
this


volume brick
  type storage/posix                   # POSIX FS translator
  option directory /mnt/mymysql/mysql       #bench        # Export this
directory
end-volume


volume plocks
  type features/posix-locks
  subvolumes brick
 # option mandatory on
end-volume


http://www.mail-archive.com/gluster-devel at nongnu.org/msg03125.html

some one faced like this ,he is using different version  .it was solved .
can any one please help me ? I am using

FUSE Version     :fuse-2.7.3glfs10
GlusterFS version: glusterfs 1.4.0rc7

 Thanks for your time

   L. Mohan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090120/c7f29e0d/attachment.html>

At 02:40 AM 1/20/2009, mohan L wrote:
>Hello. I would like to ask about having mysql data hosted on 
>glusterfs. Please see my issue below. I have posted DEBUG log 
>information on pastebin.
>
>
>[root at localhost ~]# cat /var/log/mysqld.log
>090120 20:52:20  mysqld ended
>
>090120 20:52:29  mysqld started
>InnoDB: Unable to lock ./ibdata1, error: 38
>090120 20:52:29  InnoDB: Retrying to lock the first data file
>
>
>NOTE : I am using load posix-locks translator in my server vol file like
this
>
>
>volume plocks
>   type features/posix-locks
>   subvolumes brick
>  # option mandatory on
>end-volume
un comment the option mandatory on and see if that solves your problem.
>some one faced like this ,he is using different version  .it was 
>solved . can any one please help me ? I am using
>
>FUSE Version     :fuse-2.7.3glfs10
>GlusterFS version: glusterfs 1.4.0rc7
clearly form the logs it''s a locking related problem so if that 
doesn''t solve the problem, then someone will have to look deeper.

At 02:40 AM 1/20/2009, mohan L wrote:
>Hello. I would like to ask about having mysql data hosted on 
>glusterfs. Please see my issue below. I have posted DEBUG log 
>information on pastebin.
>
>
>[root at localhost ~]# cat /var/log/mysqld.log
>090120 20:52:20  mysqld ended
>
>090120 20:52:29  mysqld started
>InnoDB: Unable to lock ./ibdata1, error: 38
>090120 20:52:29  InnoDB: Retrying to lock the first data file
>
>
>NOTE : I am using load posix-locks translator in my server vol file like
this
>
>
>volume plocks
>   type features/posix-locks
>   subvolumes brick
>  # option mandatory on
>end-volume
un comment the option mandatory on and see if that solves your problem.
>some one faced like this ,he is using different version  .it was 
>solved . can any one please help me ? I am using
>
>FUSE Version     :fuse-2.7.3glfs10
>GlusterFS version: glusterfs 1.4.0rc7
clearly form the logs it's a locking related problem so if that 
doesn't solve the problem, then someone will have to look deeper.

> Please see my issue below. I have posted DEBUG log information on pastebin.
Can you give a link to your post?

> volume brick
>   type storage/posix                   # POSIX FS translator
>   option directory /mnt/mymysql/mysql       #bench        # Export this
> directory
> end-volume
>
>
> volume plocks
>   type features/posix-locks
>   subvolumes brick
>  # option mandatory on
> end-volume
>
have you set the remote-subvolume on the client to 'plocks' and not
'brick' ? Please post your complete volfiles (on pastebin if they are
too big) and not just snippets.

avati