Guido Barosio
2007-Jun-06 01:46 UTC
[Xen-users] file system filled, xen won''t start giving confusing errors
Hello out there,
We had a problem while rebooting one of our xen boxes. The case
showed us that while trying to `xm create ...` the xm command will
exit with "err 2, no such file or directory". But `stat vm.conf`
showed that the file was there.
We realized that the file system was full, and removed some space.
However, now with free space, xm refused to start domains.
The solution came by rebooting the box, and starting again the vm''s
by hand.
While I reccon our foult on letting the / get filled, I don''t
understand the error message stderr''ored by xm, "no such file or
directory". Pretty confusing.
xen 3.03, dell 1950 with a big ram support and raid 1.
Is this a known problem? How can we help getting this sorted?
Thanks for this great app, and for the time spent on this mail, in advance.
Kind regards,
--
Guido Barosio
-----------------------
http://www.globant.com
guido.barosio@globant.com
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Guido Barosio
2007-Jun-06 21:43 UTC
[Xen-devel] Fwd: file system filled, xen won''t start giving confusing errors
---------- Forwarded message ----------
From: Guido Barosio <gbarosio@gmail.com>
Date: Jun 5, 2007 10:46 PM
Subject: file system filled, xen won''t start giving confusing errors
To: xen-users@lists.xensource.com
Hello out there,
We had a problem while rebooting one of our xen boxes. The case
showed us that while trying to `xm create ...` the xm command will
exit with "err 2, no such file or directory". But `stat vm.conf`
showed that the file was there.
We realized that the file system was full, and removed some space.
However, now with free space, xm refused to start domains.
The solution came by rebooting the box, and starting again the vm''s
by hand.
While I reccon our foult on letting the / get filled, I don''t
understand the error message stderr''ored by xm, "no such file or
directory". Pretty confusing.
xen 3.03, dell 1950 with a big ram support and raid 1.
Is this a known problem? How can we help getting this sorted?
Thanks for this great app, and for the time spent on this mail, in advance.
Kind regards,
--
Guido Barosio
-----------------------
http://www.globant.com
guido.barosio@globant.com
--
Guido Barosio
-----------------------
http://www.globant.com
guido.barosio@globant.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Mark Williamson
2007-Jun-08 16:29 UTC
Re: [Xen-devel] Fwd: file system filled, xen won''t start giving confusing errors
> We had a problem while rebooting one of our xen boxes. The case > showed us that while trying to `xm create ...` the xm command will > exit with "err 2, no such file or directory". But `stat vm.conf` > showed that the file was there. > > We realized that the file system was full, and removed some space. > However, now with free space, xm refused to start domains. > > The solution came by rebooting the box, and starting again the vm''s by > hand.Well diagnosed!> While I reccon our foult on letting the / get filled, I don''t > understand the error message stderr''ored by xm, "no such file or > directory". Pretty confusing.Indeed. I don''t suppose you took a copy of the xend.log / xend-debug.log from when the failure happened?> xen 3.03, dell 1950 with a big ram support and raid 1. > > Is this a known problem? How can we help getting this sorted?I don''t think it was know before; thank you very much for reporting it. I imagine Xen was trying to do some kind of IO in the process of starting the domain, it was failing due to the filesystem being full, and it was being misreported as "file not found". I guess it''s possible the tools were *actually* trying (and failing silently) to create a file somewhere, and then not being able to find it subsequently. This should (fingers crossed) be easy to replicate, though. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Guido Barosio
2007-Jun-08 17:30 UTC
Re: [Xen-devel] Fwd: file system filled, xen won''t start giving confusing errors
Mark, Thanks for your reply. We are heavily using xen @ globant, and we are extremely happy over other products. I''ve found a sequence that I''ve saved for explaining in the house the problem but I didn''t save the logs nor a "df -h" screen. Here is what I have: [root@xenhq01 xen]# xm create vm08_tartool.conf vmid=8 Using config file "vm08_tartool.conf". Error: (2, ''No such file or directory'') [root@xenhq01 xen]# stat vm08_tartool.conf File: `vm08_tartool.conf'' Size: 667 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 356905 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2007-06-05 22:13:05.000000000 -0300 Modify: 2007-05-22 21:40:10.000000000 -0300 Change: 2007-05-22 21:40:10.000000000 -0300 xm create /etc/xen/vm31_workflows vmid=31 [root@xenhq01 xen]# xm create vm08_tartool.conf vmid=8 Using config file "vm08_tartool.conf". Error: (2, ''No such file or directory'') [root@xenhq01 xen]# xm create /etc/xen/vm08_tartool.conf vmid=8 Using config file "/etc/xen/vm08_tartool.conf". Error: (2, ''No such file or directory'') [root@xenhq01 xen]# xm create vm17_CablescopeCVS vmid=17 Using config file "vm17_CablescopeCVS". Error: (2, ''No such file or directory'') I am actually trying to replicate the case on one of the dev boxes. I assume that tracing this starts with xm.py and the import''ed utils through the code. That''s right or a complete loose of time? Best regards, gb.- On 6/8/07, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:> > We had a problem while rebooting one of our xen boxes. The case > > showed us that while trying to `xm create ...` the xm command will > > exit with "err 2, no such file or directory". But `stat vm.conf` > > showed that the file was there. > > > > We realized that the file system was full, and removed some space. > > However, now with free space, xm refused to start domains. > > > > The solution came by rebooting the box, and starting again the vm''s by > > hand. > > Well diagnosed! > > > While I reccon our foult on letting the / get filled, I don''t > > understand the error message stderr''ored by xm, "no such file or > > directory". Pretty confusing. > > Indeed. I don''t suppose you took a copy of the xend.log / xend-debug.log from > when the failure happened? > > > xen 3.03, dell 1950 with a big ram support and raid 1. > > > > Is this a known problem? How can we help getting this sorted? > > I don''t think it was know before; thank you very much for reporting it. > > I imagine Xen was trying to do some kind of IO in the process of starting the > domain, it was failing due to the filesystem being full, and it was being > misreported as "file not found". I guess it''s possible the tools were > *actually* trying (and failing silently) to create a file somewhere, and then > not being able to find it subsequently. > > This should (fingers crossed) be easy to replicate, though. > > Cheers, > Mark > > -- > Dave: Just a question. What use is a unicyle with no seat? And no pedals! > Mark: To answer a question with a question: What use is a skateboard? > Dave: Skateboards have wheels. > Mark: My wheel has a wheel! >-- Guido Barosio ----------------------- http://www.globant.com guido.barosio@globant.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel