A little while ago, I reported a suspicion that vnodes just weren't being freed up on long running servers ... after 55days of uptime on one of my servers, here is what I'm dealing with ... 793 'samples' today (one every minute) 786 with vnlru in a vlrup state I shutdown all of the VMs running on the large hard drive (the only place unionfs is being used) and umount'd the drive ... there were some suggested back then that this might/should free everything back up again ... but it didn't: Oct 30 13:06:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57966 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:07:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57268 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:08:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 52335 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:09:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 50228 - debug.vnlru_nowhere: 209682 - vlrup Oct 30 13:10:01 venus root: debug.numvnodes: 522265 - debug.freevnodes: 44407 - debug.vnlru_nowhere: 209690 - vlrup Oct 30 13:11:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 35424 - debug.vnlru_nowhere: 209697 - vlrup Oct 30 13:12:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 34626 - debug.vnlru_nowhere: 209708 - vlrup Oct 30 13:13:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 29214 - debug.vnlru_nowhere: 209727 - vlrup Oct 30 13:14:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 24414 - debug.vnlru_nowhere: 209746 - vlrup Oct 30 13:15:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 26994 - debug.vnlru_nowhere: 209766 - vlrup The 'vlruwt' states above are while I had everything shutdown ... the vlrup's all started again after I mounted the drive and started to restart the VMs themselves ... I expect a high # of vnodes to be used ... that isn't the issue ... the issue is that even getting rid of the major mount point, so that only /, /tmp, /usr, /var are left up, the large # of vnodes that are in use on that mount point aren't being freed by vnlru :( I hate to reboot the server, but it looks like I've got no choice at this point ... is there something else that I can do, in 50 days or so, to provide more information? Thanks ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
On Sat, Oct 30, 2004 at 01:20:08PM -0300, Marc G. Fournier wrote:> > A little while ago, I reported a suspicion that vnodes just weren't being > freed up on long running servers ... after 55days of uptime on one of my > servers, here is what I'm dealing with ... > > 793 'samples' today (one every minute) > 786 with vnlru in a vlrup state > > I shutdown all of the VMs running on the large hard drive (the only place > unionfs is being used) and umount'd the drive ... there were some^^^^^^^^^^^^^^^^^^^^^ http://www.FreeBSD.org/cgi/query-pr.cgi?pr=73094> suggested back then that this might/should free everything back up again > ... but it didn't: > > Oct 30 13:06:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 57966 - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:07:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 57268 - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:08:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 52335 - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:09:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 50228 - debug.vnlru_nowhere: 209682 - vlrup > Oct 30 13:10:01 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 44407 - debug.vnlru_nowhere: 209690 - vlrup > Oct 30 13:11:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 35424 - debug.vnlru_nowhere: 209697 - vlrup > Oct 30 13:12:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 34626 - debug.vnlru_nowhere: 209708 - vlrup > Oct 30 13:13:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 29214 - debug.vnlru_nowhere: 209727 - vlrup > Oct 30 13:14:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 24414 - debug.vnlru_nowhere: 209746 - vlrup > Oct 30 13:15:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: > 26994 - debug.vnlru_nowhere: 209766 - vlrup > > The 'vlruwt' states above are while I had everything shutdown ... the > vlrup's all started again after I mounted the drive and started to restart > the VMs themselves ... > > I expect a high # of vnodes to be used ... that isn't the issue ... the > issue is that even getting rid of the major mount point, so that only /, > /tmp, /usr, /var are left up, the large # of vnodes that are in use on > that mount point aren't being freed by vnlru :( > > I hate to reboot the server, but it looks like I've got no choice at this > point ... is there something else that I can do, in 50 days or so, to > provide more information? > > Thanks ... > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664-- NO37-RIPE
Just to give an idea of what a second server, with less uptime, is looking like, with the approx. the same # of VMs on her: Oct 30 13:29:00 neptune root: debug.numvnodes: 462882 - debug.freevnodes: 132826 - debug.vnlru_nowhere: 0 - vlruwt Oct 30 13:30:00 neptune root: debug.numvnodes: 462882 - debug.freevnodes: 151976 - debug.vnlru_nowhere: 0 - vlruwt But she's only been up 7 days so far ... On Sat, 30 Oct 2004, Marc G. Fournier wrote:> > A little while ago, I reported a suspicion that vnodes just weren't being > freed up on long running servers ... after 55days of uptime on one of my > servers, here is what I'm dealing with ... > > 793 'samples' today (one every minute) > 786 with vnlru in a vlrup state > > I shutdown all of the VMs running on the large hard drive (the only place > unionfs is being used) and umount'd the drive ... there were some suggested > back then that this might/should free everything back up again ... but it > didn't: > > Oct 30 13:06:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57966 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:07:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57268 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:08:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 52335 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:09:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 50228 > - debug.vnlru_nowhere: 209682 - vlrup > Oct 30 13:10:01 venus root: debug.numvnodes: 522265 - debug.freevnodes: 44407 > - debug.vnlru_nowhere: 209690 - vlrup > Oct 30 13:11:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 35424 > - debug.vnlru_nowhere: 209697 - vlrup > Oct 30 13:12:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 34626 > - debug.vnlru_nowhere: 209708 - vlrup > Oct 30 13:13:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 29214 > - debug.vnlru_nowhere: 209727 - vlrup > Oct 30 13:14:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 24414 > - debug.vnlru_nowhere: 209746 - vlrup > Oct 30 13:15:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 26994 > - debug.vnlru_nowhere: 209766 - vlrup > > The 'vlruwt' states above are while I had everything shutdown ... the vlrup's > all started again after I mounted the drive and started to restart the VMs > themselves ... > > I expect a high # of vnodes to be used ... that isn't the issue ... the issue > is that even getting rid of the major mount point, so that only /, /tmp, > /usr, /var are left up, the large # of vnodes that are in use on that mount > point aren't being freed by vnlru :( > > I hate to reboot the server, but it looks like I've got no choice at this > point ... is there something else that I can do, in 50 days or so, to provide > more information? > > Thanks ... > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 >---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664