hey guys,
I need to mount a different volume onto /var so we have more room to
breathe. I'll be turning 3 servers into an elasticsearch cluster. And for
some reason when the servers were ordered the large local volume ended up
being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and
I'm going swap the large local volume from /usr to /var.
It looked like /opt had more than enough space to hold both directories.
/opt was 6GB and I successfully synced /usr to it. /usr was 2.5GB.
Then I went to sync /var to a temp folder in /opt. Checking I see that it
still has 1/6GB available after the first sync.
# df -h /opt
Filesystem Size Used *Avail* Use% Mounted on
/dev/mapper/SysVG-OptVol
6.0G 4.1G *1.6G* 72% /opt
The entire /var partition is only using 549MB:
# df -h /var
Filesystem Size *Used* Avail Use% Mounted on
/dev/mapper/SysVG-VarVol
6.0G *549M* 5.1G 10% /var
So that being the case, if I make a temp directory in /opt called /opt/var,
how come I am running out of space in doing my rsync? It fails at the end
and the /opt volume is filled up to 100%. Even tho I only have 549MB to
sync.
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
rsync: write failed on "/opt/var/log/lastlog": No space left on device
(28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: recv_generator: mkdir "/opt/var/www/manual/developer" failed:
No
space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/opt/var/www/manual/faq" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/opt/var/www/manual/howto" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/opt/var/www/manual/images" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/opt/var/www/manual/misc" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/opt/var/www/manual/mod" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: connection unexpectedly closed (148727 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[sender=3.0.6]
And if I do a df of the entire system, it looks like everything is still ok:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/SysVG-RootVol
2.0G 872M 1.1G 46% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 486M 87M 375M 19% /boot
/dev/mapper/SysVG-HomeVol
4.0G 137M 3.7G 4% /home
/dev/mapper/SysVG-OptVol
6.0G 4.3G 1.4G 76% /opt
/dev/mapper/SysVG-TmpVol
2.0G 130M 1.8G 7% /tmp
/dev/mapper/SysVG-UsrVol
197G 2.8G 185G 2% /usr
/dev/mapper/SysVG-VarVol
6.0G 549M 5.1G 10% /var
Does anyone have a good guess as to why these 'out of space' failures
are
occurring?
Thanks,
Tim
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
On Wed, 24 Jun 2015 12:42:19 -0400 Tim Dunphy wrote:> Does anyone have a good guess as to why these 'out of space' failures > are occurring?Hi Tim, At first glance, I don't see anything obvious, but if it were me, I'd do the following: a) add the 'n' flag to do a dry run (no actual copying) b) increase rsync's verbosity (A single -v will give you information about what files are being transferred and a brief summary at the end. Two -v options (-vv) will give you information on what files are being skipped and slightly more information at the end. A third 'v' is insanely verbose.) c) redirect standard out to a text file that you can examine for more clues. hth & regards, Carl
On Wed, 24 Jun 2015 12:42:19 -0400 Tim Dunphy wrote:> how come I am running out of space in doing my rsync?Are you running out of room for file and directory names, which is a different thing than simple free disk space? -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
Hey Carl, Hi Tim,> At first glance, I don't see anything obvious, but if it were me, I'd > do the following: > a) add the 'n' flag to do a dry run (no actual copying) > b) increase rsync's verbosity > (A single -v will give you information about what files are being > transferred and a brief summary at the end. Two -v options (-vv) > will give you information on what files are being skipped and > slightly more information at the end. A third 'v' is insanely > verbose.) > c) redirect standard out to a text file that you can examine for more > clues. > hth & regards, >Good suggestions! Thanks! Tim On Wed, Jun 24, 2015 at 1:05 PM, Carl E. Hartung <carlh04426 at gmail.com> wrote:> On Wed, 24 Jun 2015 12:42:19 -0400 > Tim Dunphy wrote: > > > Does anyone have a good guess as to why these 'out of space' failures > > are occurring? > > Hi Tim, > > At first glance, I don't see anything obvious, but if it were me, I'd > do the following: > > a) add the 'n' flag to do a dry run (no actual copying) > > b) increase rsync's verbosity > (A single -v will give you information about what files are being > transferred and a brief summary at the end. Two -v options (-vv) > will give you information on what files are being skipped and > slightly more information at the end. A third 'v' is insanely > verbose.) > > c) redirect standard out to a text file that you can examine for more > clues. > > hth & regards, > > Carl > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> Does anyone have a good guess as to why these 'out of space' failures > are > occurring?Probaly sparse files or hard links? Try # rsync -aHASWXv --delete src/ dst/
On 06/24/2015 09:42 AM, Tim Dunphy wrote:> And for > some reason when the servers were ordered the large local volume ended up > being /usr when the ES rpm likes to store it's indexes on /var. > > So I'm syncing the contents of both directories to a different place, and > I'm going swap the large local volume from /usr to /var.Have you considered just resizing the volumes? If you're trying to swap them with rsync, you're going to have to reboot anyway, and relabel your system. If any daemons are running, you might also corrupt their data this way.> The entire /var partition is only using 549MB: > > rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)Depending on what UIDs are allocated to your users, lastlog can be an enormous sparse file. You would need to use rsync's -S flag to copy it.
Gordon Messmer wrote:> On 06/24/2015 09:42 AM, Tim Dunphy wrote: >> And for some reason when the servers were ordered the large >> local volume ended up being /usr when the ES rpm likes to >> store it's indexes on /var. >> >> So I'm syncing the contents of both directories to a different place, >> and I'm going swap the large local volume from /usr to /var. > > Have you considered just resizing the volumes? If you're trying to swap > them with rsync, you're going to have to reboot anyway, and relabel your > system. If any daemons are running, you might also corrupt their data > this way. > >> The entire /var partition is only using 549MB: >> >> rsync: write failed on "/opt/var/log/lastlog": No space left on device >> (28) > > Depending on what UIDs are allocated to your users, lastlog can be an > enormous sparse file. You would need to use rsync's -S flag to copy it.Um, I've not been following this closely, but /var is 549M? And a separate partition? I haven't had /var and /usr as separate partitions in almost 10 years. Nor have I had a drive smaller than, um, 160G in about the same. That being said, why not simply mount the additional partition under var, for the directory that's running out of space? mark
Hello Tim, On 06/24/2015 07:42 PM, Tim Dunphy wrote:> rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: > Broken pipe (32) > rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)lastlog is a VERY large SPARSE file and when you rsync it it looses the sparsity and tries to copy all the data to /opt ls -al -h /var/log/lastlog -rw-r--r--. 1 root root *94G* Jun 25 09:10 /var/log/lastlog Real space on disk du -h /var/log/lastlog *60K* /var/log/lastlog Lec
> > Have you considered just resizing the volumes?That'd probably be my preference. But in my role at this company I don't have the direct access to do that. I'd probably have to open up a ticket to another department and have it done when 'they get around to it'. In say 3 or 4 weeks. On my own servers no sweat. But at work. nah. not really practical. Thanks for the suggestion anyway! On Wed, Jun 24, 2015 at 2:33 PM, Gordon Messmer <gordon.messmer at gmail.com> wrote:> On 06/24/2015 09:42 AM, Tim Dunphy wrote: > >> And for >> some reason when the servers were ordered the large local volume ended up >> being /usr when the ES rpm likes to store it's indexes on /var. >> >> So I'm syncing the contents of both directories to a different place, and >> I'm going swap the large local volume from /usr to /var. >> > > Have you considered just resizing the volumes? If you're trying to swap > them with rsync, you're going to have to reboot anyway, and relabel your > system. If any daemons are running, you might also corrupt their data this > way. > > The entire /var partition is only using 549MB: >> >> rsync: write failed on "/opt/var/log/lastlog": No space left on device >> (28) >> > > Depending on what UIDs are allocated to your users, lastlog can be an > enormous sparse file. You would need to use rsync's -S flag to copy it. > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B