CentOS - Oct 2014 - rsync question: building list taking forever

Guys,

 I've setup an rsync between two directories that I've mounted locally
on a
jump box. Long story short, the two directories are both NFS shares from
two different hosts. Our security dept won't allow us to SSH between the
two data centers, directly. But the jump host can contact both. So what
I've done is mount the NFS shares from one host in each data center on the
jump box using sshfs.

The directory I'm trying to rsync from has 111GB of data in it. I don't
think I've ever setup an rsync for quite so much data before.

 But I started the rsync at approx. 7pm last night. And as of now the rsync
is still building it's file list.

[root at sshproxygw ~]# rsync -avzp /mnt/db_space/timd/www1/
/mnt/db_space/timd/www2/svn2/
building file list ...

So my question to you is, is this a normal amount of time to wait for the
file list to be built? Considering the amount of data involved. I have it
running in a screen session I can attach to to find out what's going on.

Thanks
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

2014-10-19 18:55 GMT+03:00 Tim Dunphy <bluethundr at gmail.com>:
> Guys,
>
>  I've setup an rsync between two directories that I've mounted
locally on a
> jump box. Long story short, the two directories are both NFS shares from
> two different hosts. Our security dept won't allow us to SSH between
the
> two data centers, directly. But the jump host can contact both. So what
> I've done is mount the NFS shares from one host in each data center on
the
> jump box using sshfs.
>
> The directory I'm trying to rsync from has 111GB of data in it. I
don't
> think I've ever setup an rsync for quite so much data before.
>
>  But I started the rsync at approx. 7pm last night. And as of now the rsync
> is still building it's file list.
>
> [root at sshproxygw ~]# rsync -avzp /mnt/db_space/timd/www1/
> /mnt/db_space/timd/www2/svn2/
> building file list ...
>
> So my question to you is, is this a normal amount of time to wait for the
> file list to be built? Considering the amount of data involved. I have it
> running in a screen session I can attach to to find out what's going
on.
>
Regerating rsync file list can take a very long time :/

Make sure that you are using fast link on both ends and at least version 3
of rsync.

--
Eero

On 10/19/2014 8:55 AM, Tim Dunphy wrote:
>   I've setup an rsync between two directories that I've mounted
locally on a
> jump box. Long story short, the two directories are both NFS shares from
> two different hosts. Our security dept won't allow us to SSH between
the
> two data centers, directly. But the jump host can contact both. So what
> I've done is mount the NFS shares from one host in each data center on
the
> jump box using sshfs.
can this 'jump host' ssh to either of the servers ?   it might be worth 
using rsync-over-ssh protocol on one side of this xfer, probably the 
destination.

so   rsync -avh...  /sourcenfs/path/to...  user at desthost:path



-- 
john r pierce                                      37N 122W
somewhere on the middle of the left coast

Are you "allowed" to temporarily run an ssh tunnel (or stunnel) on
your jumpbox?
So connecting from host1 to jumpbox on port XXX would be tunneled to ssh port on
host2...

Or with netcat (if you can mkfifo)?
  mkfifo backpipe
  nc -l 12345 0<backpipe | nc host2 22 1>backpipeBut you will have to
trick ssh into accepting your jumpbox "fingerprint"...

JD