CentOS - Jul 2010 - wget and resulting filenames with mirror option

I am trying to mirror a directory using wget and the resulting files as
expected are taking the name of the string in the url after the last
"/".

Anyone know a way around this? I am using the mirror option so I don't
have to keep track of what to get making it simple.

Thanks!
jlc

Joseph L. Casale wrote, On 07/28/2010 04:26 PM:
> I am trying to mirror a directory using wget and the resulting files as
> expected are taking the name of the string in the url after the last
"/".
> 
> Anyone know a way around this? I am using the mirror option so I don't
> have to keep track of what to get making it simple.
> 
> Thanks!
> jlc
I am having a bit of trouble parsing what you wrote, but I think you are saying
you are issuing
something like:
wget --mirror http://example.com/subdir1/subdir2/fileiwant
and getting at your location
./fileiwant
which is a little confusing, because I would have expected you to get (with just
the --mirror option):
./example.com/subdir1/subdir2/fileiwant

please give an example command and output received and desired.

to get ./fileiwant from my example url above I think you would have to do:
wget --mirror ---cut-dirs=2 -no-host-directories \
	http://example.com/subdir1/subdir2/fileiwant

-- 
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter

On Wed, 28 Jul 2010, Joseph L. Casale wrote:
> Anyone know a way around this? I am using the mirror option so I don't
> have to keep track of what to get making it simple.
my apology in advance for suggesting another tool, but, there 
you are ...

use lftp for mirroring?  wget can do mirroring but it is 
clunky compared to lftp, and much harder to script.  man pages 
are worse, to my taste

Looking, it seems I have a bit over 1400 files driving lftp 
that basically look like this on a local interior mirror

[root at xps400 ~]# wc -l  *.conf *.conf_ | tail -1
  1490 total

[root at xps400 ~]# cat lftp-ORC.conf
#
#       inside at ftp.first, this is NOT automatically run
#
mirror -c -e \
         ftp://ftp.owlriver.com/pub/local/ORC  \
         /var/ftp/pub/mirror/ORC
#

-------

One can invoke it as simply as:

 	lftp -f lftp-ORC.conf

and it will of course run non-root if one desires; it can be 
wrappered up in a scriptlet (I use a flock scheme in the 
wrappering so it does not overrun into a prior day's run; 
mirroring runs continuously) ...

 	http://bugs.centos.org/view.php?id=4443

suggests a simple way to gather detail:

export YMD=`date +%Y%m%d`
cd $TOP
date > DIRECTORY_SIZES.txt
date > MANIFEST-${YMD}.txt
find . -type d -exec du -sh {} \; >> DIRECTORY_SIZES.txt
find . -type f \; >> MANIFEST-${YMD}.txt

and diff to taste, optionally automailing when more than the 
top 'date' line changes

-- Russ herrold