rsync - Jan 2017 - downloading only specific directories from directory tree

Hello:

I have read rsync manual and several howtos on how to use rsync, still
I don't know if it's doable what I want to do, and if yes, how.

The scenario:

I would like to make a local copy of openSUSE 13.2 repositories.
I use openSUSE linux.

The repos are located in a multi-level directory structure, eg:

ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/

Let's call this directory the root dir.

Each directory in the root has subdirectories, and one of the 
subdirectories
is named "openSUSE_13.2". The openSUSE_13.2 dir can be one, two, or 
more level
downstairs in the root dir, eg:

http://download.opensuse.org/repositories/Apache/openSUSE_13.2/
http://download.opensuse.org/repositories/Apache:/MirrorBrain/openSUSE_13.2/

There are plenty of directories in the root dir, and each subdirectory 
can
have several subdirectories.

Is it possible, using rsync, to download only all "openSUSE_13.2" 
directories with
their contents, without specifying directly one by one every 
subdirectory
to be included or excluded?

I tried to use filters.txt file with content:

+ **openSUSE_13.2/***

with this command:

rsync -av --include-from=filters.txt 
rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ ./

This downloads every directory. How can I exclude those are not 
openSUSE_13.2?

Thanks,

Istvan

An include only overrides an exclude that follows it.  So, you would
need something like this:

+ */
+ openSUSE_13.2/***
- *

Then add --prune-empty-dirs to your rsync command line.  I left off the
initial ** because it doesn't really mean anything unless openSUSE_13.2
isn't always the entire directory name in which case 1 * would do.

Another way to explain it is that by default everything is included.  An
exclude rule makes exceptions.  An include rule makes exceptions to the
exclude rule.

On 01/20/2017 06:08 PM, Istvan Gabor wrote:
> Hello:
> 
> I have read rsync manual and several howtos on how to use rsync, still
> I don't know if it's doable what I want to do, and if yes, how.
> 
> The scenario:
> 
> I would like to make a local copy of openSUSE 13.2 repositories.
> I use openSUSE linux.
> 
> The repos are located in a multi-level directory structure, eg:
> 
> ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/
> 
> Let's call this directory the root dir.
> 
> Each directory in the root has subdirectories, and one of the
> subdirectories
> is named "openSUSE_13.2". The openSUSE_13.2 dir can be one, two,
or more
> level
> downstairs in the root dir, eg:
> 
> http://download.opensuse.org/repositories/Apache/openSUSE_13.2/
>
http://download.opensuse.org/repositories/Apache:/MirrorBrain/openSUSE_13.2/
> 
> 
> There are plenty of directories in the root dir, and each subdirectory can
> have several subdirectories.
> 
> Is it possible, using rsync, to download only all "openSUSE_13.2"
> directories with
> their contents, without specifying directly one by one every subdirectory
> to be included or excluded?
> 
> I tried to use filters.txt file with content:
> 
> + **openSUSE_13.2/***
> 
> with this command:
> 
> rsync -av --include-from=filters.txt
> rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ ./
> 
> This downloads every directory. How can I exclude those are not
> openSUSE_13.2?
> 
> Thanks,
> 
> Istvan
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170120/800b435d/signature.sig>

On Fri, 20 Jan 2017 18:15:00 -0500, Kevin Korb wrote:
> An include only overrides an exclude that follows it.  So, you would
> need something like this:
>
> + */
> + openSUSE_13.2/***
> - *
>
> Then add --prune-empty-dirs to your rsync command line.  I left off 
> the
> initial ** because it doesn't really mean anything unless 
> openSUSE_13.2
> isn't always the entire directory name in which case 1 * would do.
>
> Another way to explain it is that by default everything is included.  
> An
> exclude rule makes exceptions.  An include rule makes exceptions to 
> the
> exclude rule.
Thank you a lot.
This worked the way I wanted.

Istvan

The following URL may also be helpful in terms of another example : 
http://www.lbackup.org/developer/advanced_excludes_file_examples

Hope that link is helpful.

--------------------------------------------------------------------
HTRAX 2013 Revitalised : EGYPTIAN HUMP HTRAX : Direct URL download :
http://henri.shustak.org/download/htrax/egyptian-hump-htrax.mp3
"Dr Who Meets B52's" - More reviews :
http://www.jessetaylor.com.au/htrax



On 21/01/2017, at 12:15 PM, Kevin Korb <kmk at sanitarium.net> wrote:
> An include only overrides an exclude that follows it.  So, you would
> need something like this:
> 
> + */
> + openSUSE_13.2/***
> - *
> 
> Then add --prune-empty-dirs to your rsync command line.  I left off the
> initial ** because it doesn't really mean anything unless openSUSE_13.2
> isn't always the entire directory name in which case 1 * would do.
> 
> Another way to explain it is that by default everything is included.  An
> exclude rule makes exceptions.  An include rule makes exceptions to the
> exclude rule.
> 
> On 01/20/2017 06:08 PM, Istvan Gabor wrote:
>> Hello:
>> 
>> I have read rsync manual and several howtos on how to use rsync, still
>> I don't know if it's doable what I want to do, and if yes, how.
>> 
>> The scenario:
>> 
>> I would like to make a local copy of openSUSE 13.2 repositories.
>> I use openSUSE linux.
>> 
>> The repos are located in a multi-level directory structure, eg:
>> 
>> ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/
>> 
>> Let's call this directory the root dir.
>> 
>> Each directory in the root has subdirectories, and one of the
>> subdirectories
>> is named "openSUSE_13.2". The openSUSE_13.2 dir can be one,
two, or more
>> level
>> downstairs in the root dir, eg:
>> 
>> http://download.opensuse.org/repositories/Apache/openSUSE_13.2/
>>
http://download.opensuse.org/repositories/Apache:/MirrorBrain/openSUSE_13.2/
>> 
>> 
>> There are plenty of directories in the root dir, and each subdirectory
can
>> have several subdirectories.
>> 
>> Is it possible, using rsync, to download only all
"openSUSE_13.2"
>> directories with
>> their contents, without specifying directly one by one every
subdirectory
>> to be included or excluded?
>> 
>> I tried to use filters.txt file with content:
>> 
>> + **openSUSE_13.2/***
>> 
>> with this command:
>> 
>> rsync -av --include-from=filters.txt
>> rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ ./
>> 
>> This downloads every directory. How can I exclude those are not
>> openSUSE_13.2?
>> 
>> Thanks,
>> 
>> Istvan
>> 
> 
> -- 
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			http://www.sanitarium.net/
> 	PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Dear Kevin:

~ 1 year ago your answer helped me to solve my problem.
This time I would like to do a similar thing but little bit modified.
I read again carefully INCLUDE/EXCLUDE PATTERN RULES section of rsync
manual but still cannot comprehend every part of it.

I understand I have to add include patterns first and exclude patterns
second. But it is not clear if I have to add all the include patterns
first and all the exclude patterns after them. Or do I have add
include/exclude groups?

Now I would like to mirror openSUSE_Leap_42.2 dirs from
ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/
but only from specified directories and without files with debuginfo
or debugsource patterns in their names.

I tried this command:

rsync -av --include-from=filters.txt --prune-empty-dirs \
rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ \
opensuse/repositories

where filters.txt contains:

+ /filesystems
+ /filesystems:
+ /games
+ /games:
+ /graphics
+ /graphics:
+ /hamradio
+ /hamradio:
+ /hardware
+ /hardware:
...
+ openSUSE_Leap_42.2/***
- *debuginfo*.rpm
- *debugsource*.rpm
- *

but it doesn't work the way I'd like.

1. The directories ending with : in their names are not mirrored.
I tried eg /games:  , /games\:  and "/games:", none of these worked.

2. The other directories are mirrored but debuginfo and debugsource
files are not excluded.

I'd appreciate if you explained what the problem is and how to solve 
it.

Thanks in advance,

Istvan





On Fri, 20 Jan 2017 18:15:00 -0500, Kevin Korb wrote:
> An include only overrides an exclude that follows it.  So, you would
> need something like this:
>
> + */
> + openSUSE_13.2/***
> - *
>
> Then add --prune-empty-dirs to your rsync command line.  I left off 
> the
> initial ** because it doesn't really mean anything unless 
> openSUSE_13.2
> isn't always the entire directory name in which case 1 * would do.
>
> Another way to explain it is that by default everything is included.  
> An
> exclude rule makes exceptions.  An include rule makes exceptions to 
> the
> exclude rule.
>
> On 01/20/2017 06:08 PM, Istvan Gabor wrote:
>> Hello:
>>
>> I have read rsync manual and several howtos on how to use rsync, 
>> still
>> I don't know if it's doable what I want to do, and if yes, how.
>>
>> The scenario:
>>
>> I would like to make a local copy of openSUSE 13.2 repositories.
>> I use openSUSE linux.
>>
>> The repos are located in a multi-level directory structure, eg:
>>
>> ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/
>>
>> Let's call this directory the root dir.
>>
>> Each directory in the root has subdirectories, and one of the
>> subdirectories
>> is named "openSUSE_13.2". The openSUSE_13.2 dir can be one,
two, or
>> more
>> level
>> downstairs in the root dir, eg:
>>
>> http://download.opensuse.org/repositories/Apache/openSUSE_13.2/
>> 
>>
http://download.opensuse.org/repositories/Apache:/MirrorBrain/openSUSE_13.2/
>>
>>
>> There are plenty of directories in the root dir, and each 
>> subdirectory can
>> have several subdirectories.
>>
>> Is it possible, using rsync, to download only all
"openSUSE_13.2"
>> directories with
>> their contents, without specifying directly one by one every 
>> subdirectory
>> to be included or excluded?
>>
>> I tried to use filters.txt file with content:
>>
>> + **openSUSE_13.2/***
>>
>> with this command:
>>
>> rsync -av --include-from=filters.txt
>> rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ ./
>>
>> This downloads every directory. How can I exclude those are not
>> openSUSE_13.2?
>>
>> Thanks,
>>
>> Istvan
>>