I have hinted in the past of wanting to go to a more selective control of the output of rsync. Here it is. PROBLEMS with the existing --verbose et al. The simple incrementing verbose doesn't allow one to restrict stdout to what actually is of interest. For instance inside dirvish i don't care about directories, or any kind of non-regular file because they show up every time even if there is no change. Someone else might not care to see reports of files that only had a permissions change. When debugging, incrementing verbose levels turn on debug messages for whole subsections that are of no concern. To work around limited control there is a continual push to add new specialised verbosity options. --stats and --progress are ones that got implemented. --apt is one that didn't but was a not unreasonable feature. And recently we evaded a request to bastardise --stats by linking it into verbose. PROPOSED APPROACH Replace verbose with a bitmask and provide a clean, extensible interface to set and test them. My first concern was that the command-line interface be clean, intuitive and extensible. Evaluating the existing options and streams i suggest the stdout controlled by --verbose be called the report. Log refers to the logging stream of rsyncd so it is already taken. I have modelled the option somewhat on stty. One controls what is output with the --report option. The option takes as a single argument (no whitespace) a comma delimited lists of keywords. Some keywords can control multiple flags so keywords may be proceeded by a minus sign. For example, suppose you wanted to see a report of files updated but didn't care about deletion. "--report=file,-delete" would do the trick. Progress and statistic reports would also be specifiable through this option so "--report=file,dir,stats,progress" would be the near equivalent of "-v --stats --progress". For backward compatability and because -v is intuitive the existing options would be preserved and mapped into the bitmasks. INTERNAL USAGE The use of verbose, do_stats and do_progress are replaced with two bitmasks: report_flags and debug_flags. Two bitmasks were chosen because debugging and verbosity should be considered separate issues. To facilitate using these two macros are created. Everywhere you have something like "if (verbose)" or "if (verbose >= n)" It would be replaced by something like "if (report(RPT_CREATE))" or "if (debug(1, DBG_CSUM))". That is correct, each subsection of code would have it's own debugging flag(s) and support up to 4(cumulative) levels of debugging. EXAMPLE To facilitate discussion, as well as to do unit testing i have created (attached) a simple example program. It has a small number of examples of using the report and debug macros which control the verbosity of it's output. The test program has very crude command-line parsing. It recognizee -v, --verbose, --stats, --progress, --report and --report=. It does not recognise -vv. --report=help will list the normal keywords and --report=dbg_help will list the debugging keywords. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
Oops, forgot the attachment. Here it is.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: jw@pegasys.ws
Remember Cernan and Schmitt
-------------- next part --------------
#include <stdio.h>
#include <string.h>
#include <limits.h>
#define RPT_CHATTY 1
#define RPT_FILE_CREATE (1 << 1)
#define RPT_FILE_UPDATE (1 << 2)
#define RPT_FILE_DELETE (1 << 3)
#define RPT_FILE_PERM (1 << 4)
#define RPT_FILE_OTHER (1 << 5)
#define RPT_FILE (RPT_FILE_CREATE | RPT_FILE_CREATE | RPT_FILE_UPDATE |
RPT_FILE_DELETE | RPT_FILE_PERM)
#define RPT_LINK (1 << 6)
#define RPT_UNSAFE (1 << 7)
#define RPT_DIR_CREATE (1 << 8)
#define RPT_DIR_DELETE (1 << 9)
#define RPT_DIR_PERM (1 << 10)
#define RPT_DIR (RPT_DIR_CREATE | RPT_DIR_PERM)
#define RPT_STAT_SUM (1 << 11)
#define RPT_STAT_DETAIL (1 << 12)
#define RPT_STAT_MEM (1 << 13)
#define RPT_STATS (RPT_STAT_SUM | RPT_STAT_DETAIL)
#define RPT_PROGRESS (1 << 14)
#define RPT_SKIP (1 << 15)
#define RPT_EXCLUDE (1 << 16)
#define RPT_VERBOSE1 (RPT_FILE | RPT_CHATTY | RPT_DIR | RPT_LINK \
| RPT_UNSAFE)
#define RPT_VERBOSE2 (RPT_UNSAFE | RPT_SKIP | RPT_EXCLUDE | DBG_FLIST \
| RPT_FILE_DELETE | RPT_FILE_OTHER | RPT_DIR_DELETE \
| DBG_FILE)
#define RPT_VERBOSE3 ~0x0
#define DBG_LVL1 1
#define DBG_LVL2 2
#define DBG_LVL3 3
#define DBG_LVL 0x00000003
#define DBG_TRACE (1 << 3)
#define DBG_MEM (1 << 4)
#define DBG_CKSUM (1 << 5)
#define DBG_FLIST (1 << 6)
#define DBG_FILE (1 << 7)
#define DBG_BLOCK (1 << 8)
#define DBG_REPORT (1 << 9)
#define DBG_VERBOSE3 ~DBG_LVL
#define debug(l, m) ((debug_flags & m) && ((debug_flags &
DBG_LVL) >= l))
#define report(m) (report_flags & m)
struct key_options {
unsigned int flag;
char * key;
char * description;
};
struct key_options report_opts[] = {
{ RPT_CHATTY, "chatty", "tty chattiness" },
{ RPT_FILE, "file", "Common file operations" },
{ RPT_FILE_CREATE, "create", " Create a file" },
{ RPT_FILE_UPDATE, "update", " Update a file" },
{ RPT_FILE_DELETE, "delete", " Delete a file" },
{ RPT_FILE_PERM, "perms", " Change file permissions" },
{ RPT_FILE_OTHER, "special", "special files" },
{ RPT_LINK, "link", "hard links" },
{ RPT_UNSAFE, "unsafe", "unsafe links" },
{ RPT_DIR, "dir", "Directories" },
{ RPT_DIR_CREATE, "d_create", " Create directory" },
{ RPT_DIR_PERM, "d_perms", " Change directory
permissions" },
{ RPT_DIR_DELETE, "d_delete", "Delete directory" },
{ RPT_STATS, "stats", "Statistics" },
{ RPT_STAT_SUM, "s_sum", " Statistics summary" },
{ RPT_STAT_DETAIL, "s_detail", " Statistics detail" },
{ RPT_STAT_MEM, "s_mem", "Memory statistics" },
{ RPT_PROGRESS, "progress", "File transfer Progress" },
{ RPT_SKIP, "skip", "Skipped paths" },
{ RPT_EXCLUDE, "exclude", "Exlusion matches" },
{ 0, NULL, NULL }
};
struct key_options debug_opts[] = {
{ DBG_LVL1, "dbg_1", "debug level 1" },
{ DBG_LVL2, "dbg_2", "debug level 2" },
{ DBG_LVL3, "dbg_3", "debug level 3" },
{ DBG_TRACE, "dbg_trace", "Trace rsync protocol?" },
{ DBG_MEM, "dbg_mem", "Memory allocation" },
{ DBG_CKSUM, "dbg_cksum", "Checksum" },
{ DBG_FLIST, "dbg_flist", "File list" },
{ DBG_FILE, "dbg_file", "Per-file debugging" },
{ DBG_BLOCK, "dbg_block", "per-block debugging" },
{ DBG_REPORT, "dbg_report", "display report settigs and
exit" },
{ 0, NULL, NULL }
};
unsigned int report_flags = 0;
unsigned int debug_flags = 0;
int verbose, do_stats, do_progress = 0;
char *do_report = NULL;
/*
* report_args(string)
* split comma delimed string into keywords and set flag
* sets according to keywords matching option sets.
*
* returns count of keywords matched.
* returns -1 on if program should exit;
*/
int
report_args (char *args)
{
char *t;
int o;
int negate;
int count = 0;
int *flag_set;
char arg[NAME_MAX + 1];
struct key_options *option_set;
if (verbose > 4) verbose = 4;
switch (verbose)
{
case 4:
case 3:
debug_flags |= DBG_VERBOSE3 | (verbose - 2);
report_flags |= RPT_VERBOSE3;
case 2:
report_flags |= RPT_VERBOSE2;
if (do_stats) report_flags |= RPT_STAT_MEM;
case 1:
report_flags |= RPT_VERBOSE1;
if (do_progress) report_flags |= RPT_PROGRESS;
}
if (do_stats) report_flags |= RPT_STATS;
if (!args || !args[0]) return 0;
strncpy(arg, args, NAME_MAX);
arg[NAME_MAX] = '\0';
for (t = strtok(arg, ","); t; t = strtok(NULL, ","))
{
negate = 0;
if (t[0] == '+' || t[0] == '-')
{
if (t[0] == '-')
++negate;
++t;
}
if (t[0] == '\0')
continue;
if (!strncmp(t, "dbg_", 4))
{
option_set = debug_opts;
flag_set = &debug_flags;
} else {
option_set = report_opts;
flag_set = &report_flags;
}
if (!strcmp(t, "help") || !strcmp(t, "dbg_help"))
{
if (*t == 'd')
printf("Report debug options (may be prefixed with - to
disable)\n");
else
printf("Report options (may be prefixed with - to disable)\n");
for(o = 0; option_set[o].flag; o++)
{
printf("%20s\t%s\n",
option_set[o].key,
option_set[o].description);
}
printf("%20s\t%s\n",
"help", "Display report options");
printf("%20s\t%s\n",
"dbg_help", "Display debugging options");
return -1; /* exit_cleanup? */
}
for(o = 0; option_set[o].flag; o++)
{
if(!strcmp(t, option_set[o].key))
{
++count;
if (debug(2, DBG_REPORT))
printf("%s\t%8.8x\t%s\n",
option_set[o].key,
option_set[o].flag,
option_set[o].description);
if (negate)
{
*flag_set &= ~option_set[o].flag;
} else {
*flag_set |= option_set[o].flag;
}
break;
}
}
}
return count;
}
main(int argc, char *argv[])
{
int i, o;
int *flag_set;
struct key_options *option_set;
for (i = 1; i < argc; ++i)
{
if (!strcmp(argv[i], "-v"))
++verbose;
else if (!strcmp(argv[i], "--verbose"))
++verbose;
else if (!strcmp(argv[i], "--stats"))
++do_stats;
else if (!strcmp(argv[i], "--progress"))
++do_progress;
else if (!strncmp(argv[i], "--report=", 9))
do_report = argv[i] + 9;
else if (!strcmp(argv[i], "--report"))
do_report = argv[++i];
}
if (report_args(do_report) < 0)
return 0;
if (debug(0, DBG_REPORT))
{
option_set = NULL;
while (1)
{
if (option_set == debug_opts) break;
if (option_set)
{
option_set = debug_opts;
flag_set = &debug_flags;
} else {
option_set = report_opts;
flag_set = &report_flags;
}
for(o = 0; option_set[o].flag; o++)
{
if (option_set[o].flag == (option_set[o].flag & *flag_set))
{
if (debug(1, DBG_REPORT))
{
printf("%s\t%8.8x\t%s\n",
option_set[o].key,
option_set[o].flag,
option_set[o].description);
} else {
printf("%s,", option_set[o].key);
}
}
}
}
printf("\n");
}
if (report(RPT_STAT_SUM))
{
printf("\t%8.8x\treport flags\n", report_flags);
printf("\t%8.8x\tdebug flags\n", debug_flags);
}
}
> Date: Thu, 15 May 2003 20:11:09 -0700 > From: jw schultz <jw@pegasys.ws> > Subject: [RFC] report options > > I have hinted in the past of wanting to go to a more > selective control of the output of rsync. Here it is. >Sounds very cool. I'm guessing that a side-effect of all this work the dry-run option will get the ability to optionally show meta changes that would happen such as owner/group/perms/timestamp changes? This was discussed some time back, and is probably still on someones to-do list. It sure would be nice to see an accurate list of what will be updated. Right now we can assume the additional files listed in a real run that were not listed in dry-run are meta-data, but we don't know which are changing before the fact so you can save off the state info before the run or adjust parameters to exclude them. -- Lee Eakin - leakin@ti.com - Internet/Naming Services, Texas Instruments The Tick Files: The Truth Is Out There. So Don't Open The Front Door.
I would just like to say that i would love to see these features implemented. They would be very usefull to me. I would be happy to help beta test an implementation if it would help. -Michael
Only two non-developers have responded to this. Barring a favourable response from other developers i'll shelve this proposal. On Thu, May 15, 2003 at 08:11:09PM -0700, jw schultz wrote:> I have hinted in the past of wanting to go to a more > selective control of the output of rsync. Here it is. > > PROBLEMS with the existing --verbose et al. > > The simple incrementing verbose doesn't allow one to > restrict stdout to what actually is of interest. For > instance inside dirvish i don't care about directories, > or any kind of non-regular file because they show up every > time even if there is no change. Someone else might not > care to see reports of files that only had a permissions > change. > > When debugging, incrementing verbose levels turn on debug > messages for whole subsections that are of no concern. > > To work around limited control there is a continual push to > add new specialised verbosity options. --stats and > --progress are ones that got implemented. --apt is one that > didn't but was a not unreasonable feature. And recently we > evaded a request to bastardise --stats by linking it > into verbose. > > PROPOSED APPROACH > > Replace verbose with a bitmask and provide a clean, > extensible interface to set and test them. > > My first concern was that the command-line interface be > clean, intuitive and extensible. Evaluating the existing > options and streams i suggest the stdout controlled by > --verbose be called the report. Log refers to the logging > stream of rsyncd so it is already taken. I have modelled the > option somewhat on stty. > > One controls what is output with the --report option. The > option takes as a single argument (no whitespace) a comma > delimited lists of keywords. Some keywords can control > multiple flags so keywords may be proceeded by a minus sign. > For example, suppose you wanted to see a report of files > updated but didn't care about deletion. > "--report=file,-delete" would do the trick. Progress and > statistic reports would also be specifiable through this > option so "--report=file,dir,stats,progress" would be the > near equivalent of "-v --stats --progress". > > For backward compatability and because -v is intuitive the > existing options would be preserved and mapped into the > bitmasks. > > INTERNAL USAGE > > The use of verbose, do_stats and do_progress are replaced > with two bitmasks: report_flags and debug_flags. Two > bitmasks were chosen because debugging and verbosity should > be considered separate issues. To facilitate using these > two macros are created. Everywhere you have something like > "if (verbose)" or "if (verbose >= n)" It would be replaced > by something like "if (report(RPT_CREATE))" or "if (debug(1, > DBG_CSUM))". That is correct, each subsection of code would > have it's own debugging flag(s) and support up to > 4(cumulative) levels of debugging. > > EXAMPLE > > To facilitate discussion, as well as to do unit testing i > have created (attached) a simple example program. > It has a small number of examples of using the report and > debug macros which control the verbosity of it's output. > > The test program has very crude command-line parsing. It > recognizee -v, --verbose, --stats, --progress, --report and > --report=. It does not recognise -vv. > --report=help will list the normal keywords and > --report=dbg_help will list the debugging keywords. > > -- > ________________________________________________________________ > J.W. Schultz Pegasystems Technologies > email address: jw@pegasys.ws > > Remember Cernan and Schmitt > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
Apparently Analagous Threads
- Creating multiple option groups
- [Bug 1765] New: Error message if key not first in authorized_keys file
- 19 commits - libswfdec/swfdec_audio_event.c libswfdec/swfdec_bits.c libswfdec/swfdec_color.c libswfdec/swfdec_color.h libswfdec/swfdec_flv_decoder.c libswfdec/swfdec_image.c libswfdec/swfdec_initialize.as libswfdec/swfdec_initialize.h