This seems simple enough...I'm just not script wizard. If someone can
throw together a starting point I can test and tweak it from there. It
seems to me:
1. Read /var/mail/mail.err or specified logfile
2. For each "failed:
read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line,
a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists
i. If attachmentHash is missing display such for possible
backup searching.
b. create link attachmentHash-userHash to hashes/attachmentHash
3. Continue to end of file
Can this be done via "pure" BASH? Need sed/awk as well?
--
Daniel
Daniel, please help me understand, since I met your same problem on 2015-02-26 as you did.> 2. For each "failed: > read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, > a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists > i. If attachmentHash is missing display such for possibleIs this a FEATURE of Dovecot SIS? Or a known bug described somewhere? Loosing detached attachments, I mean.> Can this be done via "pure" BASH? Need sed/awk as well?Well, you need a way to strip out the "-userHash" part from the string, so a little bit of sed and regex is needed. You also need to work out source and destination files. Still, personally I think the problem is somewhere else and should be fixed. Else, stop using SIS and let the SAN do the deduplication. Am I wrong? Paolo
I use this script for review. For a correct result Dovecot should not run.
#!/bin/bash
attdir="/var/files/attachments"
maildir="/var/mail"
cd "$attdir" || exit 1
find -type f -printf "%P\n" | grep -v "hashes" | sort -u
>"$maildir/attachments.s"
cd "$maildir" || exit 2
rm "$maildir/files.ok" "$maildir/files.fail" 2>/dev/null
for f in $(find -type f -name "m.*" -printf "%P\n"); do
doveadm dump -t dbox "$f" | egrep "^msg.ext-ref" | while
read z; do
set -- $z
while [ -n "$1" ]; do
if [[ $1 == */* ]]; then
test -r "$attdir/$1" && echo "$1"
>>"$maildir/files.ok" || echo
"$1" >>"$maildir/files.fail"
fi
shift
done
done
done
sort -u "$maildir/files.ok" >"$maildir/files.s"
diff -Nu "$maildir/attachments.s" "$maildir/files.s" | tee
"$maildir/files.diff" | egrep "^\-"
Am 03.03.2015 um 22:08 schrieb Daniel Miller:> This seems simple enough...I'm just not script wizard. If someone can
> throw together a starting point I can test and tweak it from there.
> It seems to me:
>
> 1. Read /var/mail/mail.err or specified logfile
> 2. For each "failed:
> read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line,
> a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists
> i. If attachmentHash is missing display such for possible
> backup searching.
> b. create link attachmentHash-userHash to hashes/attachmentHash
> 3. Continue to end of file
>
> Can this be done via "pure" BASH? Need sed/awk as well?
>
That works brilliantly! I revised my script now to take advantage of
yours -
#!/bin/bash
# These variables need to be customized for your particular installation
LISTFILE='/var/mail/files.fail'
ATTACHMENT_STORAGE_BASE='/var/files/attachments'
# These variables are based on current Dovecot behaviour and should not
require changing
HASH_FOLDER='hashes'
function usage
{
echo "Dovecot Single-Instance-Storage Attachment Repair"
echo "usage: dovesisfix [-d] [-t] [-v] [-h]"
echo " -t | --test-only perform logfile analysis and show steps
to be taken without any on-disk modification"
echo " -v | --verbose provide verbose messages at each
step"
echo " -d | --debug provide additional debug
messages"
echo " -h | --help this screen"
}
while [ "$1" != "" ]; do
case $1 in
-d | --debug ) DEBUG=1
VERBOSE=1
;;
-t | --test-only ) TESTMODE=1
;;
-v | --verbose ) VERBOSE=1
;;
-h | --help ) usage
exit
;;
* ) usage
exit 1
esac
shift
done
while read -r LINE
do
# Now extract the aa/bb/ prefix, the base attachment file name, and
user hash
CATEGORY_PATH="${LINE:0:5}"
BASE_HASH="${LINE:6:40}"
USER_HASH="${LINE:47:32}"
ATTACH_SOURCE="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$HASH_FOLDER/$BASE_HASH"
ATTACH_TARGET="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$BASE_HASH-$USER_HASH"
# If in debug/verbose mode show operation about to occur
if [ "$VERBOSE" = 1 ]; then
echo "The file $ATTACH_SOURCE must be linked to
$ATTACH_TARGET"
fi
# Verify that source exists
if [ ! -f "$ATTACH_SOURCE" ]; then
echo "ERROR: File $ATTACH_SOURCE does not exist. You must
restore this from a backup and run this utility again."
fi
# This is a Good Thing.
if [ "$DEBUG" = 1 ]; then
echo "The file $ATTACH_SOURCE appears to be a valid file."
fi
# Check if user link mysteriously reappeared
if [ -f "$ATTACH_TARGET" ]; then
echo "INFO: File $ATTACH_TARGET exists. This may mean the fault
has been previously corrected. Clearing/rotating the logfile $LOGFILE is
appropriate now."
continue
fi
# Prepare to create user link
LINK_LINE="$ATTACH_SOURCE $ATTACH_TARGET"
if [ "$DEBUG" = 1 ]; then
echo "About to execute command: ln $LINK_LINE"
fi
# If test mode, do nothing
if [ "$TESTMODE" = 1 ]; then
continue
fi
# There's probably more tests I could/should do - but I don't know
how
# So...if we're not in test mode...time to do it to it.
LINK_CREATED=$(ln $LINK_LINE)
if [ "$VERBOSE" = 1 ]; then
echo "Repair result for $ATTACH_TARGET - $LINK_CREATED"
fi
done < "$LISTFILE"
On 3/4/2015 9:29 AM, Hardy Flor wrote:> I use this script for review. For a correct result Dovecot should not
> run.
>
> #!/bin/bash
>
> attdir="/var/files/attachments"
> maildir="/var/mail"
>
> cd "$attdir" || exit 1
> find -type f -printf "%P\n" | grep -v "hashes" | sort
-u
> >"$maildir/attachments.s"
> cd "$maildir" || exit 2
> rm "$maildir/files.ok" "$maildir/files.fail"
2>/dev/null
> for f in $(find -type f -name "m.*" -printf "%P\n"); do
> doveadm dump -t dbox "$f" | egrep "^msg.ext-ref" |
while read z; do
> set -- $z
> while [ -n "$1" ]; do
> if [[ $1 == */* ]]; then
> test -r "$attdir/$1" && echo "$1"
>>"$maildir/files.ok" ||
> echo "$1" >>"$maildir/files.fail"
> fi
> shift
> done
> done
> done
> sort -u "$maildir/files.ok" >"$maildir/files.s"
> diff -Nu "$maildir/attachments.s" "$maildir/files.s" |
tee
> "$maildir/files.diff" | egrep "^\-"
>
>
>
> Am 03.03.2015 um 22:08 schrieb Daniel Miller:
>> This seems simple enough...I'm just not script wizard. If someone
>> can throw together a starting point I can test and tweak it from
>> there. It seems to me:
>>
>> 1. Read /var/mail/mail.err or specified logfile
>> 2. For each "failed:
>> read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line,
>> a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists
>> i. If attachmentHash is missing display such for possible
>> backup searching.
>> b. create link attachmentHash-userHash to hashes/attachmentHash
>> 3. Continue to end of file
>>
>> Can this be done via "pure" BASH? Need sed/awk as well?
>>
--
Daniel L. Miller, VP - Engineering, SET
AM Fire & Electronic Services, Inc. [AMFES]
dmiller at amfes.com 702-312-5276