I'm missing a HiddenStore option in OpenSSH, known from some ftp-server implementations like ProFTPd. Consider the following scenario: - A process PROCA is frequently polling the directory for a file called myfile.txt - Someone transfers this file via sftp or scp to the directory - While transfer is going on and the file is not completely written, PROCA reads in the file and removes is -> Corrupt data is seen by PROCA Knowing this problem you have to solutions: 1. PROCA must check if myfile.txt is changing (filesize, mtime...) and wait until it does not change any more 2. sftp and scp use a HiddenStore by writing the file with a unique filename (eg. .myfile.txt) and renaming it at the end of the transfer (mv .myfile.txt myfile.txt) What do you think about this? Thomas
Thomas Blank wrote:> I'm missing a HiddenStore option in OpenSSH, known from some ftp-server > implementations like ProFTPd. > > Consider the following scenario: > - A process PROCA is frequently polling the directory for a file called > myfile.txt > - Someone transfers this file via sftp or scp to the directory > - While transfer is going on and the file is not completely written, > PROCA reads in the file and removes is > -> Corrupt data is seen by PROCA > > Knowing this problem you have to solutions: > 1. PROCA must check if myfile.txt is changing (filesize, mtime...) and > wait until it does not change any more > 2. sftp and scp use a HiddenStore by writing the file with a unique > filename (eg. .myfile.txt) and renaming it at the end of the transfer > (mv .myfile.txt myfile.txt) > > What do you think about this?Why not have PROCA use inotify? See /usr/src/linux/Documentation/filesystems/inotify.txt hth, Jason.
Thomas Blank wrote:> I'm missing a HiddenStore option in OpenSSH, known from some ftp-server > implementations like ProFTPd. > > Consider the following scenario: > - A process PROCA is frequently polling the directory for a file called > myfile.txt > - Someone transfers this file via sftp or scp to the directory > - While transfer is going on and the file is not completely written, > PROCA reads in the file and removes is > -> Corrupt data is seen by PROCA > > Knowing this problem you have to solutions: > 1. PROCA must check if myfile.txt is changing (filesize, mtime...) and > wait until it does not change any more > 2. sftp and scp use a HiddenStore by writing the file with a unique > filename (eg. .myfile.txt) and renaming it at the end of the transfer > (mv .myfile.txt myfile.txt)Do you have control over the clients? If so: 3. Use sftp to upload the file with a temporary name then rename it when it's complete. Something like this (untested): sftp -b /dev/stdin server.example.com << EOD put myfile.txt .myfile.txt rename .myfile.txt myfile.txt EOD -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
Darren Tucker wrote:> Do you have control over the clients? If so: > > 3. Use sftp to upload the file with a temporary name then rename it when > it's complete. Something like this (untested): > > sftp -b /dev/stdin server.example.com << EOD > put myfile.txt .myfile.txt > rename .myfile.txt myfile.txt > EODNo, I do not have control over the clients. Jefferson's suggestion of using tunneled rsnc is therefore also not possible. Implementing this is much more work as using standard sftp-commands in a script. inotify may help but I'm using Solaris not Linux - although not knowing but inotify may not have been ported to Solaris. Any other suggestions?