we could modify the protocol and implement
rolling checksums like Niels Provos suggests:
MD5_CTX ctx1, ctx2;
MD5_Init(&ctx1);
while new page in data
MD5_Update(&ctx1, newpage, pagesize)
ctx2 = ctx1;
MD5_Final(digest, &ctx2)
if (compare with remote not equal)
break;
end while
continue data transfer.
On Wed, Sep 17, 2003 at 11:12:36AM +0800, Dmitry Lohansky
wrote:> Hello openssh@
>
> I thought about sftp's reget/reput commands.
>
> Several days ago, Damien Miller write to tech at openbsd.org (it was
> reply for my letter):
>
> > Herein lies a problem which is not easy to detect or solve. For
> > performance reasons, the sftp client does pipelined reads/writes when
> > transferring files. The protocol spec allows for a server to process
> > these requests out of order. For example:
>
> > client server
> > ------ ------
> > open file your file handle is "blah"
> > gimme bytes 0-8191
> > gimme bytes 8192-16383
> > gimme bytes 16384-24575
> > gimme bytes 24576-32767 here are bytes 24576-32767
> > close file here are bytes 16384-24575
> > here are bytes 8192-16383
> > here are bytes 0-8191
> > close successful
>
> > If the client writes the bytes our in the order they are received
(which
> > it probably should, to avoid buffering large amounts of data) then an
> > interruption will leave a full-length, but "holey" file on
disk. There
> > is no general way to determine how to do resume such a transfer.
>
> > The best the client can do to make transfers resumable is ftruncate()
> > the file at the highest contiguous byte received. This will stop the
> > potential corruption on resume.
>
> This is good method, but if client crash, we also may get a
"hole".
> What your think about next way?
>
> Storing extra-data at the end of file, for example:
>
> <---orig-part-><-extra->
> [*][ ][*][ ][*][*******]
> <---------file--------->
>
> where [*] - already loaded data, [ ] - not yet
>
> In extra part, we may store which block was already loaded and it
> offset and size. After download, extra part will be removed.
>
> Comments?
> --
> Dmitry Lohansky
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> http://www.mindrot.org/mailman/listinfo/openssh-unix-dev