openssh unix dev - Mar 2018 - SCP verbose output truncated

Fedora 27 x86_64. openssh-7.6p1-5.fc27.x86_64.

If I run a simple command like, "scp -v file target:", I get a new
prompt before all the output appears. Running interactively, I see all
that output, though some of it is after the prompt. But if I put this
command in a script, the output is truncated at the point where my
prompt would have appeared interactively. I see a message like, "Shared
connection to target closed."

I've tried putting the command in the background (&) and waiting for it,
using the LocalCommand option to add a wait, and redirecting output to a
file. So far I can't find a reliable way to get all the output when
running in a script. Any ideas?
-- 
	Dave Close

On Thu, Mar 22, 2018 at 2:47 PM, CLOSE Dave
<Dave.Close at us.thalesgroup.com> wrote:
> Fedora 27 x86_64. openssh-7.6p1-5.fc27.x86_64.
>
> If I run a simple command like, "scp -v file target:", I get a
new
> prompt before all the output appears. Running interactively, I see all
> that output, though some of it is after the prompt. But if I put this
> command in a script, the output is truncated at the point where my
> prompt would have appeared interactively. I see a message like,
"Shared
> connection to target closed."
I think part of the issue is that some of the output is to stderr.
stderr and stdout can overwrite each other in fascinating ways.

I'd urge you to use "scp -v foo bar 2>&1" to keep all of
stderr and
stdout, and avoid overwriting one with the other.
> I've tried putting the command in the background (&) and waiting
for it,
> using the LocalCommand option to add a wait, and redirecting output to a
> file. So far I can't find a reliable way to get all the output when
> running in a script. Any ideas?
Have you considered using "rsync -e ssh" instead of scp, to get more
meaningful dialogue for most transfers?

Nico Kadel-Garcia wrote:
>> Fedora 27 x86_64. openssh-7.6p1-5.fc27.x86_64.
>> 
>> If I run a simple command like, "scp -v file target:", I get
a new
>> prompt before all the output appears. Running interactively, I see
>> all that output, though some of it is after the prompt. But if I
>> put this command in a script, the output is truncated at the point
>> where my prompt would have appeared interactively. I see a message
>> like, "Shared connection to target closed."
> 
> I think part of the issue is that some of the output is to stderr. 
> stderr and stdout can overwrite each other in fascinating ways.
> 
> I'd urge you to use "scp -v foo bar 2>&1" to keep all
of stderr and
> stdout, and avoid overwriting one with the other.
Basically, that is what I was doing. However, the problem remains that
the script continues when SCP returns and that is long before the output
is done. I don't know if this is due to buffering but the time involved
seems far too long for that (~5s on an otherwise idle system).

I've found that if I use "scp -v file target: 2>log
>/dev/null", the log
file does eventually contain all the output. The problem then becomes
how the script can determine that the SCP command is finished and the
output file is complete. I'm currently using "lsof -f -- log
2>/dev/null
| grep -q " 2w ", but that seems a bit like a kludge.
> Have you considered using "rsync -e ssh" instead of scp, to get
more
> meaningful dialogue for most transfers?
I know there are other ways I can transfer a file. My interest is in
diagnosing a real-life transfer started by others.
-- 
Dave Close, Thales InFlyt Experience, Irvine California USA.
cell +1 949 394 2124, dave.close at us.thalesgroup.com

 "1800: Joseph Marie Jacquard teaches a loom to read punch cards,
  creating the first heavily multi-threaded processing unit."