libvirt users - Apr 2016 - Re: stream finish throws exception via python API

On 26.04.16 14:14, Shahar Havivi wrote:
> On 25.04.16 09:11, Cole Robinson wrote:
> > On 04/25/2016 08:10 AM, Shahar Havivi wrote:
> > > On 17.04.16 15:41, Shahar Havivi wrote:
> > >> Hi,
> > >> The following snippet works fine e.g. receiving the data but
when calling
> > >> stream.finish() we get the following error:
> > >>
> > >> stream = con.newStream()
> > >> vol.download(stream, 0, 0, 0)
> > >> buf = stream.recv(1024)
> > >> stream.finish()
> > >>
> > >> libvirt: I/O Stream Utils error : internal error: I/O helper
exited abnormally
> > >> Traceback (most recent call last):
> > >>   File "./helpers/kvm2ovirt", line 149, in
<module>
> > >>     download_volume(vol, item[1], diskno, disksitems, pksize)
> > >>   File "./helpers/kvm2ovirt", line 102, in
download_volume
> > >>     stream.finish()
> > >>   File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 5501, in finish
> > >>     if ret == -1: raise libvirtError ('virStreamFinish()
failed')
> > >> libvirt.libvirtError: internal error: I/O helper exited
abnormally
> > >>  
> > 
> > The error message sucks, I'll send patches to improve it a little
at least.
> > 
> > What's happening here is that the you haven't read all the
data you requested
> > (it's vol.download(path, offset, length, flags), length == 0 means
read the
> > whole file which I suspect you haven't done). In this case the
iohelper
> > program that libvirt uses won't complete feeding us all the data,
and it exits
> > with SIGPIPE when we close the read end of the pipe.
> > 
> > Now whether that should actually be an error condition is open to
debate.
> > virStreamFinish docs make it sound like it's legitimate to throw
an error if
> > it appears that all requested data wasn't read.
> Thanks checking...
Thanks Cole,
I modify our script and for checking if stream.recv() returns zero the finish
works fine.
Our API needs to know the size of bytes that you are going to stream back
which is not provided by the allocation and capacity.
(ie the same as du -s /path/to/disk)

  Shahar.
> > 
> > /**
> >  * virStreamFinish:
> >  * @stream: pointer to the stream object
> >  *
> >  * Indicate that there is no further data to be transmitted
> >  * on the stream. For output streams this should be called once
> >  * all data has been written. For input streams this should be
> >  * called once virStreamRecv returns end-of-file.
> >  *
> >  * This method is a synchronization point for all asynchronous
> >  * errors, so if this returns a success code the application can
> >  * be sure that all data has been successfully processed.
> >  *
> >  * Returns 0 on success, -1 upon error
> >  */
> > 
> > So maybe in your case you want virStreamAbort instead (which also
> > unconditionally throws an error which is also an issue IMO but can be
ignored)
> > 
> > CCing danpb for this thoughts on the erroring semantics
> > 
> > - Cole

On 26.04.16 15:30, Shahar Havivi wrote:
> On 26.04.16 14:14, Shahar Havivi wrote:
> > On 25.04.16 09:11, Cole Robinson wrote:
> > > On 04/25/2016 08:10 AM, Shahar Havivi wrote:
> > > > On 17.04.16 15:41, Shahar Havivi wrote:
> > > >> Hi,
> > > >> The following snippet works fine e.g. receiving the data
but when calling
> > > >> stream.finish() we get the following error:
> > > >>
> > > >> stream = con.newStream()
> > > >> vol.download(stream, 0, 0, 0)
> > > >> buf = stream.recv(1024)
> > > >> stream.finish()
> > > >>
> > > >> libvirt: I/O Stream Utils error : internal error: I/O
helper exited abnormally
> > > >> Traceback (most recent call last):
> > > >>   File "./helpers/kvm2ovirt", line 149, in
<module>
> > > >>     download_volume(vol, item[1], diskno, disksitems,
pksize)
> > > >>   File "./helpers/kvm2ovirt", line 102, in
download_volume
> > > >>     stream.finish()
> > > >>   File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 5501, in finish
> > > >>     if ret == -1: raise libvirtError
('virStreamFinish() failed')
> > > >> libvirt.libvirtError: internal error: I/O helper exited
abnormally
> > > >>  
> > > 
> > > The error message sucks, I'll send patches to improve it a
little at least.
> > > 
> > > What's happening here is that the you haven't read all
the data you requested
> > > (it's vol.download(path, offset, length, flags), length == 0
means read the
> > > whole file which I suspect you haven't done). In this case
the iohelper
> > > program that libvirt uses won't complete feeding us all the
data, and it exits
> > > with SIGPIPE when we close the read end of the pipe.
> > > 
> > > Now whether that should actually be an error condition is open to
debate.
> > > virStreamFinish docs make it sound like it's legitimate to
throw an error if
> > > it appears that all requested data wasn't read.
> > Thanks checking...
> Thanks Cole,
> I modify our script and for checking if stream.recv() returns zero the
finish
> works fine.
> Our API needs to know the size of bytes that you are going to stream back
> which is not provided by the allocation and capacity.
> (ie the same as du -s /path/to/disk)
> 
>   Shahar.
We need the size of the image "Image end offset" you get from
"qemu-img check",
Does libvirt have an API for that?

Thanks,
  Shahar.
> > > 
> > > /**
> > >  * virStreamFinish:
> > >  * @stream: pointer to the stream object
> > >  *
> > >  * Indicate that there is no further data to be transmitted
> > >  * on the stream. For output streams this should be called once
> > >  * all data has been written. For input streams this should be
> > >  * called once virStreamRecv returns end-of-file.
> > >  *
> > >  * This method is a synchronization point for all asynchronous
> > >  * errors, so if this returns a success code the application can
> > >  * be sure that all data has been successfully processed.
> > >  *
> > >  * Returns 0 on success, -1 upon error
> > >  */
> > > 
> > > So maybe in your case you want virStreamAbort instead (which also
> > > unconditionally throws an error which is also an issue IMO but
can be ignored)
> > > 
> > > CCing danpb for this thoughts on the erroring semantics
> > > 
> > > - Cole

On 04/26/2016 09:35 AM, Shahar Havivi wrote:
> On 26.04.16 15:30, Shahar Havivi wrote:
>> On 26.04.16 14:14, Shahar Havivi wrote:
>>> On 25.04.16 09:11, Cole Robinson wrote:
>>>> On 04/25/2016 08:10 AM, Shahar Havivi wrote:
>>>>> On 17.04.16 15:41, Shahar Havivi wrote:
>>>>>> Hi,
>>>>>> The following snippet works fine e.g. receiving the
data but when calling
>>>>>> stream.finish() we get the following error:
>>>>>>
>>>>>> stream = con.newStream()
>>>>>> vol.download(stream, 0, 0, 0)
>>>>>> buf = stream.recv(1024)
>>>>>> stream.finish()
>>>>>>
>>>>>> libvirt: I/O Stream Utils error : internal error: I/O
helper exited abnormally
>>>>>> Traceback (most recent call last):
>>>>>>   File "./helpers/kvm2ovirt", line 149, in
<module>
>>>>>>     download_volume(vol, item[1], diskno, disksitems,
pksize)
>>>>>>   File "./helpers/kvm2ovirt", line 102, in
download_volume
>>>>>>     stream.finish()
>>>>>>   File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 5501, in finish
>>>>>>     if ret == -1: raise libvirtError
('virStreamFinish() failed')
>>>>>> libvirt.libvirtError: internal error: I/O helper exited
abnormally
>>>>>>  
>>>>
>>>> The error message sucks, I'll send patches to improve it a
little at least.
>>>>
>>>> What's happening here is that the you haven't read all
the data you requested
>>>> (it's vol.download(path, offset, length, flags), length ==
0 means read the
>>>> whole file which I suspect you haven't done). In this case
the iohelper
>>>> program that libvirt uses won't complete feeding us all the
data, and it exits
>>>> with SIGPIPE when we close the read end of the pipe.
>>>>
>>>> Now whether that should actually be an error condition is open
to debate.
>>>> virStreamFinish docs make it sound like it's legitimate to
throw an error if
>>>> it appears that all requested data wasn't read.
>>> Thanks checking...
>> Thanks Cole,
>> I modify our script and for checking if stream.recv() returns zero the
finish
>> works fine.
>> Our API needs to know the size of bytes that you are going to stream
back
>> which is not provided by the allocation and capacity.
>> (ie the same as du -s /path/to/disk)
>>
>>   Shahar.
> We need the size of the image "Image end offset" you get from
"qemu-img check",
> Does libvirt have an API for that?
> 
Libvirt doesn't invoke qemu-img check anywhere AFAIK, so if that's the
only
way to get that info, then it isn't available

- Cole