search for: bytearray

Displaying 20 results from an estimated 87 matches for "bytearray".

2016 Sep 26
3
[PATCH] Improve memory management of nbdkit python plugin example
Hi, the nbdkit python plugin example has suboptimal memory management: - it creates the disk image as a string on init - it casts the string to bytearray on every read - it copies the string before and the string after the written region, then reassembles those pieces together with the written region to a new disk image string This is not a problem as long as the image is small, but in my tests with a 5 GB sized image nbdkit already used 15 GB RAM...
2016 Sep 26
1
Re: [PATCH] Improve memory management of nbdkit python plugin example
...ous attempt. On 26.09.2016 17:19, Pino Toscano wrote: > On Monday, 26 September 2016 17:07:41 CEST Carl-Daniel Hailfinger wrote: >> the nbdkit python plugin example has suboptimal memory management: >> - it creates the disk image as a string on init >> - it casts the string to bytearray on every read >> - it copies the string before and the string after the written region, >> then reassembles those pieces together with the written region to a new >> disk image string >> >> This is not a problem as long as the image is small, but in my tests >> w...
2019 Nov 23
2
Re: [PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...W.M. Jones <rjones@redhat.com> wrote: > > > +def pread(h, count, offset, flags): > > > + assert flags == 0 > > > + return h['disk'][offset:offset+count] > > > > Very nice and simple test plugin! > > > > But this returns always a bytearray, which is also what nbdkit python plugin > > expects. But real code using HTTPConnection return bytes: > > > > >>> c = http.client.HTTPSConnection("www.python.org") > > >>> c.request("GET", "/") > > >>> r = c.get...
2016 Sep 26
0
Re: [PATCH] Improve memory management of nbdkit python plugin example
Hi, On Monday, 26 September 2016 17:07:41 CEST Carl-Daniel Hailfinger wrote: > the nbdkit python plugin example has suboptimal memory management: > - it creates the disk image as a string on init > - it casts the string to bytearray on every read > - it copies the string before and the string after the written region, > then reassembles those pieces together with the written region to a new > disk image string > > This is not a problem as long as the image is small, but in my tests > with a 5 GB sized image...
2019 Nov 25
3
[PATCH nbdkit 0/2] python: Implement pread passing buffer for v2 API.
As suggested by Nir, here: https://www.redhat.com/archives/libguestfs/2019-November/thread.html#00220
2019 Nov 22
3
Re: [PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...gt; + if k == "cfg": > + cfg = pickle.loads (codecs.decode (v.encode(), "base64")) > + > +def config_complete (): > + print ("set_error = %r" % nbdkit.set_error) > + > +def open (readonly): > + return { > + 'disk': bytearray (cfg.get ('size', 0)) > + } > + > +def get_size (h): > + return len (h['disk']) > + > +def is_rotational(h): > + return cfg.get ('is_rotational', False) > + > +def can_multi_conn (h): > + return cfg.get ('can_multi_conn', Fal...
2015 Oct 29
2
[nbdkit] can't import mmap
...value)) def open(readonly): print ("open: readonly=%d" % readonly) f = open("disk","r+") m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE) return (f,m) def get_size(h): size = 1024*1024 def pread(h, count, offset): (f,m) = h bytes = bytearray(m[offset:offset+count]) return bytes def pwrite(h, buf, offset): (f,m) = h m[offset:offset+len(buf)] = bytearray(buf) When I run this: /usr/sbin/nbdkit -f -v python script=single_file.py I get this error: nbdkit: debug: registering /usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-...
2019 Nov 23
8
[PATCH nbdkit v3 0/7] Implement nbdkit API v2 for Python plugins.
v2 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00163.html I pushed patch 1 (with spelling fix), patch 4 and patch 5 since those were previously ACKed on the list. Differences in v3: - Add error checking to PyModule_AddIntConstant. - Use API_VERSION constant instead of function. - Add max API version supported to --dump-plugin output. - Print API_VERSION selected by
2019 Nov 25
7
[PATCH nbdkit v2 0/7] Implement nbdkit API v2 for Python plugins.
v3 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00209.html In v4: - Rebase on top of current master. Includes various fixes and updates required because of Nir's patches that went into master. - Fix api_version() -> API_VERSION in patch 2 noted previously on the mailing list. Rich.
2019 Nov 24
1
Re: [PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...def pread(h, count, offset, flags): > > > > > + assert flags == 0 > > > > > + return h['disk'][offset:offset+count] > > > > > > > > Very nice and simple test plugin! > > > > > > > > But this returns always a bytearray, which is also what nbdkit > python plugin > > > > expects. But real code using HTTPConnection return bytes: > > > > > > > > >>> c = http.client.HTTPSConnection("www.python.org") > > > > >>> c.request("GET", &q...
2018 Apr 05
5
[nbdkit PATCH 0/3] Test zero callback of python plugin
I'm planning on tweaking the language callbacks to support fua; first up is the python bindings. I want to move from: def zero(h, count, offset, may_trim): to a nicer: def zero(h, count, offset, may_trim=False, fua=False): where the C code passes keywords for the flags (we can add new flags as needed), perhaps by using introspection to learn whether the plugin has a mandatory may_trim
2019 Nov 24
0
Re: [PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...; wrote: > > > > +def pread(h, count, offset, flags): > > > > + assert flags == 0 > > > > + return h['disk'][offset:offset+count] > > > > > > Very nice and simple test plugin! > > > > > > But this returns always a bytearray, which is also what nbdkit python plugin > > > expects. But real code using HTTPConnection return bytes: > > > > > > >>> c = http.client.HTTPSConnection("www.python.org") > > > >>> c.request("GET", "/") > > &g...
2019 Nov 23
0
Re: [PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...Nov 22, 2019 at 9:55 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > +def pread(h, count, offset, flags): > > + assert flags == 0 > > + return h['disk'][offset:offset+count] > > Very nice and simple test plugin! > > But this returns always a bytearray, which is also what nbdkit python plugin > expects. But real code using HTTPConnection return bytes: > > >>> c = http.client.HTTPSConnection("www.python.org") > >>> c.request("GET", "/") > >>> r = c.getresponse() > >>...
2019 Nov 22
0
[PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.
...est.py | 20 +++-- 4 files changed, 148 insertions(+), 48 deletions(-) diff --git a/plugins/python/example.py b/plugins/python/example.py index 60f9d7f..25a0049 100644 --- a/plugins/python/example.py +++ b/plugins/python/example.py @@ -34,6 +34,13 @@ import errno disk = bytearray(1024 * 1024) +# There are several variants of the API. nbdkit will call this +# function first to determine which one you want to use. This is the +# latest version at the time this example was written. +def api_version(): + return 2 + + # This just prints the extra command line parameter...
2019 Nov 23
0
[PATCH nbdkit v3 2/7] python: Implement nbdkit API version 2.
...est.py | 20 +++-- 4 files changed, 156 insertions(+), 48 deletions(-) diff --git a/plugins/python/example.py b/plugins/python/example.py index 60f9d7f..c85d2f8 100644 --- a/plugins/python/example.py +++ b/plugins/python/example.py @@ -34,6 +34,12 @@ import errno disk = bytearray(1024 * 1024) +# There are several variants of the API. nbdkit will call this +# function first to determine which one you want to use. This is the +# latest version at the time this example was written. +API_VERSION = 2 + + # This just prints the extra command line parameters, but real plugi...
2019 Nov 22
0
[PATCH nbdkit v2 10/10] tests: Test the Python plugin thoroughly.
...= {} + +def config (k, v): + global cfg + if k == "cfg": + cfg = pickle.loads (codecs.decode (v.encode(), "base64")) + +def config_complete (): + print ("set_error = %r" % nbdkit.set_error) + +def open (readonly): + return { + 'disk': bytearray (cfg.get ('size', 0)) + } + +def get_size (h): + return len (h['disk']) + +def is_rotational(h): + return cfg.get ('is_rotational', False) + +def can_multi_conn (h): + return cfg.get ('can_multi_conn', False) + +def can_write (h): + return cfg.get (...
2019 Aug 10
0
[PATCH libnbd 1/5] python: Change aio_buffer into nbd.Buffer class.
...ator/generator +++ b/generator/generator @@ -3981,8 +3981,10 @@ raise_exception () pr "extern PyObject *nbd_internal_py_%s (PyObject *self, PyObject *args);\n" name; ) ([ "create"; "close"; - "alloc_aio_buffer"; "aio_buffer_from_bytearray"; - "aio_buffer_to_bytearray" ] @ List.map fst handle_calls); + "alloc_aio_buffer"; + "aio_buffer_from_bytearray"; + "aio_buffer_to_bytearray"; + "aio_buffer_size" ] @ List.map fst handle_calls); pr "\n&q...
2019 Nov 23
3
[PATCH nbdkit] python: Pass memoryview to pwrite()
Passing a memoryview we avoid unneeded copy of the original buffer. On the python side memoryview object can be used for slicing, writing to file, or sending to socket. This may break plugins assuming that the they get a bytearray, but good python code should not care about the type of the buffer, only about the behaviour. Testing with a plugin writing to /dev/null shows 2.7x speedup. Real plugin will probably show much smaller improvement. Without patch: $ time qemu-img convert -p -f raw -O raw -n /var/tmp/disk.img nbd:/...
2019 Nov 22
18
[PATCH nbdkit v2 00/10] Implement nbdkit API v2 for Python plugins.
v1: https://www.redhat.com/archives/libguestfs/2019-November/msg00153.html v2: - Fix implementation of can_cache. - Add implementation of can_fua. - Add a very thorough test suite which tests every command + flag combination.
2018 Aug 04
2
Re: [PATCH nbdkit] protocol: Implement NBD_OPT_GO.
...4**2, 'it works'); print c.read(1024**2, 8)" > INFO:nbd:Connecting to '/tmp/nbd.sock' 'export' > DEBUG:nbd:Received server flags: 3 > DEBUG:nbd:Sending client flags: 1: > DEBUG:nbd:Sending option: 'IHAVEOPT\x00\x00\x00\x07\x00\x00\x00\x0c' data: > bytearray(b'\x00\x00\x00\x06export\x00\x00') > DEBUG:nbd:Received reply [magic=3e889045565a9 option=7 type=3 len=12] > DEBUG:nbd:Received export info [size=1073741824 flags=109] > DEBUG:nbd:Received reply [magic=3e889045565a9 option=7 type=1 len=0] > INFO:nbd:Ready for transmission > i...