Syslinux - Nov 2016 - [PATCH 0/2] improve Lua API for files and initramfs objects

hpa at zytor.com writes:
> On November 19, 2016 6:06:43 AM PST, wferi at niif.hu wrote:
>
>>"H. Peter Anvin" <hpa at zytor.com> writes:
>>
>>> On 11/10/16 04:38, Paul Emmerich via Syslinux wrote:
>>> 
>>>> Ferenc W?gner <wferi at niif.hu>:
>>>>
>>>>> for reading configuration files from disk.  Does it not
work with
>>>>> HTTP/ TFTP for you?  Using that, load_file could be
implemented in
>>>>> Lua as:
>>>> 
>>>> that does work, yes. It just looked like a file object should
have
>>>> a some way to get the contents. I'm actually not using it
in our
>>>> scripts.
>>>
>>> It does seem useful.  So what you are saying here is that it
isn't
>>> actually broken, it is just suboptimal?
>>
>> No, this interface is completely missing at the moment.  You can read
>> a file using standard Lua I/O calls and work with its content.  Or
>> you can read a file using loadfile() provided by Syslinux, but then
>> you can't look at its contents from Lua.  You can run it as a
kernel
>> or use it as an initramfs, but the only available Lua methods are
>> name and size now.  Paul adds the data method to export the contents
>> to Lua, mainly for the sake of completeness, if I understand him
>> correctly.
>
> It was the "for completeness" part that I wanted to know.
>
> Incidentally, why do we copy the data if it is in a Lua string (or are
> we?) Seems pointless to me.
Are you asking about the following hunk?  (No, you probably don't, skip!)

+static int sl_filedata(lua_State * L)
+{
+    const syslinux_file *file = luaL_checkudata(L, 1, SYSLINUX_FILE);
+
+    lua_pushlstring(L, file->data, file->size);
+
+    return 1;
+}

This indeed copies the data from the syslinux_file structure (which is
an opaque SYSLINUX_FILE object on the Lua side) into an (interned) Lua
string to make it available to the Lua string functions.

You probably ask why we copy the data when we create those SYSLINUX_FILE
userdata objects.  We don't, the data is read directly into the
syslinux_file structure by the loadfile() Syslinux function.  This was
already the case when I first saw the Lua binding in Syslinux.  Thinking
about it now, the SYSLINUX_FILE userdata objects and their entire
interface could be replaced by Lua tables with two string fields for
much the same effect, unless the native Lua string I/O is much less
efficient than loadfile().  If we can drop the name method, even simple
Lua strings would suffice.  The complication arises when you assemble
initramfs objects incrementally: either you'll have to copy the data, or
keep a reference to each constituent Lua string and delay creating the
actual initramfs structure until its point of usage (and destroy it if
you somehow fail back to Lua).  This might be worth exploring.
-- 
Feri

On 11/20/16 00:13, Ferenc W?gner via Syslinux wrote:
> 
> This indeed copies the data from the syslinux_file structure (which is
> an opaque SYSLINUX_FILE object on the Lua side) into an (interned) Lua
> string to make it available to the Lua string functions.
> 
> You probably ask why we copy the data when we create those SYSLINUX_FILE
> userdata objects.  We don't, the data is read directly into the
> syslinux_file structure by the loadfile() Syslinux function.  This was
> already the case when I first saw the Lua binding in Syslinux.  Thinking
> about it now, the SYSLINUX_FILE userdata objects and their entire
> interface could be replaced by Lua tables with two string fields for
> much the same effect, unless the native Lua string I/O is much less
> efficient than loadfile().  If we can drop the name method, even simple
> Lua strings would suffice.  The complication arises when you assemble
> initramfs objects incrementally: either you'll have to copy the data,
or
> keep a reference to each constituent Lua string and delay creating the
> actual initramfs structure until its point of usage (and destroy it if
> you somehow fail back to Lua).  This might be worth exploring.
> 
This seems like a very sensible approach to me.  Keeping a reference and
just pointing into the Lua memory representation makes a lot more sense.

[f]loadfile() is "smart" in the sense that if, and only if, the size
of
the file is known ahead of time it allocates the buffer in one shot,
rather than using realloc().  Do note that [f]loadfile() (as opposed to
initramfs_load_file()) just loads data into a memory buffer.  If there
is a way to represent a memory buffer allocated by C as a Lua string
that would pretty much take care of the same thing.  floadfile() even
allows a chunk of data to be prepended to the buffer, which could be
useful in the case of a Lua string needing something prepended to the
data buffer.

Creating struct initramfs at the point of execution seems reasonable,
but on the other hand, is there a reason we can't simply add a reference
to the Lua string and let it be referenced by struct initramfs using
initramfs_add_file()?  The main shortcoming I see with that is the
object is opaque, and it could be nice to be allowed to examine it, but
that seems like a second-order feature.

	-hpa

"H. Peter Anvin" <hpa at zytor.com> writes:
> On 11/20/16 00:13, Ferenc W?gner via Syslinux wrote:
> 
>> This indeed copies the data from the syslinux_file structure (which is
>> an opaque SYSLINUX_FILE object on the Lua side) into an (interned) Lua
>> string to make it available to the Lua string functions.
>> 
>> You probably ask why we copy the data when we create those
SYSLINUX_FILE
>> userdata objects.  We don't, the data is read directly into the
>> syslinux_file structure by the loadfile() Syslinux function.  This was
>> already the case when I first saw the Lua binding in Syslinux. 
Thinking
>> about it now, the SYSLINUX_FILE userdata objects and their entire
>> interface could be replaced by Lua tables with two string fields for
>> much the same effect, unless the native Lua string I/O is much less
>> efficient than loadfile().  If we can drop the name method, even simple
>> Lua strings would suffice.  The complication arises when you assemble
>> initramfs objects incrementally: either you'll have to copy the
data, or
>> keep a reference to each constituent Lua string and delay creating the
>> actual initramfs structure until its point of usage (and destroy it if
>> you somehow fail back to Lua).  This might be worth exploring.
>> 
>
> This seems like a very sensible approach to me.  Keeping a reference and
> just pointing into the Lua memory representation makes a lot more sense.
>
> [f]loadfile() is "smart" in the sense that if, and only if, the
size of
> the file is known ahead of time it allocates the buffer in one shot,
> rather than using realloc().  Do note that [f]loadfile() (as opposed to
> initramfs_load_file()) just loads data into a memory buffer.
The catch is that it's impossible to create a Lua string without copying
the memory buffer constituting it.  At least I couldn't find a way.  It
can be an issue even when Lua itself reads a file into a string: first
it reads the stream contents into a series of buffers of geometrically
growing sizes, then collects them all into the final string.  This
guarrantees at least twice the memory usage of the minimum needed (in
the best case).  We could replace the implementation of read_all() in
liolib.c with floadfile() to get this "best case" all the time, but it
we can't afford the final copy (the double memory footprint) anyway, it
doesn't gain us much.
> If there is a way to represent a memory buffer allocated by C as a Lua
> string that would pretty much take care of the same thing.
I'm afraid this is not possible, which pretty much ends this story.
An unfortunate turn.
> floadfile() even allows a chunk of data to be prepended to the buffer,
> which could be useful in the case of a Lua string needing something
> prepended to the data buffer.
That's clever and gets us halfway there.  But that other half...
certainly not supported by the official API.  And we don't really want
to use kernels and initramfs images as strings, so it probably isn't
worth mucking with the internals.
> Creating struct initramfs at the point of execution seems reasonable,
> but on the other hand, is there a reason we can't simply add a
reference
> to the Lua string and let it be referenced by struct initramfs using
> initramfs_add_file()?  The main shortcoming I see with that is the
> object is opaque, and it could be nice to be allowed to examine it, but
> that seems like a second-order feature.
I'm not sure I follow you.  Based on the above, we can't exclusively use
Lua strings to build the initramfs.  We can represent the initramfs as a
Lua list (a special table), which can contain an arbitrary mix of
Syslinux file objects (opaque from Lua) and Lua strings, and build the
struct initramfs at the point of execution.
-- 
Regards,
Feri