openssh unix dev - Jan 2018 - PEM file opened without DIRECT I/O which makes private key readable by attacker exploiting MELTDOWN

On Sat, Jan 6, 2018 at 5:38 PM, Philipp Marek <philipp at marek.priv.at>
wrote:
> I think we are possibly interested in switching to DIRECT IO (given that it
>> bypasses any caching system including page cache) when reading *.PEM
file
>>
> Sorry, but this makes no sense.
> The data could just as well be read from the SSH process
> memory space.
>
I think that's actually not true. SSH process's stack and heap
aren't
mapped/stored into the kernel space mapped in all user space programs, so
how come a malicious program running in victim's system could have access
to stack/heap memory of SSH process which is only mapped in its address
space? Please correct me if I'm wrong.

>
> Direct IO has some additional complexity; this may well
> be avoided.
>
Actually, it's only about adding a flag to open and making sure IO is DMA
aligned.

>
>
> It makes *zero* sense to panic now and start "hardening"
> [which direct IO wouldn't even be!] individual programs -
> if separate memory spaces are not available,
>    "all hope is lost".
>
I agree with this sentiment though, better to think it through and come up
with informed decisions, but I think this is possibly a direction worth
considering.

>> I think we are possibly interested in switching to DIRECT IO (given 
>> that it
>>> bypasses any caching system including page cache) when reading
*.PEM
>>> file
>>> 
>> Sorry, but this makes no sense.
>> The data could just as well be read from the SSH process
>> memory space.
> I think that's actually not true. SSH process's stack and heap
aren't
> mapped/stored into the kernel space mapped in all user space programs, 
> so
> how come a malicious program running in victim's system could have 
> access
> to stack/heap memory of SSH process which is only mapped in its address
> space? Please correct me if I'm wrong.
Because with the recent faults all the memory could be read,
and the kernel has all of RAM mapped.

>> Direct IO has some additional complexity; this may well
>> be avoided.
> Actually, it's only about adding a flag to open and making sure IO is 
> DMA
> aligned.
Ha! If only.

What alignment would you use? open(2) says

   > In Linux alignment restrictions vary by filesystem
   > and kernel version and might be absent entirely.
   > However there is currently no filesystem-independent
   > interface for an application to discover these
   > restrictions for a given file or filesystem.

and

   > EINVAL The filesystem does not support the O_DIRECT flag.

So it's not quite that easy.


Furthermore -- as you said, direct IO means no page cache.

Well, that means that _every_ access has to be done from (or to) stable
storage... now imagine a loop going to a few hundred servers, or having
the home directory with the private key on a network filesystem, etc.

That's a lot of seeks and/or transfer operations.



So, please don't do that.

Hello,

On Sun, Jan 7, 2018 at 10:38 AM, Philipp Marek <philipp at marek.priv.at>
wrote:
> I think we are possibly interested in switching to DIRECT IO (given that it
>>>
>>>> bypasses any caching system including page cache) when reading
*.PEM
>>>> file
>>>>
>>>> Sorry, but this makes no sense.
>>> The data could just as well be read from the SSH process
>>> memory space.
>>>
>> I think that's actually not true. SSH process's stack and heap
aren't
>> mapped/stored into the kernel space mapped in all user space programs,
so
>> how come a malicious program running in victim's system could have
access
>> to stack/heap memory of SSH process which is only mapped in its address
>> space? Please correct me if I'm wrong.
>>
>
?That's the whole point of the many bugs shown by Google: don't
overtrust
page table isolation. It's broken. ?


> Because with the recent faults all the memory could be read,
> and the kernel has all of RAM mapped.

?I think this is not really an issue - that's a problem that the KPTI patch
will solve under linux (I don't know the status of this situation on other
OSes, but I suspect some kind of mitigation is going to be implemented).

The main problem is not related to meltdown (kernel memory read) but to
spectre (disclosure of memory of other processes), and direct IO is not
going to solve it - careful change of code will.
?
>
>
>
> Direct IO has some additional complexity; this may well
>>> be avoided.
>>>
>> Actually, it's only about adding a flag to open and making sure IO
is DMA
>> aligned.
>>
> Ha! If only.
>
> What alignment would you use? open(2) says
>
>   > In Linux alignment restrictions vary by filesystem
>   > and kernel version and might be absent entirely.
>   > However there is currently no filesystem-independent
>   > interface for an application to discover these
>   > restrictions for a given file or filesystem.
>
> and
>
>   > EINVAL The filesystem does not support the O_DIRECT flag.
>
> So it's not quite that easy.
>
>
> Furthermore -- as you said, direct IO means no page cache.
>
> Well, that means that _every_ access has to be done from (or to) stable
> storage... now imagine a loop going to a few hundred servers, or having
> the home directory with the private key on a network filesystem, etc.
>
> That's a lot of seeks and/or transfer operations.
>
>
>
> So, please don't do that.
>
>
?Best regards,

-- Emmanuel Deloget?

On Sun, Jan 7, 2018 at 7:38 AM, Philipp Marek <philipp at marek.priv.at>
wrote:
> I think we are possibly interested in switching to DIRECT IO (given that it
>>>
>>>> bypasses any caching system including page cache) when reading
*.PEM
>>>> file
>>>>
>>>> Sorry, but this makes no sense.
>>> The data could just as well be read from the SSH process
>>> memory space.
>>>
>> I think that's actually not true. SSH process's stack and heap
aren't
>> mapped/stored into the kernel space mapped in all user space programs,
so
>> how come a malicious program running in victim's system could have
access
>> to stack/heap memory of SSH process which is only mapped in its address
>> space? Please correct me if I'm wrong.
>>
> Because with the recent faults all the memory could be read,
> and the kernel has all of RAM mapped.

I was wrong here, thanks for pointing it out. Talked to Rik Van Riel who
helped making kpti happen, and the kernel indeed maps all of physical
memory in its address space. So private key going to page cache widens the
time range that private key is vulnerable (stays there until purged which
is a time proportional to memory available and caching frequency), whereas
with direct io flag the private key would only be vulnerable while a ssh
program is running.

>
>
>
> Direct IO has some additional complexity; this may well
>>> be avoided.
>>>
>> Actually, it's only about adding a flag to open and making sure IO
is DMA
>> aligned.
>>
> Ha! If only.
>
> What alignment would you use? open(2) says
>
>   > In Linux alignment restrictions vary by filesystem
>   > and kernel version and might be absent entirely.
>   > However there is currently no filesystem-independent
>   > interface for an application to discover these
>   > restrictions for a given file or filesystem.
>
> and
>
>   > EINVAL The filesystem does not support the O_DIRECT flag.
>
> So it's not quite that easy.
>
you're right, querying dma alignment may be specific to file system, like
XFS when uses a different block size or it falls back to buffered mode.
 4096 usually works but usually isn't enough for a widely used program
which needs to work everywhere with all kind of settings.

>
>
> Furthermore -- as you said, direct IO means no page cache.
>
> Well, that means that _every_ access has to be done from (or to) stable
> storage... now imagine a loop going to a few hundred servers, or having
> the home directory with the private key on a network filesystem, etc.
>
> That's a lot of seeks and/or transfer operations.
>
>
>
> So, please don't do that.
>
Agree with all the sentiment here. thanks for your time in explaining this
all!