Xen devel - Oct 2013 - Problems when using latest git tree to boot xen on OMAP5

Hi Julien,

Since Ian has merged most my OMAP5 patches, I decide to try to update my working
tree to it today. However, it seems it doesn't work as expected. There
mainly two kind of problems that I met:

1. It seems that xen no longer maps some of io memory regions described in dts.
For example, the memory regions in "ocp" node of omap5.dtsi.

2. commit c88d32f4 seems to be conflict with the UART console hack in DTS, where
we mark the status of target UART "disabled".

Both of the problems look like being introduced by new device tree related
patches.

Any ideas？

Cheers,

Baozi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Oct 4, 2013, at 10:11 PM, Chen Baozi <baozich@gmail.com> wrote:
> Hi Julien,
> 
> Since Ian has merged most my OMAP5 patches, I decide to try to update my
working tree to it today. However, it seems it doesn't work as expected.
There mainly two kind of problems that I met:
> 
> 1. It seems that xen no longer maps some of io memory regions described in
dts. For example, the memory regions in "ocp" node of omap5.dtsi.
I looked into the codes today. It is because of checking
dt_device_is_available() before map_device(). Some node, for example
/ocp/mmc@480d1000, is disabled in omap5-uevm.dts separately from omap5.dtsi
where it is original defined. In this case, xen won't map its memory region
for dom0. However, it seems dom0 kernel still accesses those regions.

I guess there would be a bug dealing with this situations?

Cheers,

Baozi
> 
> 2. commit c88d32f4 seems to be conflict with the UART console hack in DTS,
where we mark the status of target UART "disabled".
> 
> Both of the problems look like being introduced by new device tree related
patches.
> 
> Any ideas？
> 
> Cheers,
> 
> Baozi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, 2013-10-06 at 22:21 +0800, Chen Baozi wrote:
> On Oct 4, 2013, at 10:11 PM, Chen Baozi <baozich@gmail.com> wrote:
> 
> > Hi Julien,
> > 
> > Since Ian has merged most my OMAP5 patches, I decide to try to update
my working tree to it today. However, it seems it doesn''t work as
expected. There mainly two kind of problems that I met:
> > 
> > 1. It seems that xen no longer maps some of io memory regions
described in dts. For example, the memory regions in "ocp" node of
omap5.dtsi.
> 
> I looked into the codes today. It is because of checking
> dt_device_is_available() before map_device(). Some node, for
> example /ocp/mmc@480d1000, is disabled in omap5-uevm.dts separately
> from omap5.dtsi where it is original defined. In this case, xen
won''t
> map its memory region for dom0. However, it seems dom0 kernel still
> accesses those regions.
> I guess there would be a bug dealing with this situations?
It is certainly a bug in the kernel if it is accessing something which
is disabled. It may also independently be a bug in the dts that this
devices is disabled.

However in v3.12-rc4 I don''t see mmc@480d1000 being disabled in
omap5-uevm.dts and I can''t see anything in the history of that file
either. Where did your copy come from?

Hopefully we won''t need a converse to the device blacklist, i.e. a
whitelist of disabled devices to pass through regardless of the DTS
enabled state.

Ian.

On 10/07/2013 04:39 PM, Ian Campbell wrote:
> On Sun, 2013-10-06 at 22:21 +0800, Chen Baozi wrote:
>> On Oct 4, 2013, at 10:11 PM, Chen Baozi <baozich@gmail.com>
wrote:
>>
>>> Hi Julien,
>>>
>>> Since Ian has merged most my OMAP5 patches, I decide to try to
update my working tree to it today. However, it seems it doesn''t work
as expected. There mainly two kind of problems that I met:
>>>
>>> 1. It seems that xen no longer maps some of io memory regions
described in dts. For example, the memory regions in "ocp" node of
omap5.dtsi.
>>
>> I looked into the codes today. It is because of checking
>> dt_device_is_available() before map_device(). Some node, for
>> example /ocp/mmc@480d1000, is disabled in omap5-uevm.dts separately
>> from omap5.dtsi where it is original defined. In this case, xen
won''t
>> map its memory region for dom0. However, it seems dom0 kernel still
>> accesses those regions.
> 
>> I guess there would be a bug dealing with this situations?
> 
> It is certainly a bug in the kernel if it is accessing something which
> is disabled. It may also independently be a bug in the dts that this
> devices is disabled.
> 
> However in v3.12-rc4 I don''t see mmc@480d1000 being disabled in
> omap5-uevm.dts and I can''t see anything in the history of that
file
> either. Where did your copy come from?
I''m currently working on the "omap5-v3.11-rc3" branch from
git://github.com/rogerq/linux.git, which contains a few necessary
platform patches not upstreamed. In omap5-uevm.dts, there are lines like:

253 &mmc4 {
254     status = "disabled";
255 };
256
257 &mmc5 {
258     status = "disabled";
259 };

the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:

417         mmc4: mmc@480d1000 {

I checked Linus'' mainline git tree. It is the same about disabled mmc4
in omap5-uevm.dts. And the change is introduced in commit 5dd18b0 of the
mainline kernel.

Anyway, I''ll see what exactly happened in the dom0 kernel dealing with
those "disabled" regions.

Cheers,

Baozi
> 
> Hopefully we won''t need a converse to the device blacklist, i.e. a
> whitelist of disabled devices to pass through regardless of the DTS
> enabled state.
> 
> Ian.
> 
>

On Mon, 2013-10-07 at 17:58 +0800, Chen Baozi wrote:
> On 10/07/2013 04:39 PM, Ian Campbell wrote:
> > On Sun, 2013-10-06 at 22:21 +0800, Chen Baozi wrote:
> >> On Oct 4, 2013, at 10:11 PM, Chen Baozi <baozich@gmail.com>
wrote:
> >>
> >>> Hi Julien,
> >>>
> >>> Since Ian has merged most my OMAP5 patches, I decide to try to
update my working tree to it today. However, it seems it doesn''t work
as expected. There mainly two kind of problems that I met:
> >>>
> >>> 1. It seems that xen no longer maps some of io memory regions
described in dts. For example, the memory regions in "ocp" node of
omap5.dtsi.
> >>
> >> I looked into the codes today. It is because of checking
> >> dt_device_is_available() before map_device(). Some node, for
> >> example /ocp/mmc@480d1000, is disabled in omap5-uevm.dts
separately
> >> from omap5.dtsi where it is original defined. In this case, xen
won''t
> >> map its memory region for dom0. However, it seems dom0 kernel
still
> >> accesses those regions.
> > 
> >> I guess there would be a bug dealing with this situations?
> > 
> > It is certainly a bug in the kernel if it is accessing something which
> > is disabled. It may also independently be a bug in the dts that this
> > devices is disabled.
> > 
> > However in v3.12-rc4 I don''t see mmc@480d1000 being disabled
in
> > omap5-uevm.dts and I can''t see anything in the history of
that file
> > either. Where did your copy come from?
> 
> I''m currently working on the "omap5-v3.11-rc3" branch
from
> git://github.com/rogerq/linux.git, which contains a few necessary
> platform patches not upstreamed. In omap5-uevm.dts, there are lines like:
> 
> 253 &mmc4 {
> 254     status = "disabled";
> 255 };
> 256
> 257 &mmc5 {
> 258     status = "disabled";
> 259 };
> 
> the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
> 
> 417         mmc4: mmc@480d1000 {
> 
> I checked Linus'' mainline git tree. It is the same about disabled
mmc4
> in omap5-uevm.dts. And the change is introduced in commit 5dd18b0 of the
> mainline kernel.
I was grepping for mmc@480d1 not mmc4 so I missed this.
> Anyway, I''ll see what exactly happened in the dom0 kernel dealing
with
> those "disabled" regions.
Thanks.
Ian.
> 
> Cheers,
> 
> Baozi
> 
> > 
> > Hopefully we won''t need a converse to the device blacklist,
i.e. a
> > whitelist of disabled devices to pass through regardless of the DTS
> > enabled state.
> > 
> > Ian.
> > 
> >

On 10/04/2013 03:11 PM, Chen Baozi wrote:
> Hi Julien,
Hi Chen,
> 2. commit c88d32f4 seems to be conflict with the UART console hack in DTS,
where we mark the status of target UART "disabled".
You can remove the hack from the DTS. It was used to avoid dom0 to use
the UART. Now, Xen removes the UART from the device tree.

Cheers,

-- 
Julien Grall

On 10/07/2013 10:58 AM, Chen Baozi wrote:
> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>
>> It is certainly a bug in the kernel if it is accessing something which
>> is disabled. It may also independently be a bug in the dts that this
>> devices is disabled.
>>
>> However in v3.12-rc4 I don''t see mmc@480d1000 being disabled
in
>> omap5-uevm.dts and I can''t see anything in the history of that
file
>> either. Where did your copy come from?
> 
> I''m currently working on the "omap5-v3.11-rc3" branch
from
> git://github.com/rogerq/linux.git, which contains a few necessary
> platform patches not upstreamed. In omap5-uevm.dts, there are lines like:
> 
> 253 &mmc4 {
> 254     status = "disabled";
> 255 };
> 256
> 257 &mmc5 {
> 258     status = "disabled";
> 259 };
> 
> the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
> 
> 417         mmc4: mmc@480d1000 {
> 
> I checked Linus'' mainline git tree. It is the same about disabled
mmc4
> in omap5-uevm.dts. And the change is introduced in commit 5dd18b0 of the
> mainline kernel.
> 
> Anyway, I''ll see what exactly happened in the dom0 kernel dealing
with
> those "disabled" regions.
I looked at the Linux code. It will populate the different devices via
the of_platform_populate (drivers/of/platform.c).

This function checks in of_platform_create_pdata if the device is
available. So the mmc driver (driver/mmc/host/omap_hsmmc.c) should not
be called for mmc4.

Can you find who generates the data abort? You can use addr2line for
this purpose.

Cheers,

-- 
Julien Grall

On Oct 7, 2013, at 6:59 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 10/07/2013 10:58 AM, Chen Baozi wrote:
>> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>> 
>>> It is certainly a bug in the kernel if it is accessing something
which
>>> is disabled. It may also independently be a bug in the dts that
this
>>> devices is disabled.
>>> 
>>> However in v3.12-rc4 I don''t see mmc@480d1000 being
disabled in
>>> omap5-uevm.dts and I can''t see anything in the history of
that file
>>> either. Where did your copy come from?
>> 
>> I''m currently working on the "omap5-v3.11-rc3"
branch from
>> git://github.com/rogerq/linux.git, which contains a few necessary
>> platform patches not upstreamed. In omap5-uevm.dts, there are lines
like:
>> 
>> 253 &mmc4 {
>> 254     status = "disabled";
>> 255 };
>> 256
>> 257 &mmc5 {
>> 258     status = "disabled";
>> 259 };
>> 
>> the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
>> 
>> 417         mmc4: mmc@480d1000 {
>> 
>> I checked Linus'' mainline git tree. It is the same about
disabled mmc4
>> in omap5-uevm.dts. And the change is introduced in commit 5dd18b0 of
the
>> mainline kernel.
>> 
>> Anyway, I''ll see what exactly happened in the dom0 kernel
dealing with
>> those "disabled" regions.
> 
> I looked at the Linux code. It will populate the different devices via
> the of_platform_populate (drivers/of/platform.c).
> 
> This function checks in of_platform_create_pdata if the device is
> available. So the mmc driver (driver/mmc/host/omap_hsmmc.c) should not
> be called for mmc4.
> 
> Can you find who generates the data abort? You can use addr2line for
> this purpose.
addr2line would lead to __raw_readl() in arch/arm/include/asm/io.h. So
it doesn''t that useful.

I guess omap platform codes access the memory address when doing low-level
setup and there are some hard coded address in its implementation. Note that
there are hard coded "mmc4" structure in 
arch/arm/mach-omap2/omap_hwmod_54xx_data.c

Cheers,

Baozi

On 10/07/2013 04:28 PM, Chen Baozi wrote:
> 
> addr2line would lead to __raw_readl() in arch/arm/include/asm/io.h. So
> it doesn''t that useful.
Ah right ... but __raw_readl is usually inlined so you can use objdump
and find the instruction. After that, if you go a bit above you can find
the caller name.

-- 
Julien Grall

On Oct 7, 2013, at 11:44 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 10/07/2013 04:28 PM, Chen Baozi wrote:
> 
>> 
>> addr2line would lead to __raw_readl() in arch/arm/include/asm/io.h. So
>> it doesn''t that useful.
> 
> Ah right ... but __raw_readl is usually inlined so you can use objdump
> and find the instruction. After that, if you go a bit above you can find
> the caller name.
It is called by omap_hwmod_read(). And it is mostly like to be called
ultimately by omap_device_build_from_dt in _omap_device_notifier_call.

I guess it might relate to the OCP platform bus initialization and 
hard coded "omap_hwmod" information. But I can''t point out
the exact
calling stack of omap_hwmod_read sadly...

Cheers,

Baozi

On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall
wrote:
> On 10/07/2013 10:58 AM, Chen Baozi wrote:
> > On 10/07/2013 04:39 PM, Ian Campbell wrote:
> >>
> >> It is certainly a bug in the kernel if it is accessing something
which
> >> is disabled. It may also independently be a bug in the dts that
this
> >> devices is disabled.
> >>
> >> However in v3.12-rc4 I don''t see mmc@480d1000 being
disabled in
> >> omap5-uevm.dts and I can''t see anything in the history of
that file
> >> either. Where did your copy come from?
> > 
> > I''m currently working on the "omap5-v3.11-rc3"
branch from
> > git://github.com/rogerq/linux.git, which contains a few necessary
> > platform patches not upstreamed. In omap5-uevm.dts, there are lines
like:
> > 
> > 253 &mmc4 {
> > 254     status = "disabled";
> > 255 };
> > 256
> > 257 &mmc5 {
> > 258     status = "disabled";
> > 259 };
> > 
> > the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
> > 
> > 417         mmc4: mmc@480d1000 {
> > 
> > I checked Linus'' mainline git tree. It is the same about
disabled mmc4
> > in omap5-uevm.dts. And the change is introduced in commit 5dd18b0 of
the
> > mainline kernel.
> > 
> > Anyway, I''ll see what exactly happened in the dom0 kernel
dealing with
> > those "disabled" regions.
> 
> I looked at the Linux code. It will populate the different devices via
> the of_platform_populate (drivers/of/platform.c).
> 
> This function checks in of_platform_create_pdata if the device is
> available. So the mmc driver (driver/mmc/host/omap_hsmmc.c) should not
> be called for mmc4.
I''ve discuessed this issue on linux-omap@vger.kernel.org. The TI guy
says
that "DT disabled" means that device won''t be created but
hwmod bus
initially would try to initialize all supported modules by doing reset
access in their io memory address regions. That''s why dom0 have
accessed
mmc4 address even though it has been disabled.

Cheers,

Baozi

On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
> On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall wrote:
> > On 10/07/2013 10:58 AM, Chen Baozi wrote:
> > > On 10/07/2013 04:39 PM, Ian Campbell wrote:
> > >>
> > >> It is certainly a bug in the kernel if it is accessing
something which
> > >> is disabled. It may also independently be a bug in the dts
that this
> > >> devices is disabled.
> > >>
> > >> However in v3.12-rc4 I don''t see mmc@480d1000 being
disabled in
> > >> omap5-uevm.dts and I can''t see anything in the
history of that file
> > >> either. Where did your copy come from?
> > > 
> > > I''m currently working on the "omap5-v3.11-rc3"
branch from
> > > git://github.com/rogerq/linux.git, which contains a few necessary
> > > platform patches not upstreamed. In omap5-uevm.dts, there are
lines like:
> > > 
> > > 253 &mmc4 {
> > > 254     status = "disabled";
> > > 255 };
> > > 256
> > > 257 &mmc5 {
> > > 258     status = "disabled";
> > > 259 };
> > > 
> > > the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
> > > 
> > > 417         mmc4: mmc@480d1000 {
> > > 
> > > I checked Linus'' mainline git tree. It is the same about
disabled mmc4
> > > in omap5-uevm.dts. And the change is introduced in commit 5dd18b0
of the
> > > mainline kernel.
> > > 
> > > Anyway, I''ll see what exactly happened in the dom0
kernel dealing with
> > > those "disabled" regions.
> > 
> > I looked at the Linux code. It will populate the different devices via
> > the of_platform_populate (drivers/of/platform.c).
> > 
> > This function checks in of_platform_create_pdata if the device is
> > available. So the mmc driver (driver/mmc/host/omap_hsmmc.c) should not
> > be called for mmc4.
> 
> I''ve discuessed this issue on linux-omap@vger.kernel.org. The TI
guy says
> that "DT disabled" means that device won''t be created
but hwmod bus
> initially would try to initialize all supported modules by doing reset
> access in their io memory address regions. That''s why dom0 have
accessed
> mmc4 address even though it has been disabled.
ePAPR lists the option of "status = fail" which is "Indicates
that the
device is not operational. A serious error was detected in the device,
and it is unlikely to become operational without repair." I''m not
sure
that is quite right though.

Rather than whitelisting and mapping disabled devices through perhaps we
should implement them as read 0xf (or 0x0) and write ignore?

Or maybe we should just be mapping non-blacklisted disabled devices
through to dom0, for it to use or ignore as it pleases. Julien, what was
the reasoning here again?

Ian.

On Oct 9, 2013, at 4:11 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
>> On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall wrote:
>>> On 10/07/2013 10:58 AM, Chen Baozi wrote:
>>>> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>>>> 
>>>>> It is certainly a bug in the kernel if it is accessing
something which
>>>>> is disabled. It may also independently be a bug in the dts
that this
>>>>> devices is disabled.
>>>>> 
>>>>> However in v3.12-rc4 I don''t see mmc@480d1000
being disabled in
>>>>> omap5-uevm.dts and I can''t see anything in the
history of that file
>>>>> either. Where did your copy come from?
>>>> 
>>>> I''m currently working on the
"omap5-v3.11-rc3" branch from
>>>> git://github.com/rogerq/linux.git, which contains a few
necessary
>>>> platform patches not upstreamed. In omap5-uevm.dts, there are
lines like:
>>>> 
>>>> 253 &mmc4 {
>>>> 254     status = "disabled";
>>>> 255 };
>>>> 256
>>>> 257 &mmc5 {
>>>> 258     status = "disabled";
>>>> 259 };
>>>> 
>>>> the mmc4 refers to mmc@480d1000, which defines at omap5.dtsi:
>>>> 
>>>> 417         mmc4: mmc@480d1000 {
>>>> 
>>>> I checked Linus'' mainline git tree. It is the same
about disabled mmc4
>>>> in omap5-uevm.dts. And the change is introduced in commit
5dd18b0 of the
>>>> mainline kernel.
>>>> 
>>>> Anyway, I''ll see what exactly happened in the dom0
kernel dealing with
>>>> those "disabled" regions.
>>> 
>>> I looked at the Linux code. It will populate the different devices
via
>>> the of_platform_populate (drivers/of/platform.c).
>>> 
>>> This function checks in of_platform_create_pdata if the device is
>>> available. So the mmc driver (driver/mmc/host/omap_hsmmc.c) should
not
>>> be called for mmc4.
>> 
>> I''ve discuessed this issue on linux-omap@vger.kernel.org. The
TI guy says
>> that "DT disabled" means that device won''t be
created but hwmod bus
>> initially would try to initialize all supported modules by doing reset
>> access in their io memory address regions. That''s why dom0
have accessed
>> mmc4 address even though it has been disabled.
> 
> ePAPR lists the option of "status = fail" which is
"Indicates that the
> device is not operational. A serious error was detected in the device,
> and it is unlikely to become operational without repair." I''m
not sure
> that is quite right though.
I guess when TI implemented omap_hwmod, DT is still not widely accepted 
on ARM platform. So when omap_hwmod doing initialization, it doesn''t
follow ePAPR standard strictly. Maybe we can call this a historical
issue?

The TI people said they have addressed proposals that those resets will
be removed to avoid our issue. But it seems this won''t be too soon I
guess...

Cheers,

Baozi
> 
> Rather than whitelisting and mapping disabled devices through perhaps we
> should implement them as read 0xf (or 0x0) and write ignore?
> 
> Or maybe we should just be mapping non-blacklisted disabled devices
> through to dom0, for it to use or ignore as it pleases. Julien, what was
> the reasoning here again?
> 
> Ian.
>

On 10/09/2013 09:11 AM, Ian Campbell wrote:
> Rather than whitelisting and mapping disabled devices through perhaps we
> should implement them as read 0xf (or 0x0) and write ignore?
We can''t assume that these values (0x0 or 0xf) won''t affect
dom0 behaviour.

What about a new quirk to map all disabled device in dom0
(PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?

-- 
Julien Grall

On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
> On 10/09/2013 09:11 AM, Ian Campbell wrote:
> > Rather than whitelisting and mapping disabled devices through perhaps
we
> > should implement them as read 0xf (or 0x0) and write ignore?
> 
> We can''t assume that these values (0x0 or 0xf) won''t
affect dom0 behaviour.
0xff is what you would typically get back if there was no actual
hardware present.
> What about a new quirk to map all disabled device in dom0
> (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
Why is this not the default behaviour?

On Wed, 9 Oct 2013, Ian Campbell wrote:
> On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
> > On 10/09/2013 09:11 AM, Ian Campbell wrote:
> > > Rather than whitelisting and mapping disabled devices through
perhaps we
> > > should implement them as read 0xf (or 0x0) and write ignore?
> > 
> > We can''t assume that these values (0x0 or 0xf) won''t
affect dom0 behaviour.
> 
> 0xff is what you would typically get back if there was no actual
> hardware present.
> 
> > What about a new quirk to map all disabled device in dom0
> > (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
> 
> Why is this not the default behaviour?
I think this is not a good idea: it prevents us from assigning one
device directly to a domain other than dom0 from Xen at boot time
(without going through something like PCI passthough).
Also if we really don''t want dom0 to use a device, it makes perfect
sense that we don''t map the device to it.

I would rather introduce PLATFORM_QUIRK_DOM0_EMULATE_DISABLED_DEVICE
and implement it with a simple write ignore, read 0xff.

On 10/09/2013 12:19 PM, Ian Campbell wrote:
> On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
>> On 10/09/2013 09:11 AM, Ian Campbell wrote:
>>> Rather than whitelisting and mapping disabled devices through
perhaps we
>>> should implement them as read 0xf (or 0x0) and write ignore?
>>
>> We can''t assume that these values (0x0 or 0xf) won''t
affect dom0 behaviour.
> 
> 0xff is what you would typically get back if there was no actual
> hardware present.
> 
>> What about a new quirk to map all disabled device in dom0
>> (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
> 
> Why is this not the default behaviour?
From our discussion a months a ago (http://patches.linaro.org/19590/),
status = "disabled" means : "not present/wired-up on this variant
of
the platform".

As Linux should cope with this status, why do we need to map these
devices in DOM0?

If we go further, with passthrough we may want remove device from the
device tree. If Linux has hardcoded device (but also in the device tree)
it will likely fail.

-- 
Julien Grall

On Wed, 2013-10-09 at 12:29 +0100, Stefano Stabellini
wrote:
> On Wed, 9 Oct 2013, Ian Campbell wrote:
> > On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
> > > On 10/09/2013 09:11 AM, Ian Campbell wrote:
> > > > Rather than whitelisting and mapping disabled devices
through perhaps we
> > > > should implement them as read 0xf (or 0x0) and write ignore?
> > > 
> > > We can''t assume that these values (0x0 or 0xf)
won''t affect dom0 behaviour.
> > 
> > 0xff is what you would typically get back if there was no actual
> > hardware present.
> > 
> > > What about a new quirk to map all disabled device in dom0
> > > (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
> > 
> > Why is this not the default behaviour?
> 
> I think this is not a good idea: it prevents us from assigning one
> device directly to a domain other than dom0 from Xen at boot time
> (without going through something like PCI passthough).
No it doesn''t, it would just be a default behaviour which would be
overrideable by some new command line option.
 
Note that we don''t even go as far as you suggest for PCI passthrough on
x86, where dom0 still sees the device and the hide option is a dom0 one.
> Also if we really don''t want dom0 to use a device, it makes
perfect
> sense that we don''t map the device to it.
> 
> I would rather introduce PLATFORM_QUIRK_DOM0_EMULATE_DISABLED_DEVICE
> and implement it with a simple write ignore, read 0xff.

On Wed, 2013-10-09 at 12:29 +0100, Julien Grall wrote:
> On 10/09/2013 12:19 PM, Ian Campbell wrote:
> > On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
> >> On 10/09/2013 09:11 AM, Ian Campbell wrote:
> >>> Rather than whitelisting and mapping disabled devices through
perhaps we
> >>> should implement them as read 0xf (or 0x0) and write ignore?
> >>
> >> We can''t assume that these values (0x0 or 0xf)
won''t affect dom0 behaviour.
> > 
> > 0xff is what you would typically get back if there was no actual
> > hardware present.
> > 
> >> What about a new quirk to map all disabled device in dom0
> >> (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
> > 
> > Why is this not the default behaviour?
> 
> From our discussion a months a ago (http://patches.linaro.org/19590/),
> status = "disabled" means : "not present/wired-up on this
variant of
> the platform".
Actually ePAPR says:
        Indicates that the device is not presently operational, but it
        might
        become operational in the future (for example, something is not
        plugged in, or switched off).
        Refer to the device binding for details on what disabled means
        for
        a given device.

This doesn''t strictly speaking say "and you must not prod any
associated
MMIO".
> As Linux should cope with this status, why do we need to map these
> devices in DOM0?
Because in practice Linux doesn''t actually cope like you asserted it
would.
> If we go further, with passthrough we may want remove device from the
> device tree. If Linux has hardcoded device (but also in the device tree)
> it will likely fail.
If/when we come to be passing through a device which Linux is poking
despite it being disabled then we have a kernel bug which needs fixing,
nothing more, and there is no reason for this to be a factor in deciding
what to do by default.

Ian.

On Wed, 9 Oct 2013, Ian Campbell wrote:
> On Wed, 2013-10-09 at 12:29 +0100, Stefano Stabellini wrote:
> > On Wed, 9 Oct 2013, Ian Campbell wrote:
> > > On Wed, 2013-10-09 at 12:15 +0100, Julien Grall wrote:
> > > > On 10/09/2013 09:11 AM, Ian Campbell wrote:
> > > > > Rather than whitelisting and mapping disabled devices
through perhaps we
> > > > > should implement them as read 0xf (or 0x0) and write
ignore?
> > > > 
> > > > We can''t assume that these values (0x0 or 0xf)
won''t affect dom0 behaviour.
> > > 
> > > 0xff is what you would typically get back if there was no actual
> > > hardware present.
> > > 
> > > > What about a new quirk to map all disabled device in dom0
> > > > (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
> > > 
> > > Why is this not the default behaviour?
> > 
> > I think this is not a good idea: it prevents us from assigning one
> > device directly to a domain other than dom0 from Xen at boot time
> > (without going through something like PCI passthough).
> 
> No it doesn''t, it would just be a default behaviour which would be
> overrideable by some new command line option.
I see.
I would still keep it off by default and maybe enable the workaround
only where we actually need it. If we enable it by default we might not
know exactly which ones are the platforms that actually require it.

> Note that we don''t even go as far as you suggest for PCI
passthrough on
> x86, where dom0 still sees the device and the hide option is a dom0 one.
The main difference is that in that case we can hide a given device
using pciback (or pcistub), in this case we can''t. Also on x86 we are
constrained by the PCI bus, while here we might be able to have a better
design for non-PCI devices.

On Wed, Oct 09, 2013 at 09:11:16AM +0100, Ian Campbell
wrote:
> On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
> > On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall wrote:
> > > On 10/07/2013 10:58 AM, Chen Baozi wrote:
> > > > On 10/07/2013 04:39 PM, Ian Campbell wrote:
> > > >>
> > > >> It is certainly a bug in the kernel if it is accessing
something which
> > > >> is disabled. It may also independently be a bug in the
dts that this
> > > >> devices is disabled.
> > > >>
> > > >> However in v3.12-rc4 I don''t see mmc@480d1000
being disabled in
> > > >> omap5-uevm.dts and I can''t see anything in the
history of that file
> > > >> either. Where did your copy come from?
> > > > 
> > > > I''m currently working on the
"omap5-v3.11-rc3" branch from
> > > > git://github.com/rogerq/linux.git, which contains a few
necessary
> > > > platform patches not upstreamed. In omap5-uevm.dts, there
are lines like:
> > > > 
> > > > 253 &mmc4 {
> > > > 254     status = "disabled";
> > > > 255 };
> > > > 256
> > > > 257 &mmc5 {
> > > > 258     status = "disabled";
> > > > 259 };
> > > > 
> > > > the mmc4 refers to mmc@480d1000, which defines at
omap5.dtsi:
> > > > 
> > > > 417         mmc4: mmc@480d1000 {
> > > > 
> > > > I checked Linus'' mainline git tree. It is the same
about disabled mmc4
> > > > in omap5-uevm.dts. And the change is introduced in commit
5dd18b0 of the
> > > > mainline kernel.
> > > > 
> > > > Anyway, I''ll see what exactly happened in the dom0
kernel dealing with
> > > > those "disabled" regions.
> > > 
> > > I looked at the Linux code. It will populate the different
devices via
> > > the of_platform_populate (drivers/of/platform.c).
> > > 
> > > This function checks in of_platform_create_pdata if the device is
> > > available. So the mmc driver (driver/mmc/host/omap_hsmmc.c)
should not
> > > be called for mmc4.
> > 
> > I''ve discuessed this issue on linux-omap@vger.kernel.org. The
TI guy says
> > that "DT disabled" means that device won''t be
created but hwmod bus
> > initially would try to initialize all supported modules by doing reset
> > access in their io memory address regions. That''s why dom0
have accessed
> > mmc4 address even though it has been disabled.
> 
> ePAPR lists the option of "status = fail" which is
"Indicates that the
> device is not operational. A serious error was detected in the device,
> and it is unlikely to become operational without repair." I''m
not sure
> that is quite right though.
> 
> Rather than whitelisting and mapping disabled devices through perhaps we
> should implement them as read 0xf (or 0x0) and write ignore?
> 
> Or maybe we should just be mapping non-blacklisted disabled devices
> through to dom0, for it to use or ignore as it pleases. Julien, what was
> the reasoning here again?
If we just simple don''t pass the disabled node in DT to the dom0,
anything
inpropriate?

Baozi.

On 10/10/2013 09:05 AM, Chen Baozi wrote:
> On Wed, Oct 09, 2013 at 09:11:16AM +0100, Ian Campbell wrote:
>> On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
>>> On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall wrote:
>>>> On 10/07/2013 10:58 AM, Chen Baozi wrote:
>>>>> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>>>>>
>>>>>> It is certainly a bug in the kernel if it is accessing
something which
>>>>>> is disabled. It may also independently be a bug in the
dts that this
>>>>>> devices is disabled.
>>>>>>
>>>>>> However in v3.12-rc4 I don''t see mmc@480d1000
being disabled in
>>>>>> omap5-uevm.dts and I can''t see anything in the
history of that file
>>>>>> either. Where did your copy come from?
>>>>>
>>>>> I''m currently working on the
"omap5-v3.11-rc3" branch from
>>>>> git://github.com/rogerq/linux.git, which contains a few
necessary
>>>>> platform patches not upstreamed. In omap5-uevm.dts, there
are lines like:
>>>>>
>>>>> 253 &mmc4 {
>>>>> 254     status = "disabled";
>>>>> 255 };
>>>>> 256
>>>>> 257 &mmc5 {
>>>>> 258     status = "disabled";
>>>>> 259 };
>>>>>
>>>>> the mmc4 refers to mmc@480d1000, which defines at
omap5.dtsi:
>>>>>
>>>>> 417         mmc4: mmc@480d1000 {
>>>>>
>>>>> I checked Linus'' mainline git tree. It is the same
about disabled mmc4
>>>>> in omap5-uevm.dts. And the change is introduced in commit
5dd18b0 of the
>>>>> mainline kernel.
>>>>>
>>>>> Anyway, I''ll see what exactly happened in the dom0
kernel dealing with
>>>>> those "disabled" regions.
>>>>
>>>> I looked at the Linux code. It will populate the different
devices via
>>>> the of_platform_populate (drivers/of/platform.c).
>>>>
>>>> This function checks in of_platform_create_pdata if the device
is
>>>> available. So the mmc driver (driver/mmc/host/omap_hsmmc.c)
should not
>>>> be called for mmc4.
>>>
>>> I''ve discuessed this issue on linux-omap@vger.kernel.org.
The TI guy says
>>> that "DT disabled" means that device won''t be
created but hwmod bus
>>> initially would try to initialize all supported modules by doing
reset
>>> access in their io memory address regions. That''s why dom0
have accessed
>>> mmc4 address even though it has been disabled.
>>
>> ePAPR lists the option of "status = fail" which is
"Indicates that the
>> device is not operational. A serious error was detected in the device,
>> and it is unlikely to become operational without repair."
I''m not sure
>> that is quite right though.
>>
>> Rather than whitelisting and mapping disabled devices through perhaps
we
>> should implement them as read 0xf (or 0x0) and write ignore?
>>
>> Or maybe we should just be mapping non-blacklisted disabled devices
>> through to dom0, for it to use or ignore as it pleases. Julien, what
was
>> the reasoning here again?
> 
> If we just simple don''t pass the disabled node in DT to the dom0,
anything
> inpropriate?
Few mails ago, you said there is hardcoded access in Linux for OMAP,
right? I don''t think removing the node from the Device Tree will change
something with this problem.

-- 
Julien Grall

On Oct 10, 2013, at 7:07 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 10/10/2013 09:05 AM, Chen Baozi wrote:
>> On Wed, Oct 09, 2013 at 09:11:16AM +0100, Ian Campbell wrote:
>>> On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
>>>> On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall wrote:
>>>>> On 10/07/2013 10:58 AM, Chen Baozi wrote:
>>>>>> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>>>>>> 
>>>>>>> It is certainly a bug in the kernel if it is
accessing something which
>>>>>>> is disabled. It may also independently be a bug in
the dts that this
>>>>>>> devices is disabled.
>>>>>>> 
>>>>>>> However in v3.12-rc4 I don''t see
mmc@480d1000 being disabled in
>>>>>>> omap5-uevm.dts and I can''t see anything in
the history of that file
>>>>>>> either. Where did your copy come from?
>>>>>> 
>>>>>> I''m currently working on the
"omap5-v3.11-rc3" branch from
>>>>>> git://github.com/rogerq/linux.git, which contains a few
necessary
>>>>>> platform patches not upstreamed. In omap5-uevm.dts,
there are lines like:
>>>>>> 
>>>>>> 253 &mmc4 {
>>>>>> 254     status = "disabled";
>>>>>> 255 };
>>>>>> 256
>>>>>> 257 &mmc5 {
>>>>>> 258     status = "disabled";
>>>>>> 259 };
>>>>>> 
>>>>>> the mmc4 refers to mmc@480d1000, which defines at
omap5.dtsi:
>>>>>> 
>>>>>> 417         mmc4: mmc@480d1000 {
>>>>>> 
>>>>>> I checked Linus'' mainline git tree. It is the
same about disabled mmc4
>>>>>> in omap5-uevm.dts. And the change is introduced in
commit 5dd18b0 of the
>>>>>> mainline kernel.
>>>>>> 
>>>>>> Anyway, I''ll see what exactly happened in the
dom0 kernel dealing with
>>>>>> those "disabled" regions.
>>>>> 
>>>>> I looked at the Linux code. It will populate the different
devices via
>>>>> the of_platform_populate (drivers/of/platform.c).
>>>>> 
>>>>> This function checks in of_platform_create_pdata if the
device is
>>>>> available. So the mmc driver (driver/mmc/host/omap_hsmmc.c)
should not
>>>>> be called for mmc4.
>>>> 
>>>> I''ve discuessed this issue on
linux-omap@vger.kernel.org. The TI guy says
>>>> that "DT disabled" means that device won''t
be created but hwmod bus
>>>> initially would try to initialize all supported modules by
doing reset
>>>> access in their io memory address regions. That''s why
dom0 have accessed
>>>> mmc4 address even though it has been disabled.
>>> 
>>> ePAPR lists the option of "status = fail" which is
"Indicates that the
>>> device is not operational. A serious error was detected in the
device,
>>> and it is unlikely to become operational without repair."
I''m not sure
>>> that is quite right though.
>>> 
>>> Rather than whitelisting and mapping disabled devices through
perhaps we
>>> should implement them as read 0xf (or 0x0) and write ignore?
>>> 
>>> Or maybe we should just be mapping non-blacklisted disabled devices
>>> through to dom0, for it to use or ignore as it pleases. Julien,
what was
>>> the reasoning here again?
>> 
>> If we just simple don''t pass the disabled node in DT to the
dom0, anything
>> inpropriate?
> 
> Few mails ago, you said there is hardcoded access in Linux for OMAP,
> right? I don''t think removing the node from the Device Tree will
change
> something with this problem.
Oh, I just had a assumption about hardcoded stuffs then. However, after
discussed
with TI guys, it turns out I''ve just made a wrong assumption. There is
no
hard-coding rather a different requirement why omap_hwmod accesses the
module''s
io memory space, according to their explanation. Sorry for not making it clear
when I got the new explanation.

Thanks.

Baozi

Hi,

On Thu, Oct 10, 2013 at 7:36 AM, Chen Baozi <baozich@gmail.com>
wrote:
>
> On Oct 10, 2013, at 7:07 PM, Julien Grall <julien.grall@linaro.org>
wrote:
>
>> On 10/10/2013 09:05 AM, Chen Baozi wrote:
>>> On Wed, Oct 09, 2013 at 09:11:16AM +0100, Ian Campbell wrote:
>>>> On Wed, 2013-10-09 at 15:46 +0800, Chen Baozi wrote:
>>>>> On Mon, Oct 07, 2013 at 11:59:16AM +0100, Julien Grall
wrote:
>>>>>> On 10/07/2013 10:58 AM, Chen Baozi wrote:
>>>>>>> On 10/07/2013 04:39 PM, Ian Campbell wrote:
>>>>>>>>
>>>>>>>> It is certainly a bug in the kernel if it is
accessing something which
>>>>>>>> is disabled. It may also independently be a bug
in the dts that this
>>>>>>>> devices is disabled.
>>>>>>>>
>>>>>>>> However in v3.12-rc4 I don''t see
mmc@480d1000 being disabled in
>>>>>>>> omap5-uevm.dts and I can''t see
anything in the history of that file
>>>>>>>> either. Where did your copy come from?
>>>>>>>
>>>>>>> I''m currently working on the
"omap5-v3.11-rc3" branch from
>>>>>>> git://github.com/rogerq/linux.git, which contains a
few necessary
>>>>>>> platform patches not upstreamed. In omap5-uevm.dts,
there are lines like:
>>>>>>>
>>>>>>> 253 &mmc4 {
>>>>>>> 254     status = "disabled";
>>>>>>> 255 };
>>>>>>> 256
>>>>>>> 257 &mmc5 {
>>>>>>> 258     status = "disabled";
>>>>>>> 259 };
>>>>>>>
>>>>>>> the mmc4 refers to mmc@480d1000, which defines at
omap5.dtsi:
>>>>>>>
>>>>>>> 417         mmc4: mmc@480d1000 {
>>>>>>>
>>>>>>> I checked Linus'' mainline git tree. It is
the same about disabled mmc4
>>>>>>> in omap5-uevm.dts. And the change is introduced in
commit 5dd18b0 of the
>>>>>>> mainline kernel.
>>>>>>>
>>>>>>> Anyway, I''ll see what exactly happened in
the dom0 kernel dealing with
>>>>>>> those "disabled" regions.
>>>>>>
>>>>>> I looked at the Linux code. It will populate the
different devices via
>>>>>> the of_platform_populate (drivers/of/platform.c).
>>>>>>
>>>>>> This function checks in of_platform_create_pdata if the
device is
>>>>>> available. So the mmc driver
(driver/mmc/host/omap_hsmmc.c) should not
>>>>>> be called for mmc4.
>>>>>
>>>>> I''ve discuessed this issue on
linux-omap@vger.kernel.org. The TI guy says
>>>>> that "DT disabled" means that device
won''t be created but hwmod bus
>>>>> initially would try to initialize all supported modules by
doing reset
>>>>> access in their io memory address regions. That''s
why dom0 have accessed
>>>>> mmc4 address even though it has been disabled.
>>>>
>>>> ePAPR lists the option of "status = fail" which is
"Indicates that the
>>>> device is not operational. A serious error was detected in the
device,
>>>> and it is unlikely to become operational without repair."
I''m not sure
>>>> that is quite right though.
>>>>
>>>> Rather than whitelisting and mapping disabled devices through
perhaps we
>>>> should implement them as read 0xf (or 0x0) and write ignore?
>>>>
>>>> Or maybe we should just be mapping non-blacklisted disabled
devices
>>>> through to dom0, for it to use or ignore as it pleases. Julien,
what was
>>>> the reasoning here again?
>>>
>>> If we just simple don''t pass the disabled node in DT to
the dom0, anything
>>> inpropriate?
>>
>> Few mails ago, you said there is hardcoded access in Linux for OMAP,
>> right? I don''t think removing the node from the Device Tree
will change
>> something with this problem.
>
> Oh, I just had a assumption about hardcoded stuffs then. However, after
discussed
> with TI guys, it turns out I''ve just made a wrong assumption.
There is no
> hard-coding rather a different requirement why omap_hwmod accesses the
module''s
> io memory space, according to their explanation. Sorry for not making it
clear
> when I got the new explanation.
>
I think most of the hardcoded access has been removed. Kernel would just try to
access the modules that are listed in DT and then attemtp to do a
soft-reset to get
those modules to a known state.

I am still trying to wrap my head around how Xen works but from what i
understood
so far the soft reset in both Dom0 and DomU is the problem here?

On Oct 9, 2013, at 7:15 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 10/09/2013 09:11 AM, Ian Campbell wrote:
>> Rather than whitelisting and mapping disabled devices through perhaps
we
>> should implement them as read 0xf (or 0x0) and write ignore?
> 
> We can''t assume that these values (0x0 or 0xf) won''t
affect dom0 behaviour.
> 
> What about a new quirk to map all disabled device in dom0
> (PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE)?
I''ve implemented the PLATFORM_QUIRK_DOM0_MAP_DISABLED_DEVICE quirk in
my
own working tree. It seems to work very well on fixing this issue. +1 
to this idea.

Then I finished new smp initialization callbacks for omap5.

However, the new xen version still has some issues to boot dom0 on OMAP5.

Right now, Dumping cpu0 guest state shows that PC stays at __loop_delay,
while cpu1 stays at cpu_relax() line of ipi_cpu_stop(), using addr2line
to translate PC to the C code location.

Any ideas?

Cheers,

Baozi