Greg KH
2025-Sep-24 10:52 UTC
[PATCH v4 1/6] nova-core: bitfield: Move bitfield-specific code from register! into new macro
On Sun, Sep 21, 2025 at 03:47:55PM +0200, Danilo Krummrich wrote:> On Sun Sep 21, 2025 at 2:45 PM CEST, Greg KH wrote: > > Again, regmap handles this all just fine, why not just make bindings to > > that api here instead? > > The idea is to use this for the register!() macro, e.g. > > register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" { > 28:24 architecture_0 as u8, "Lower bits of the architecture"; > 23:20 implementation as u8, "Implementation version of the architecture"; > 8:8 architecture_1 as u8, "MSB of the architecture"; > 7:4 major_revision as u8, "Major revision of the chip"; > 3:0 minor_revision as u8, "Minor revision of the chip"; > }); > > (More examples in [1].)Wonderful, but I fail to see where the endian-ness of this is set anywhere. Am I just missing that? The regmap api enforces this idea, and so the> > This generates a structure with the relevant accessors; we can also implement > additional logic, such as: > > impl NV_PMC_BOOT_0 { > /// Combines `architecture_0` and `architecture_1` to obtain the architecture of the chip. > pub(crate) fn architecture(self) -> Result<Architecture> { > Architecture::try_from( > self.architecture_0() | (self.architecture_1() << Self::ARCHITECTURE_0_RANGE.len()), > ) > } > > /// Combines `architecture` and `implementation` to obtain a code unique to the chipset. > pub(crate) fn chipset(self) -> Result<Chipset> { > self.architecture() > .map(|arch| { > ((arch as u32) << Self::IMPLEMENTATION_RANGE.len()) > | u32::from(self.implementation()) > }) > .and_then(Chipset::try_from) > } > } > > This conviniently allows us to read the register with > > let boot0 = regs::NV_PMC_BOOT_0::read(bar); > > and obtain an instance of the entire Chipset structure with > > let chipset = boot0.chipset()?; > > or pass it to a constructor that creates a Revision instance > > let rev = Revision::from_boot0(boot0); > > Analogously it allows us to modify and write registers without having to mess > with error prone shifts, masks and casts, because that code is generated by the > register!() macro. (Of course, unless we have more complicated cases where > multiple fields have to be combined as illustrated above.) > > Note that bar is of type pci::Bar<BAR0_SIZE> where BAR0_SIZE in our case is > SZ_16M. > > However, the type required by read() as generated by the register!() macro > actually only requires something that implements an I/O backend, i.e > kernel::io::Io<SIZE>. > > pci::Bar is a specific implementation of kernel::io::Io. > > With this we can let the actual I/O backend handle the endianness of the bus.Ok, great, but right now it's not doing that from what I am seeing when reading the code. Shouldn't IoMem::new() take that as an argument? But, that feels odd as our current iomem api in C doesn't care about endian issues at all because it "assumes" that the caller has already handle this properly and all that the caller "wants" is to write/read to some memory chunk and not twiddle bits.> (Actually, we could even implement an I/O backend that uses regmap.)That would probably be best to do eventually as most platform drivers use regmap today as it's the sanest api we have at the moment.> So, I think the register!() stuff is rather orthogonal.I think it's very relevant as people seem to just be "assuming" that all the world (hardware and cpus) are little-endian, while in reality, they are anything but. As proof, the code that uses this register!() logic today totally ignores endian issues and just assumes that it is both running on a little-endian system, AND the hardware is little-endian. As a crazy example, look at the USB host controllers that at runtime, have to be queried to determine what endian they are running on and the kernel drivers have to handle this "on the fly". Yes, one can argue that the hardware developers who came up with that should be forced to write the drivers as penance for such sins, but in the end, it's us that has to deal with it... So ignoring it will get us quite a ways forward with controlling sane hardware on sane systems, but when s390 finally realizes they can be writing their drivers in rust, we are going to have to have these conversations again :) thanks, greg k-h
Danilo Krummrich
2025-Sep-24 11:28 UTC
[PATCH v4 1/6] nova-core: bitfield: Move bitfield-specific code from register! into new macro
On Wed Sep 24, 2025 at 12:52 PM CEST, Greg KH wrote:> Ok, great, but right now it's not doing that from what I am seeing when > reading the code. Shouldn't IoMem::new() take that as an argument?That's correct, neither IoMem nor pci::Bar do consider it yet; it's on the list of things that still need to be done.> But, that feels odd as our current iomem api in C doesn't care about > endian issues at all because it "assumes" that the caller has already > handle this properly and all that the caller "wants" is to write/read to > some memory chunk and not twiddle bits.Yet it seems to be the correct place to deal with it. As mentioned below, regmap could just become part of an I/O backend implementation to do exactly that.>> (Actually, we could even implement an I/O backend that uses regmap.) > > That would probably be best to do eventually as most platform drivers > use regmap today as it's the sanest api we have at the moment.I agree it's what we should do eventually.>> So, I think the register!() stuff is rather orthogonal. > > I think it's very relevant as people seem to just be "assuming" that all > the world (hardware and cpus) are little-endian, while in reality, they > are anything but. As proof, the code that uses this register!() logic > today totally ignores endian issues and just assumes that it is both > running on a little-endian system, AND the hardware is little-endian. > > As a crazy example, look at the USB host controllers that at runtime, > have to be queried to determine what endian they are running on and the > kernel drivers have to handle this "on the fly". Yes, one can argue > that the hardware developers who came up with that should be forced to > write the drivers as penance for such sins, but in the end, it's us that > has to deal with it... > > So ignoring it will get us quite a ways forward with controlling sane > hardware on sane systems, but when s390 finally realizes they can be > writing their drivers in rust, we are going to have to have these > conversations again :)I think it's not really that anyone is ignoring it (intentionally). It's two different things that should be addressed here; yet they are related: (1) Implementation of an abstract representation of a register that drivers can interact with. (2) The I/O layer that lays out the raw data on the physcial bus. The register!() macro intends to provide an abstract representation of a register for drivers to interact with. Think of it as an abstract box, where the memory layout does not matter at all -- could be anything. Theoretically, this abstraction could even store every single field of a register in its own u32 or u64, etc. Of course, that's a waste of memory, which is why we're using this bitfield thing instead. The only thing that matters is that there is a contract between the struct representing a register (generated by the register!() macro) and the I/O backend layer that lays out the raw value on the bus. This works attempts to address (1), whereas you are (rightfully) asking for (2). And I think the answer for (2) simply is, we still have to address it.
Yury Norov
2025-Sep-24 14:38 UTC
[PATCH v4 1/6] nova-core: bitfield: Move bitfield-specific code from register! into new macro
On Wed, Sep 24, 2025 at 12:52:41PM +0200, Greg KH wrote:> On Sun, Sep 21, 2025 at 03:47:55PM +0200, Danilo Krummrich wrote: > > On Sun Sep 21, 2025 at 2:45 PM CEST, Greg KH wrote: > > > Again, regmap handles this all just fine, why not just make bindings to > > > that api here instead? > > > > The idea is to use this for the register!() macro, e.g. > > > > register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" { > > 28:24 architecture_0 as u8, "Lower bits of the architecture"; > > 23:20 implementation as u8, "Implementation version of the architecture"; > > 8:8 architecture_1 as u8, "MSB of the architecture"; > > 7:4 major_revision as u8, "Major revision of the chip"; > > 3:0 minor_revision as u8, "Minor revision of the chip"; > > }); > > > > (More examples in [1].) > > Wonderful, but I fail to see where the endian-ness of this is set > anywhere. Am I just missing that? The regmap api enforces this idea, > and so the > > > > > This generates a structure with the relevant accessors; we can also implement > > additional logic, such as: > > > > impl NV_PMC_BOOT_0 { > > /// Combines `architecture_0` and `architecture_1` to obtain the architecture of the chip. > > pub(crate) fn architecture(self) -> Result<Architecture> { > > Architecture::try_from( > > self.architecture_0() | (self.architecture_1() << Self::ARCHITECTURE_0_RANGE.len()), > > ) > > } > > > > /// Combines `architecture` and `implementation` to obtain a code unique to the chipset. > > pub(crate) fn chipset(self) -> Result<Chipset> { > > self.architecture() > > .map(|arch| { > > ((arch as u32) << Self::IMPLEMENTATION_RANGE.len()) > > | u32::from(self.implementation()) > > }) > > .and_then(Chipset::try_from) > > } > > } > > > > This conviniently allows us to read the register with > > > > let boot0 = regs::NV_PMC_BOOT_0::read(bar); > > > > and obtain an instance of the entire Chipset structure with > > > > let chipset = boot0.chipset()?; > > > > or pass it to a constructor that creates a Revision instance > > > > let rev = Revision::from_boot0(boot0); > > > > Analogously it allows us to modify and write registers without having to mess > > with error prone shifts, masks and casts, because that code is generated by the > > register!() macro. (Of course, unless we have more complicated cases where > > multiple fields have to be combined as illustrated above.) > > > > Note that bar is of type pci::Bar<BAR0_SIZE> where BAR0_SIZE in our case is > > SZ_16M. > > > > However, the type required by read() as generated by the register!() macro > > actually only requires something that implements an I/O backend, i.e > > kernel::io::Io<SIZE>. > > > > pci::Bar is a specific implementation of kernel::io::Io. > > > > With this we can let the actual I/O backend handle the endianness of the bus. > > Ok, great, but right now it's not doing that from what I am seeing when > reading the code. Shouldn't IoMem::new() take that as an argument? > > But, that feels odd as our current iomem api in C doesn't care about > endian issues at all because it "assumes" that the caller has already > handle this properly and all that the caller "wants" is to write/read to > some memory chunk and not twiddle bits. > > > (Actually, we could even implement an I/O backend that uses regmap.) > > That would probably be best to do eventually as most platform drivers > use regmap today as it's the sanest api we have at the moment. > > > So, I think the register!() stuff is rather orthogonal. > > I think it's very relevant as people seem to just be "assuming" that all > the world (hardware and cpus) are little-endian, while in reality, they > are anything but. As proof, the code that uses this register!() logic > today totally ignores endian issues and just assumes that it is both > running on a little-endian system, AND the hardware is little-endian. > > As a crazy example, look at the USB host controllers that at runtime, > have to be queried to determine what endian they are running on and the > kernel drivers have to handle this "on the fly". Yes, one can argue > that the hardware developers who came up with that should be forced to > write the drivers as penance for such sins, but in the end, it's us that > has to deal with it... > > So ignoring it will get us quite a ways forward with controlling sane > hardware on sane systems, but when s390 finally realizes they can be > writing their drivers in rust, we are going to have to have these > conversations again :)Hi Greg, all, Endianess is not the only problem when dealing with registers mapped to the memory, right? I recall some built-in 12-bit ADCs in 8-bit AVR microcontrollers. That required to read 4-bit LO register before 8-bit HI, if you didn't want to loose those 4 bits. Bitfields don't address that issue as well. In my understanding, it's done on purpose: bitfields encapsulate shifts and masks, and don't pretend that they are suitable for direct access to a hardware. Notice another rust bitfield project. It tries to account for endianess and everything else: https://docs.rs/bitfield-struct/latest/bitfield_struct/ I didn't ask explicitly, and maybe it's a good time to ask now: Joel, Danilo and everyone, have you considered adopting this project in kernel? The bitfield_struct builds everything into the structure: use bitfield_struct::bitfield; #[bitfield(u8, order = Msb)] struct MyMsbByte { /// The first field occupies the *most* significant bits #[bits(4)] kind: usize, system: bool, #[bits(2)] level: usize, present: bool } let my_byte_msb = MyMsbByte::new() .with_kind(10) .with_system(false) .with_level(2) .with_present(true); // .- kind // | .- system // | | .- level // | | | .- present assert_eq!(my_byte_msb.0, 0b1010_0_10_1); Here MSB is not BE. For BE you'd specify: #[bitfield(u16, repr = be16, from = be16::from_ne, into = be16::to_ne)] struct MyBeBitfield { #[bits(4)] first_nibble: u8, #[bits(12)] other: u16, } The "from = be16::from_ne", is seemingly the same as cpu_to_be32() here. It looks like bitfield_struct tries to resolve hw access problems by outsourcing them to 'from' and 'to' callbacks, and that looks similar to what regmap API does (is that correct?). Greg, Is that what you're asking about? This is another bitfield crate with the similar approach https://crates.io/crates/bitfield So we're not the first, and we need to discuss what is already done. As far as I understand, Joel decided to go in the other direction: bitfields are always native in terms of endianess and not designed to be mapped on registers directly. Which means they don't specify order of accesses, number of accesses, access timing, atomicity, alignment, cacheability and whatever else I/O related. I discussed with Joel about the hw register access and he confirmed that the idea of his bitfields is to be a simple wrapper around logical ops, while the I/O is a matter of 'backbone', which is entirely different thing: reg = nova_register(addr, be64, strong_ordered, lo_first, ...) reg.read() bf = reg.bf() val = bf.field1() | MY_FLAG bf.set_field1(val) reg.set_bf() reg.write() In this design, .read() and .write() are the only accessors to the mapped registers memory, they do endianess conversion if needed, and everything else. I'm not an expert in regmaps, but from what I can see, the complexity of the backbone I/O might exceed complexity of bitfields themself; and what the bitfield_struct (and the other project I've googled) does looks like dictating to potentially more complex projects about how their API should look. Because rust has no out-of-the-box bitfields, like C does, I think that we should have simple API that resembles C bitfields syntax and functionality. Joel mentioned rcu_special, and I can guess there's more examples like flags, where people just need a compact data structure with a per-bit access capability. With that, from bitops perspective I think bitfields are anyways useful addition to rust. How the rust code would address I/O problems is a more complex and seemingly not immediately related subject. Does that make sense? Thanks, Yury
Danilo Krummrich
2025-Sep-24 15:53 UTC
[PATCH v4 1/6] nova-core: bitfield: Move bitfield-specific code from register! into new macro
On Wed Sep 24, 2025 at 4:38 PM CEST, Yury Norov wrote:> I didn't ask explicitly, and maybe it's a good time to ask now: Joel, > Danilo and everyone, have you considered adopting this project in > kernel? > > The bitfield_struct builds everything into the structure: > > use bitfield_struct::bitfield; > > #[bitfield(u8, order = Msb)] > struct MyMsbByte { > /// The first field occupies the *most* significant bits > #[bits(4)] > kind: usize, > system: bool, > #[bits(2)] > level: usize, > present: bool > } > let my_byte_msb = MyMsbByte::new() > .with_kind(10) > .with_system(false) > .with_level(2) > .with_present(true); > > // .- kind > // | .- system > // | | .- level > // | | | .- present > assert_eq!(my_byte_msb.0, 0b1010_0_10_1); > > Here MSB is not BE. For BE you'd specify: > > #[bitfield(u16, repr = be16, from = be16::from_ne, into = be16::to_ne)] > struct MyBeBitfield { > #[bits(4)] > first_nibble: u8, > #[bits(12)] > other: u16, > } > > The "from = be16::from_ne", is seemingly the same as cpu_to_be32() here. > > It looks like bitfield_struct tries to resolve hw access problems > by outsourcing them to 'from' and 'to' callbacks, and that looks > similar to what regmap API does (is that correct?). > > Greg, Is that what you're asking about? > > This is another bitfield crate with the similar approach > > https://crates.io/crates/bitfield > > So we're not the first, and we need to discuss what is already done. > > As far as I understand, Joel decided to go in the other direction: > bitfields are always native in terms of endianess and not designed to > be mapped on registers directly. Which means they don't specify order > of accesses, number of accesses, access timing, atomicity, alignment, > cacheability and whatever else I/O related. > > I discussed with Joel about the hw register access and he confirmed > that the idea of his bitfields is to be a simple wrapper around logical > ops, while the I/O is a matter of 'backbone', which is entirely different > thing:When I was working on initial Rust driver support about a year ago, I also thought about how Rust drivers can deal with registers and added the TODO in [1]. This was picked up by Alex, who came up with a great implementation for the register!() macro, which Joel splitted up into separate register!() and bitfield parts in the context of moving it from a nova internal implementation into a core kernel API. As also described in [2], the idea is to have a macro, register!(), that creates an abstract representation of a register, in order to remove the need for drivers to manually construct values through shift operations, masks, etc. At the same time the idea also is to get proper documentation of the hardware registers in the kernel; the register!() macro encourages that, by its definition trying to come close to how registers are typically documented in datasheets, i.e. get rid of thousands of lines of auto-generated #defines for base addresses, shift and masks with cryptic names and provide something like register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" { 28:24 architecture_0 as u8, "Lower bits of the architecture"; 23:20 implementation as u8, "Implementation version of the architecture"; 8:8 architecture_1 as u8, "MSB of the architecture"; 7:4 major_revision as u8, "Major revision of the chip"; 3:0 minor_revision as u8, "Minor revision of the chip"; }); instead. (It has quite some more features that also allow you to directly derive complex types from primitives and define arrays of registers, such as in register!(NV_PFALCON_FBIF_TRANSCFG @ PFalconBase[0x00000600[8]] { 1:0 target as u8 ?=> FalconFbifTarget; 2:2 mem_type as bool => FalconFbifMemType; }); which makes dealing with such registers in drivers way less error prone. Here's one example of how it looks like to alter a single field within a register: // `bar` is the `pci::Bar` I/O backend. regs::NV_PFALCON_FALCON_ENGINE::alter(bar, |v| v.set_reset(true)); Of course you could also alter multiple fields at once by doing more changes within the closure.) It intentionally avoids encoding hardware bus specific endianness, because otherwise we'd need to define this for every single register definition, which also falls apart when the device can sit on top of multiple different busses. Instead, the only thing that matters is that there is a contract between the abstract register representation and the I/O backends, such that the data can be correctly layed out by the I/O backend, which has to be aware of the actual hardware bus instead. As mentioned in another thread, one option for that is to use regmap within the I/O backends, but that still needs to be addressed. So, for the register!() macro, I think we should keep it an abstract representation and deal with endianness in the I/O backend. However, that's or course orthogonal to the actual feature set of the bitfield macro itself. - Danilo [1] https://docs.kernel.org/gpu/nova/core/todo.html#generic-register-abstraction-rega [2] https://lore.kernel.org/lkml/DD0ZTZM8S84H.1YDWSY7DF14LM at kernel.org/
Joel Fernandes
2025-Sep-24 17:46 UTC
[PATCH v4 1/6] nova-core: bitfield: Move bitfield-specific code from register! into new macro
On 9/24/2025 4:38 PM, Yury Norov wrote:> On Wed, Sep 24, 2025 at 12:52:41PM +0200, Greg KH wrote: >> On Sun, Sep 21, 2025 at 03:47:55PM +0200, Danilo Krummrich wrote: >>> On Sun Sep 21, 2025 at 2:45 PM CEST, Greg KH wrote: >>>> Again, regmap handles this all just fine, why not just make bindings to >>>> that api here instead? >>> >>> The idea is to use this for the register!() macro, e.g. >>> >>> register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" { >>> 28:24 architecture_0 as u8, "Lower bits of the architecture"; >>> 23:20 implementation as u8, "Implementation version of the architecture"; >>> 8:8 architecture_1 as u8, "MSB of the architecture"; >>> 7:4 major_revision as u8, "Major revision of the chip"; >>> 3:0 minor_revision as u8, "Minor revision of the chip"; >>> }); >>> >>> (More examples in [1].) >> >> Wonderful, but I fail to see where the endian-ness of this is set >> anywhere. Am I just missing that? The regmap api enforces this idea, >> and so the >> >>> >>> This generates a structure with the relevant accessors; we can also implement >>> additional logic, such as: >>> >>> impl NV_PMC_BOOT_0 { >>> /// Combines `architecture_0` and `architecture_1` to obtain the architecture of the chip. >>> pub(crate) fn architecture(self) -> Result<Architecture> { >>> Architecture::try_from( >>> self.architecture_0() | (self.architecture_1() << Self::ARCHITECTURE_0_RANGE.len()), >>> ) >>> } >>> >>> /// Combines `architecture` and `implementation` to obtain a code unique to the chipset. >>> pub(crate) fn chipset(self) -> Result<Chipset> { >>> self.architecture() >>> .map(|arch| { >>> ((arch as u32) << Self::IMPLEMENTATION_RANGE.len()) >>> | u32::from(self.implementation()) >>> }) >>> .and_then(Chipset::try_from) >>> } >>> } >>> >>> This conviniently allows us to read the register with >>> >>> let boot0 = regs::NV_PMC_BOOT_0::read(bar); >>> >>> and obtain an instance of the entire Chipset structure with >>> >>> let chipset = boot0.chipset()?; >>> >>> or pass it to a constructor that creates a Revision instance >>> >>> let rev = Revision::from_boot0(boot0); >>> >>> Analogously it allows us to modify and write registers without having to mess >>> with error prone shifts, masks and casts, because that code is generated by the >>> register!() macro. (Of course, unless we have more complicated cases where >>> multiple fields have to be combined as illustrated above.) >>> >>> Note that bar is of type pci::Bar<BAR0_SIZE> where BAR0_SIZE in our case is >>> SZ_16M. >>> >>> However, the type required by read() as generated by the register!() macro >>> actually only requires something that implements an I/O backend, i.e >>> kernel::io::Io<SIZE>. >>> >>> pci::Bar is a specific implementation of kernel::io::Io. >>> >>> With this we can let the actual I/O backend handle the endianness of the bus. >> >> Ok, great, but right now it's not doing that from what I am seeing when >> reading the code. Shouldn't IoMem::new() take that as an argument? >> >> But, that feels odd as our current iomem api in C doesn't care about >> endian issues at all because it "assumes" that the caller has already >> handle this properly and all that the caller "wants" is to write/read to >> some memory chunk and not twiddle bits. >> >>> (Actually, we could even implement an I/O backend that uses regmap.) >> >> That would probably be best to do eventually as most platform drivers >> use regmap today as it's the sanest api we have at the moment. >> >>> So, I think the register!() stuff is rather orthogonal. >> >> I think it's very relevant as people seem to just be "assuming" that all >> the world (hardware and cpus) are little-endian, while in reality, they >> are anything but. As proof, the code that uses this register!() logic >> today totally ignores endian issues and just assumes that it is both >> running on a little-endian system, AND the hardware is little-endian. >> >> As a crazy example, look at the USB host controllers that at runtime, >> have to be queried to determine what endian they are running on and the >> kernel drivers have to handle this "on the fly". Yes, one can argue >> that the hardware developers who came up with that should be forced to >> write the drivers as penance for such sins, but in the end, it's us that >> has to deal with it... >> >> So ignoring it will get us quite a ways forward with controlling sane >> hardware on sane systems, but when s390 finally realizes they can be >> writing their drivers in rust, we are going to have to have these >> conversations again :) > > Hi Greg, all, > > Endianess is not the only problem when dealing with registers mapped > to the memory, right? > > I recall some built-in 12-bit ADCs in 8-bit AVR microcontrollers. That > required to read 4-bit LO register before 8-bit HI, if you didn't want to > loose those 4 bits. > > Bitfields don't address that issue as well. In my understanding, it's > done on purpose: bitfields encapsulate shifts and masks, and don't > pretend that they are suitable for direct access to a hardware. > > Notice another rust bitfield project. It tries to account for endianess > and everything else: > > https://docs.rs/bitfield-struct/latest/bitfield_struct/ > > I didn't ask explicitly, and maybe it's a good time to ask now: Joel, > Danilo and everyone, have you considered adopting this project in > kernel? > > The bitfield_struct builds everything into the structure: > > use bitfield_struct::bitfield; > > #[bitfield(u8, order = Msb)] > struct MyMsbByte { > /// The first field occupies the *most* significant bits > #[bits(4)] > kind: usize, > system: bool, > #[bits(2)] > level: usize, > present: bool > }Thanks for raising this. The syntax seems quite different from what we need, in particular since register! macro is based on our bitfield! macro, this syntax is incompatible with the need to specify bit ranges, not just the number of bits. In other words, it appears the out-of-crate does not satisfy the requirement. They have to specific 'order' property mainly because they don't have the notion of bitfield index, just number of bits. Regarding endianness in that crate, it appears to be configurable based on user's requirement so we can make it such if needed for any kernel usecases. But the default in that crate is native-endianness just like our implementation right? Thanks.