Syslinux - Sep 2003 - Network Boot IP Configuration Dilemma

Background:  I recently wrote to Peter Anvin regarding IPAPPEND
functionality, and he suggested I take the issue to this mailing list.
I see it has been touched upon several times, after reviewing the
archives.  Let me see if I can add to it with a good level of detail:

When using PXE to network boot a linux client, PXE must first
acquire the client's IP address, by dhcp request, then, the downloaded
linux kernel must get the IP address either from pxelinux, or by a
"duplicate" dhcp request.

If the pxelinux.cfg "IPAPPEND 1" entry is used, an "ip=..."
string is
appended to the parameters passed to the kernel.  This is, as I will
ultimately conclude below, a less problematic approach, however,
it is in need of being extended to use the currently unused "device"
parameter.

If the pxelinux.cfg "ip=::::::dhcp" entry is used, a dhcp request
will be made by the kernel.

Discussion of the dilemma:

Under ideal circumstances, it makes no difference which approach
is used.  In real world, problematic environments, it may make a very
significant difference.  The first complicating factor is if more than
one network interface is present.  When using the Tyan s2880 Opteron
board, the two on-board interfaces can be turned both on or both off.
It is impossible to use only one of them.  This discussion of
potential problems should apply to any multi-homed netbooting system:

Line 1214 of ipconfig.c (2.4.22 kernel) contains
"ic_first_dev->next,"
which implements the objective of the comment on lines 1203-1206:
   "If the config information is insufficient (e.g., ...or we have
    multiple network interfaces and no default was set), use BOOTP
    or RARP to get the missing values."
So if there is more than one device found (the linked list contains
more than a single element), the enumerated "ip=..." values are
ignored and dynamic reconfiguration is attempted, and if dynamic
prococals haven't been compiled into ipconfig, the network
configuration simply fails.  This functionality is understandable,
though I think its wisdom questionable.

I have been able to think of three situations where relevant problems
may occur.

1)  The second network interface, rather than the first, has
     PXE active, and the dhcp issued IP address intended for the
     second interface's identity is used by the first.

2)  The BIOS ordering of the interfaces is different than the
     kernel ordering, so even if the first BIOS located interface
     makes the dhcp request, the kernel sees it as the second.
     I have observed such a similar ordering problem with SCSI
     interfaces, so the same possibility probably exists for network
     interfaces.

3)  A second DHCP server answers the request differently than
     the first.  This is from my experience, the much more likely
     and frequent problem.  Where different servers, possibly
     unintentionally, are answering on the same network, one may
     often answer first for the first request, and the other may
     often answer first for subsequent requests, as the busier, but
     faster system has dhcpd swapped in.

Using the two dhcp request approach may reduce problems #1 and #2.
Using the single dhcp request approach reduces problem #3.
Problems #1 and #2 are solid configuration errors with limited
potential for trouble.  If a system is using an IP address on one
interface that is intended for another of its interfaces, I consider
this the lesser problem.  If, however, a system is intermittently
failing to receive its correct IP address at differing stages of the
boot process, it is a more difficult problem, less likely to be
reported in a way that a system administrator can correctly diagnose it.

I suggest that the syntax of IPAPPEND be changed to allow a specific
device to be named:  "IPAPPEND 1|devname"  ("devname" will
usually be
eth0).  Unless and until this is done, here is my hack to change
"ip=#:#:#:#" to "ip=#:#:#:#::eth0:" in pxelinux.asm (after
"genipopt"):

...
                 mov eax,[Netmask]
                 call gendotquad ; Zero-terminates its output
                 mov al,':'              ; HACK
                 stosb                   ; HACK
                 mov al,':'              ; HACK
                 stosb                   ; HACK
                 mov al,'e'              ; HACK
                 stosb                   ; HACK
                 mov al,'t'              ; HACK
                 stosb                   ; HACK
                 mov al,'h'              ; HACK
                 stosb                   ; HACK
                 mov al,'0'              ; HACK
                 stosb                   ; HACK
                 mov al,':'              ; HACK
                 stosb                   ; HACK
                 mov al,0                ; HACK
                 stosb                   ; HACK
                 sub di,IPOption
                 mov [IPOptionLen],di
                 popad
                 ret

--Fred

PS:  I don't want to forget to offer thanks for how really good the
syslinux/pxelinux utilities already are.  3 years ago, when etherboot
was a less vendor specific protocol, I used etherboot/mknbi much more,
but now that numerous vendor's come with PXE already in firmware, I've
almost completely switched over to PXE/pxelinux!


-- 
Fred Feirtag
Chief Technologist
Integrity Linux Systems, a division of Integrity Networking Systems, Inc.
1001 Medical Arts Avenue, NE
Albuquerque, NM  87012
email: ffeirtag at integrityns.com
phone: (505) 294 - 7747
fax: (505) 275 - 1125

Fred Feirtag wrote:
> 
> If the pxelinux.cfg "IPAPPEND 1" entry is used, an
"ip=..." string is
> appended to the parameters passed to the kernel.  This is, as I will
> ultimately conclude below, a less problematic approach, however,
> it is in need of being extended to use the currently unused
"device"
> parameter.
> 
Except, it's not.  If the "device" parameter is specified then the
kernel may very well configure a different device.  Always tagging
::eth0: at the end is highly problematic (in fact, it's downright wrong.)

There is another issue to keep in mind: unless you're running a dhcp
server in userspace, your dhcp lease will eventually time out and the
dhcp server may assign an IP address conflict.
> 3)  A second DHCP server answers the request differently than
>    the first.  This is from my experience, the much more likely
>    and frequent problem.  Where different servers, possibly
>    unintentionally, are answering on the same network, one may
>    often answer first for the first request, and the other may
>    often answer first for subsequent requests, as the busier, but
>    faster system has dhcpd swapped in.
Of all your problem descriptions, this is the only one which isn't
actually a problem.  You will either obtain a legal configuration, or
you have a much more serious problem of two unsynchronized dhcp servers
stomping on each other.

Since both your #1 and #2 are problems that would occur *with* your
change, and #3 is not a problem in a properly set up environment, I
don't see this as beneficial in any form.  I am, however, willing to
listen to arguments to the contrary.

	-hpa

Fred Feirtag wrote:
> Background:  I recently wrote to Peter Anvin regarding IPAPPEND
> functionality, and he suggested I take the issue to this mailing list.
> I see it has been touched upon several times, after reviewing the
> archives.  Let me see if I can add to it with a good level of detail:
> 
> When using PXE to network boot a linux client, PXE must first
> acquire the client's IP address, by dhcp request, then, the downloaded
> linux kernel must get the IP address either from pxelinux, or by a
> "duplicate" dhcp request.
> 
> If the pxelinux.cfg "IPAPPEND 1" entry is used, an
"ip=..." string is
> appended to the parameters passed to the kernel.  This is, as I will
> ultimately conclude below, a less problematic approach, however,
> it is in need of being extended to use the currently unused
"device"
> parameter.
> 
> If the pxelinux.cfg "ip=::::::dhcp" entry is used, a dhcp request
> will be made by the kernel.
> 
> Discussion of the dilemma:
> 
> Under ideal circumstances, it makes no difference which approach
> is used.  In real world, problematic environments, it may make a very
> significant difference.  The first complicating factor is if more than
> one network interface is present.  When using the Tyan s2880 Opteron
> board, the two on-board interfaces can be turned both on or both off.
> It is impossible to use only one of them.  This discussion of
> potential problems should apply to any multi-homed netbooting system:
> 
> Line 1214 of ipconfig.c (2.4.22 kernel) contains
"ic_first_dev->next,"
> which implements the objective of the comment on lines 1203-1206:
>   "If the config information is insufficient (e.g., ...or we have
>    multiple network interfaces and no default was set), use BOOTP
>    or RARP to get the missing values."
> So if there is more than one device found (the linked list contains
> more than a single element), the enumerated "ip=..." values are
> ignored and dynamic reconfiguration is attempted, and if dynamic
> prococals haven't been compiled into ipconfig, the network
> configuration simply fails.  This functionality is understandable,
> though I think its wisdom questionable.
> 
> I have been able to think of three situations where relevant problems
> may occur.
> 
> 1)  The second network interface, rather than the first, has
>     PXE active, and the dhcp issued IP address intended for the
>     second interface's identity is used by the first.
> 
> 2)  The BIOS ordering of the interfaces is different than the
>     kernel ordering, so even if the first BIOS located interface
>     makes the dhcp request, the kernel sees it as the second.
>     I have observed such a similar ordering problem with SCSI
>     interfaces, so the same possibility probably exists for network
>     interfaces.
> 
> 3)  A second DHCP server answers the request differently than
>     the first.  This is from my experience, the much more likely
>     and frequent problem.  Where different servers, possibly
>     unintentionally, are answering on the same network, one may
>     often answer first for the first request, and the other may
>     often answer first for subsequent requests, as the busier, but
>     faster system has dhcpd swapped in.
> 
> Using the two dhcp request approach may reduce problems #1 and #2.
> Using the single dhcp request approach reduces problem #3.
> Problems #1 and #2 are solid configuration errors with limited
> potential for trouble.  If a system is using an IP address on one
> interface that is intended for another of its interfaces, I consider
> this the lesser problem.  If, however, a system is intermittently
> failing to receive its correct IP address at differing stages of the
> boot process, it is a more difficult problem, less likely to be
> reported in a way that a system administrator can correctly diagnose it.
> 
I think my understanding of DHCP is a lot different to yours.
Maybe you should read the DHCP/BOOTP RFCs again.

Here is my understanding of the issue: -
NIC card sends it's own MAC address in the DHCP request.
The DHCP server replies with an IP address, and other info, e.g. default 
gateway, tftp server address for net booting etc.
The DHCP server will then link that IP address with that MAC address, 
until an expiry period set by the DHCP server.
If the NIC card sends out a DHCP request any time before the expiry 
period, it will reset the exiry period on the DHCP server, and the DHCP 
server guarantees to give it the same IP address again.
The NIC card can at any time send a DHCP release, whereby on the next 
DHCP request it might get any IP address in the range.
So, this guartees that the IP address the NIC card got during PXE boot, 
will be exactly the same as the IP address linux kernel dhcp will get.
It is also beneficial that the linux kernel re-does the request, because 
it might understand more of the reply items than PXE did.

This leaves (3), which will cause problems in any network, and should 
never happen, unless both servers lease tables can be perfectly 
syncronised. (3) will cause all sorts of problems when DHCP clients try 
to renew IP address before the expiry timer times out.

Cheers
James

I am having an issue along these same lines...

I am attempting to boot several systems that are each using different
NIC's for PXE i.e. eth0 on host1 eth2 on host2 etc.  The primary reason
for not always using eth0 on each machine is due to the hardware
characteristics of the various systems i.e. on host1 eth0 is a 10/100
NIC while eth0 on host2 is a gigabit NIC.  The desire is to maintain a
"management" network where all PXE/DHCP/TFTP occurs using the 10/100
interface if one exists.

Since as I mentioned I am using a diverse set of hardware - I am using
the "IPAPPEND" parameter, but I have turned off CONFIG_IP_PNP in the
kernel and manually parse /proc/cmdline for the ip= stuff after I have
booted.  This is primarily so that I can use kudzu to identify the
appropriate NIC driver to load.  I also pass an additional kernel
parameter "mgmtdevice=ethx" so that once I have booted the ramdisk I
know which interface to use.  This works, but requires me to know ahead
of time which interface will be used for PXE on each machine and also to
maintain separate pxelinux.cfg files for each machine.

It would be great if pxelinux could add the interface that was used to
PXE as the "device" argument to ip=.  Or create an additional
parameter
that would let me know which NIC was used to PXE once I am in the
ramdisk.  Perhaps there is some other way to determine the NIC that was
used to PXE?  I have no idea how difficult this would be - so please
forgive me if I am suggesting the impossible :-)

Regards,

Jon Nalley

> Message: 2
> Date: Fri, 05 Sep 2003 16:08:57 -0600
> From: Fred Feirtag <ffeirtag at integrityns.com>
> Subject: [syslinux] Network Boot IP Configuration Dilemma
> To: syslinux at zytor.com
> Message-ID: <3F590979.40504 at integrityns.com>
<snip>
> I suggest that the syntax of IPAPPEND be changed to allow a specific
> device to be named:  "IPAPPEND 1|devname"  ("devname"
will usually be
> eth0).  Unless and until this is done, here is my hack to change
> "ip=#:#:#:#" to "ip=#:#:#:#::eth0:" in pxelinux.asm
(after "genipopt"):
<snip>

Jon Nalley wrote:
> 
> It would be great if pxelinux could add the interface that was used to
> PXE as the "device" argument to ip=.  Or create an additional
parameter
> that would let me know which NIC was used to PXE once I am in the
> ramdisk.  Perhaps there is some other way to determine the NIC that was
> used to PXE?  I have no idea how difficult this would be - so please
> forgive me if I am suggesting the impossible :-)
> 
You are.  There is no way for PXELINUX to know what Linux is going to 
call the device once it starts -- it may, in fact, not have a name at 
all or depend on the ordering of devices.

PXELINUX could find out what the hardware (MAC) address are of the 
booting device, but there is no convenient way to pass it to the kernel.

	-hpa

So your only protest to the modification is the formatting of the
address?  The original change used the 'MACStr' variable as setup
by the code.  I then used copy & paste to put the string into the
MAC address format. Should we go back to simply outputting the 
'MACStr' variable in hopes of being compliant with a change you
might make? Outputting the 'MACStr' variable would allow me to remove
the majority of the changes I made.

Stan Smith

-----Original Message-----
From: H. Peter Anvin [mailto:hpa at zytor.com]
Sent: Monday, September 08, 2003 6:23 PM
To: Jon Nalley
Cc: syslinux at zytor.com; Stanley Smith
Subject: Re: [syslinux] Re: Network Boot IP Configuration Dilemma


Jon Nalley wrote:
> I mentioned the possibility of the mac=<mac address> append option to
my
> friend Stan Smith, and he had it added in no time.  So, here is a patch
> to add the mac=<mac address> to the kernel append line in case you
> decide to support it.
Sorry, but thanks for playing.  Breaks on non-Ethernets.

There is a reason I try to talk about "hardware address" and not
"Ethernet address" or "MAC address" (although I didn't
really in this
thread) -- PXELINUX is deliberately agnostic about the type of network
it is talking to.  This is the reason the hardware address is formatted
the way it is in the query, and that needs to be maintained.

	-hpa

Peter,

We tested the mod and found a problem.  The value of 2 
is never stored in the IPAppend variable due to the logic
when reading ipappend from the config file.  The problem 
is in parseconfig.inc:57 where you use the following 
instructions to set the value of bl which is ultimately
stored in the IPAppend variable.

    and bx,bx
    setnz bl

The 'setnz' causes the value of IPAppend to always be 1.

Stan Smith

-----Original Message-----
From: H. Peter Anvin [mailto:hpa at zytor.com]
Sent: Tuesday, September 09, 2003 9:47 AM
To: stanley.smith at rlx.com
Cc: jon.nalley at rlx.com; syslinux at zytor.com
Subject: Re: [syslinux] Re: Network Boot IP Configuration Dilemma


stanley.smith at rlx.com wrote:
> So your only protest to the modification is the formatting of the
> address?  The original change used the 'MACStr' variable as setup
> by the code.  I then used copy & paste to put the string into the
> MAC address format. Should we go back to simply outputting the 
> 'MACStr' variable in hopes of being compliant with a change you
> might make? Outputting the 'MACStr' variable would allow me to
remove
> the majority of the changes I made.
> 
> Stan Smith
> 
Right.  I have already integrated this into 2.07-pre2; please test this 
out :)

	-hpa

Peter,

One last mod.  Just after the .noipappend1 label,
the IPAppend value needs to be reloaded into al.
This needs to be done in case ipappend is set to
3 in the config file and the user wants both the 
'ip=' and 'BOOTIF=' parameters passed to the kernel.
During processing of the 'ip=' parameter, al is 
disturbed.

Thanks for all your help,

Stan Smith

-----Original Message-----
From: H. Peter Anvin [mailto:hpa at zytor.com]
Sent: Tuesday, September 09, 2003 2:38 PM
To: H. Peter Anvin
Cc: stanley.smith at rlx.com; jon.nalley at rlx.com; syslinux at zytor.com
Subject: Re: [syslinux] Re: Network Boot IP Configuration Dilemma


H. Peter Anvin wrote:
> stanley.smith at rlx.com wrote:
> 
>>Peter,
>>
>>We tested the mod and found a problem.  The value of 2 
>>is never stored in the IPAppend variable due to the logic
>>when reading ipappend from the config file.  The problem 
>>is in parseconfig.inc:57 where you use the following 
>>instructions to set the value of bl which is ultimately
>>stored in the IPAppend variable.
>>
>>    and bx,bx
>>    setnz bl
>>
>>The 'setnz' causes the value of IPAppend to always be 1.
>>
> 
> 
> Thanks.  I forgot about that... will fix :)
> 
Okay... fix released as 2.07-pre4.

	-hpa