Syslinux - Sep 2014 - UEFI PXE / split config / TFTP attempted to DHCP server, not TFTP server

Gene,

All good questions.  Thanks for taking time to respond. You've given me
great leads to follow up w/ DHCP appliance consultant.
>filename being file.  I presume it looks like
"/path/to/syslinux.efi"
>or "/path/to/bootx64.efi" and not a TFTP URL?
Correct.
>>     TFTP Server Name (66) -- a string.  I have tried:
>>          - TFTP Server's IP address (as string),
>
>As a dotted decimal is considered "typical", ie
"192.0.2.5"
Correct.  That's what's working at my house.  I tried that at work. 
Dotted
decimal.  Not working.  Also tried short hostname, FQDN.  No love.
>>     bootfile-name (67) -- set UEFI NBP's full path
>
>Same as filename?
Yes.
>Have you tried skipping options 66/67 or setting siaddr/file to 0 as
>additional data points to report to Dell?
I have tried the first.  siaddr/file only.

I have not tried the second.  Let me try.  OK, just tried.  Same.  Good
suggestion however.
>> (BTW, I'm using the embedded Broadcom NIC that comes w/ the R710). 
I
have
>> flashed the BIOS and NIC firmware up to latest from vendor.
>
>Vendor being Dell, I hope?  Are you on the latest R710 firmwares as
>can be obtained with the USC (a special onboard UEFI boot to load
>firmware and perform basic configs)?  Have you checked if any newer
>firmwares might be posted on Dell's support site?
Yes, correct. (Spot-on terminology.)  I used the USC to inventory all
the various firmware on system.  Then I went to dell.com to see the
latest version.  I was back-rev on a couple, so I used USC to flash to
latest from Dell's site for R710.
>> Has anyone else ever seen this?  Does anyone else do a split config?
>
>Can't say I've seen this behavior but a split config is pretty
common
>but recall there's two styles of split: 1) all offers/acks from 1
>server
That's what I'm doing for testing in lab. VLAN's IP helper lists one
and
only one IP address.  So all offers/acks from 1 DHCP server.
> 2) addressing from server-a and siaddr/file from server-b.
>The latter might show different results (and would also more strongly
>affirm a bug).
I'm not clear I understand this.  I'm googling.  Is it like this?

http://blogs.technet.com/b/dominikheinz/archive/2011/03/18/dhcp-amp-pxe-basics.aspx

Where instead of  D-O-R-A, it does D-O-R-A-R-A?

If I'm understanding you correctly -- you may have nailed it!
I'm handling the first style of split only, this R710 UEFI firmware may
want this second style of split only.

Will follow-up on this w/ DHCP appliance consultant.
>Have you attempted a packet capture on the DHCP server so as to assert
>that your configuration is being interpreted in the manner you expect?
Yes.  I don't manage the DHCP appliances at work.  (Although I have an admin
account on this test DHCP appliance).  Since it's a vendor appliance
(that generates ISC code), I'm unable to run wireshark directly on DHCP
appliance.

But I have my PXE boot client's switch port port-mirrored to another port.
I have a 3rd server, which is my wireshark server.  It has a public NIC
and the port-mirrored port is attached to its second NIC.

So I'm capturing absolutely all traffic to/from client.  By having wireshark
on wireshark server listen on 2nd NIC.

I've got the full wireshark captures, that's how I know it's doing
the
initial
TFTP read request to the DHCP server's IP address, instead of the TFTP
server's
IP address.

Someone else responded that 3 DHCP pools is overkill for this.  I agree, but
this DHCP appliance (BlueCat Adonis) uses a GUI that generates the ISC code
in the back-end.

The vendor (BlueCat) recommends not mucking with the back-end ISC code, but
rather to use the front-end GUI.  By defining 3 DHCP Match classes, I'm able
to implement this with 3 DHCP pools.  Inherit most of the DHCP options from
super-scope, update only specific DHCP options unique to each of my 3 pools.

At my house, I'm using raw ISC code with a Linux dhcp3-server. At my house,
it works.  Simple D-O-R-A, then TFTP to TFTP server.  Aka 1st split.

1. I need to look at the vendor's raw ISC code for my UEFI pool to verify
its
GUI is writing correct ISC code.  I'll do that tomorrow, with the
consultant.

2. Outside of that, I'm looking at cabling a newer test server to that
specific
switch port that's port mirrrored.

3. Will definitely talk to vendor about 2nd style of split. That may be it!

Spike
>Date: Sun, 21 Sep 2014 13:54:38 -0400
>From: Gene Cumm <gene.cumm at gmail.com>
>To: Spike White <spikewhitetx at gmail.com>
>Cc: For discussion of Syslinux and tftp-hpa <syslinux at zytor.com>
>Subject: Re: [syslinux] UEFI PXE / split config / TFTP attempted to
>        DHCP server, not TFTP server
>Message-ID:
>        <CAD0Rxekir+chmCSsGEqSvDvb=bp=39zFP1_b9DdJ3hky7auROA at
mail.gmail.com
>
>Content-Type: text/plain; charset=UTF-8
>
>On Sun, Sep 21, 2014 at 11:44 AM, Spike White <spikewhitetx at
gmail.com>
wrote:
>> All,
>>
>> I realize this is not strictly a PXELINUX question.  So I hope
you'll
>> indulge me; hopefully some of these PXELINUX experts have seen this
before.
>>
>> And can tell me what I'm doing wrong.  Or confirm my suspicions.
>>
>> I have a test lab server at work. Split config.  The network team
manages
>> the DHCP servers, points to our TFTP server.
>>
>> Test subnet has 3 DHCP pools.  BIOS PXE, UEFI PXE and catch-all
(straight
>> DHCP).
>>
>> DHCP server is filtering on client's initial VCI (option 60).
>>    PXEClient:Arch:00000* => BIOS PXE boot client
>>    PXEClient:Arch:00007* => UEFI PXE boot client
>>    empty => std DHCP request
>>
>> This is all working correctly.  Based on the OFFERed DHCP addresses
(&
>> wireshark analysis), I can tell it's being associated with the
correct
DHCP
>> pool in all cases.
>>
>> My PXE BIOS boot and std DHCP requests are working fine.
>>
>> My problem is with UEFI boot.  It goes through the "DORA"
conversation
with
>> the DHCP server fine.  (DISCOVER-OFFER-REQUEST-ACK).
>>
>> Then it attempts to TFTP from the DHCP server instead of the TFTP
server!!
>> (Which of course fails.)  This only occurs with UEFI; under BIOS boot
it
>> properly requests from the TFTP server.
>>
>> My test server is a Dell R710 server, which is a 3-4 year old server. 
It
>> does
>> BIOS boot by default, but will do UEFI boot.
>>
>> Here are the relevant DHCP options I have set in my UEFI pool:
>>     next-server (BOOTP flag) -- set to TFTP server's IP address
>
>next-server (ISC DHCP config) being siaddr (RFC 2131)
>
>>     filename (BOOTP flag) -- set to UEFI NBP's full path
>
>filename being file.  I presume it looks like
"/path/to/syslinux.efi"
>or "/path/to/bootx64.efi" and not a TFTP URL?
>
>>     Domain name (15) -- set to DNS domain name
>>     TFTP Server Name (66) -- a string.  I have tried:
>>          - TFTP Server's IP address (as string),
>
>As a dotted decimal is considered "typical", ie
"192.0.2.5"
>
>>          - FQDN,
>>          - short name
>>
>>     bootfile-name (67) -- set UEFI NBP's full path
>
>Same as filename?
>
>Have you tried skipping options 66/67 or setting siaddr/file to 0 as
>additional data points to report to Dell?
>
>> The OFFER & ACK packets additionally have this relevant option set:
>>     DHCP Server Identifier (54) -- DHCP server's IP address
>
>Typical.  Your request should also have this set if I recall correctly.
>
>> I do this same split config at my house (VMWare workstation 10.0
Hardware
>> Compatibility) and it works fine.  That is, it does the initial DORA
>> conversation, then correctly does the TFTP read request of the UEFI NBP
from
>> the TFTP server.
>>
>> So I strongly suspect this is a bug in the Dell R710 UEFI boot firmware
>> code.
>
>Agreed.
>
>> (BTW, I'm using the embedded Broadcom NIC that comes w/ the R710). 
I
have
>> flashed the BIOS and NIC firmware up to latest from vendor.
>
>Vendor being Dell, I hope?  Are you on the latest R710 firmwares as
>can be obtained with the USC (a special onboard UEFI boot to load
>firmware and perform basic configs)?  Have you checked if any newer
>firmwares might be posted on Dell's support site?
>
>> Has anyone else ever seen this?  Does anyone else do a split config?
>
>Can't say I've seen this behavior but a split config is pretty
common
>but recall there's two styles of split: 1) all offers/acks from 1
>server  2) addressing from server-a and siaddr/file from server-b.
>The latter might show different results (and would also more strongly
>affirm a bug).
>
>Have you attempted a packet capture on the DHCP server so as to assert
>that your configuration is being interpreted in the manner you expect?
>
>--
>-Gene

On Sun, Sep 21, 2014 at 4:50 PM, Spike White <spikewhitetx at gmail.com>
wrote:
> Gene,
>
> All good questions.  Thanks for taking time to respond. You've given me
> great leads to follow up w/ DHCP appliance consultant.
>>Have you tried skipping options 66/67 or setting siaddr/file to 0 as
>>additional data points to report to Dell?
>
> I have tried the first.  siaddr/file only.
>
> I have not tried the second.  Let me try.  OK, just tried.  Same.  Good
> suggestion however.
At least it's another datapoint for Dell.
>>> (BTW, I'm using the embedded Broadcom NIC that comes w/ the
R710).  I
> have
>>> flashed the BIOS and NIC firmware up to latest from vendor.
>>
>>Vendor being Dell, I hope?  Are you on the latest R710 firmwares as
>>can be obtained with the USC (a special onboard UEFI boot to load
>>firmware and perform basic configs)?  Have you checked if any newer
>>firmwares might be posted on Dell's support site?
>
> Yes, correct. (Spot-on terminology.)  I used the USC to inventory all
> the various firmware on system.  Then I went to dell.com to see the
> latest version.  I was back-rev on a couple, so I used USC to flash to
> latest from Dell's site for R710.
>
>>> Has anyone else ever seen this?  Does anyone else do a split
config?
>>
>>Can't say I've seen this behavior but a split config is pretty
common
>>but recall there's two styles of split: 1) all offers/acks from 1
>>server
>
> That's what I'm doing for testing in lab. VLAN's IP helper
lists one and
> only one IP address.  So all offers/acks from 1 DHCP server.
You may have to consider pushing to a second ip-helper, one for
addressing, one for boot services.
>> 2) addressing from server-a and siaddr/file from server-b.
>>The latter might show different results (and would also more strongly
>>affirm a bug).
>
> I'm not clear I understand this.  I'm googling.  Is it like this?
>
>
http://blogs.technet.com/b/dominikheinz/archive/2011/03/18/dhcp-amp-pxe-basics.aspx
>
> Where instead of  D-O-R-A, it does D-O-R-A-R-A?
It's more D-OO-R-A-R-A since both servers return offers and some
clients may do both Requests near-simultaneously.  A common scenario
is a DHCP server and separate PXE-style boot server like WDS or SCCM.
> If I'm understanding you correctly -- you may have nailed it!
> I'm handling the first style of split only, this R710 UEFI firmware may
> want this second style of split only.
>
> Will follow-up on this w/ DHCP appliance consultant.
>
>>Have you attempted a packet capture on the DHCP server so as to assert
>>that your configuration is being interpreted in the manner you expect?
>
> Yes.  I don't manage the DHCP appliances at work.  (Although I have an
admin
> account on this test DHCP appliance).  Since it's a vendor appliance
> (that generates ISC code), I'm unable to run wireshark directly on DHCP
> appliance.
>
> But I have my PXE boot client's switch port port-mirrored to another
port.
> I have a 3rd server, which is my wireshark server.  It has a public NIC
> and the port-mirrored port is attached to its second NIC.
Port-Mirrors are often the easiest comprehensive way to get a packet
capture of a client (with an inline tap being best).  A capture from
the DHCP server is normally the easiest to get going (and would have
been enough in this situation).
> So I'm capturing absolutely all traffic to/from client.  By having
wireshark
> on wireshark server listen on 2nd NIC.
>
> I've got the full wireshark captures, that's how I know it's
doing the
> initial
> TFTP read request to the DHCP server's IP address, instead of the TFTP
> server's
> IP address.
But did you verify the contents of the DHCP Offer/Acknowledge packets?
> Someone else responded that 3 DHCP pools is overkill for this.  I agree,
but
> this DHCP appliance (BlueCat Adonis) uses a GUI that generates the ISC code
> in the back-end.
>
> The vendor (BlueCat) recommends not mucking with the back-end ISC code, but
> rather to use the front-end GUI.  By defining 3 DHCP Match classes, I'm
able
> to implement this with 3 DHCP pools.  Inherit most of the DHCP options from
> super-scope, update only specific DHCP options unique to each of my 3
pools.
>
> At my house, I'm using raw ISC code with a Linux dhcp3-server. At my
house,
> it works.  Simple D-O-R-A, then TFTP to TFTP server.  Aka 1st split.
But the two are very different test environments.  Any chance of
testing a VMware-based VM at VMHWv10 at work?
> 1. I need to look at the vendor's raw ISC code for my UEFI pool to
verify
> its
> GUI is writing correct ISC code.  I'll do that tomorrow, with the
> consultant.
Check the packets.
> 2. Outside of that, I'm looking at cabling a newer test server to that
> specific
> switch port that's port mirrrored.
>
> 3. Will definitely talk to vendor about 2nd style of split. That may be it!
Check the packets.

-- 
-Gene