[CentOS] Kickstart with multiple eth devices

Jason Warr
Wed, 25 Feb 2015 22:53:34 -0000

On Wed, 25 Feb 2015 16:45:04 -0600, Ashley M. Kirchner <ashley at
pcraft.com>
wrote:

> Out of order meaning, it puts the additional ethernet card as eth0, with
> the built-in ports as eth1 and eth2 respectively. WITHOUT the additional
> card >installed, it puts the built-in ports as eth0 and eth1, which is
> what I want it to do. The additional card should be eth2, at least
> that's what I want it to >do.

So if you really need them to be in that order your best bet may be to
blacklist the kernel module for the add in card so it does not get
enumerated during install.  If you do that and create the udev rules file
as I do then on first boot udev should enforce your rules file and then
enumerate the add in card as eth2.

Add "rdblacklist=MODULE_NAME" to your append line in pxelinux.conf
file.

This should get you into a usable state post-install so you can setup
additional interfaces.

>
> Looking at persistent-net.rules, it always puts the additional card
> first, both without your script as well as with. I need it to be last.
> The system's built->ins should always be eth0 and eth1 respectively.

It looks like the add in card is in a bus slot that is getting enumerated,
module loaded before the on-board interfaces.

>
> And dmesg confirms it as well, it identifies the added card first (and
> assigns it eth0), then identifies the built-in ports. I'd grab the
> actual output except >I need to manually reconfigure the interfaces so I
> can actually get ON the machine. Right now I'm just looking at its
> console


>
>
> On Wed, Feb 25, 2015 at 3:37 PM, Jason Warr <jason at warr.net>
wrote:
>> Define out of order in this case just so I know for sure what you mean.
>>
>> What my solution does, or at least does reliably in my case, is make
>> sure the interfaces are in the same order once installed as the install
>> kernel saw >>them.  It won't re-order them to be sequential
based on
>> bus, mac or driver.  I am working on that but it will also include
>> naming the devices based on >>the module name, similar to how
FreeBSD
>> and Solaris do it.
>>
>> Just to get an idea of what might be going on can you run "dmesg |
grep
>> eth" so I can see some of what udev is doing?
>>
>>
>>
>> On Wed, 25 Feb 2015 16:28:31 -0600, Ashley M. Kirchner
>> <ashley at pcraft.com> wrote:
>>
>>> Ok, when I run that, it creates a now "custom"
>>> 70-persistent-net.rules, but
>>> the interfaces are still out of order, with the added one listed
>>> first, and
>>> the built-ins 2nd and 3rd.
>>>
>>> On Wed, Feb 25, 2015 at 2:00 PM, Jason Warr <jason at
warr.net> wrote:
>>>
>>>> Here is my script for post install if you want to try it.
>>>>
>>>> In order for the shuffling to not occur you do need to create
the udev
>>>> rules file somehow.  I am not sure how mangled this will be in
email
>>>> but it
>>>> is worth a try.  It should run OK with nothing else.  I have a
better
>>>> version in the works but the enhancements are mainly useful for
Fedora
>>>> 19-21.
>>>>
>>>> I did forget to say I also block NetworkManager from the
interfaces.
>>>>
>>>> ############################
>>>>
>>>> #!/bin/bash
>>>> ## BIND MAC address to proper interfaces so they stay
persistent
>>>> ## I want them to stay as they were in kickstart
>>>>
>>>> ## This can cause issues with VLAN interfaces for both bond
dev's and
>>>> base
>>>> eth dev's.
>>>> ##  The bond one was solved by adding in the
"KERNEL="eth?*" as that
>>>> will
>>>> only apply to physical
>>>> ##  Devices.  Once we have VLAN's on a real device instead
of just on
>>>> BOND's this then applies
>>>> ##  to the hardware devices as well.  The core issue is that
the MAC
>>>> address is inherited
>>>> ##  by all of the children devices and thus the UDEV rule has
to be
>>>> crafted to only apply
>>>> ##  to the base physical device.
>>>> ##  This one was solved via adding DRIVERS=="?*" as
the VLAN int's
>>>> wont
>>>> have one
>>>>
>>>>    echo "[KICKSTART] Binding eth interfaces to the
expected MAC
>>>> address
>>>> in UDEV"
>>>>    echo "## Created by Kickstart to keep network
interfaces in an
>>>> expected order" > \
>>>>        /etc/udev/rules.d/70-persistent-net.rules
>>>>    echo "" >> \
>>>>        /etc/udev/rules.d/70-persistent-net.rules
>>>>
>>>>    cd /sys/class/net/
>>>>    for NETDEV in $(ls | grep eth | sort)
>>>>    do
>>>>        ## Create a UDEV rule for each eth interface
>>>>        echo "## ${NETDEV} interface" >> \
>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>
>>>>        ## We throw this one in here as it can contain some
useful
>>>> information
>>>>        echo "## $(dmesg | grep ${NETDEV} | grep -i -v -e
"console" -e
>>>> "Command line" | head -1)" >> \
>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>
>>>>        echo -n "SUBSYSTEM==\"net\",
ACTION==\"add\", DRIVERS==\"?*\",
>>>> "
>>>>
>>>>> \
>>>>>>
>>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>        echo -n "ATTR{address}==\"$(cat
${NETDEV}/address)\", " >> \
>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>        echo -n "ATTR{dev_id}==\"0x0\",
ATTR{type}==\"1\",
>>>> KERNEL==\"eth?*\", " >> \
>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>        echo -e "NAME=\"${NETDEV}\"\n"
>> \
>>>>            /etc/udev/rules.d/70-persistent-net.rules
>>>>
>>>>        ## Make a log of the devices present during install
>>>>        echo -e "${NETDEV} $(cat ${NETDEV}/address)\n"
>>
>>>> /root/ksnet-devices
>>>>
>>>>        ## Also remove the HWADDR line from all of the net
config files
>>>>        grep -v -e NAME -e HWADDR -e NM_CONTROLLED \
>>>>            /etc/sysconfig/network-scripts/ifcfg-${NETDEV} | sed
>>>> 's/\"//g' \
>>>>            >
/etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
>>>>        echo "NM_CONTROLLED=no" >>
/etc/sysconfig/network-
>>>> scripts/ifcfg-${NETDEV}-tmp
>>>>        /usr/bin/perl -p -i -e 's/dhcp/none/'
/etc/sysconfig/network-
>>>> scripts/ifcfg-${NETDEV}-tmp
>>>>        mv -f /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
\
>>>>              /etc/sysconfig/network-scripts/ifcfg-${NETDEV}
>>>>    done
>>>>
>>>> ###########################
>>>>
>>>>
>>>> On Wed, 25 Feb 2015 14:53:40 -0600, Ashley M. Kirchner
>>>> <ashley at pcraft.com>
>>>> wrote:
>>>>
>>>> Thanks for that Jason but it didn't solve the problem. The
system is
>>>> still
>>>>> coming up with the interfaces shuffled. It seems to
*always* want to
>>>>> use
>>>>> the added ethernet card as eth0.
>>>>>
>>>>> On Wed, Feb 25, 2015 at 1:37 PM, Jason Warr <jason at
warr.net> wrote:
>>>>>
>>>>> Starting back in RHEL/Cent 5 I found that the only way to
make sure
>>>>> your
>>>>>> interface enumeration was consistent after install with
what you had
>>>>>> during
>>>>>> install was to create a udev rules file using the mac
addresses as
>>>>>> the
>>>>>> key.  It is easy to run a short script in postinstall
to create it
>>>>>> based
>>>>>> on
>>>>>> how anaconda has seen them.
>>>>>>
>>>>>> In order for this to work on Cent 6 you have to set
biosdevname=0
>>>>>> on the
>>>>>> kernel boot for the installed system.
>>>>>>
>>>>>> PXE boot options:
>>>>>>
>>>>>> label c6inst-sda
>>>>>>    kernel /linux-boot/cent6-x64/vmlinuz
>>>>>>    append initrd=/linux-boot/cent6-x64/initrd.img
ksdevice=bootif
>>>>>> ip=dhcp
ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg
>>>>>>    ipappend 2
>>>>>>
>>>>>> In kickstart:
>>>>>>
>>>>>> BOOTOPTS="biosdevname=0"
>>>>>>
>>>>>> Also in kickstart I do not specify the config for ANY
network
>>>>>> interfaces.
>>>>>> I let anaconda pull in only the config for the boot
interface from
>>>>>> PXE.
>>>>>> I
>>>>>> manually configure everything else.  The only thing I
do to non-boot
>>>>>> interfaces is set the DHCP and ONBOOT to no.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 25 Feb 2015 14:21:18 -0600, Ashley M. Kirchner
<
>>>>>> ashley at pcraft.com>
>>>>>> wrote:
>>>>>>
>>>>>> Version 6.6 ...
>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 25, 2015 at 1:17 PM, Jim Perrin
<jperrin at centos.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> <overly trimmed>
>>>>>>>
>>>>>>>>
>>>>>>>> On 02/25/2015 01:56 PM, Ashley M. Kirchner
wrote:
>>>>>>>>> Ok, so some of this now works, but I'm
still having problems.
>>>>>>>>> With
>>>>>>>> the
>>>>>>>>> bootif option, the system now correctly
configures and uses the
>>>>>>>>> same
>>>>>>>>> interface to get its kickstart file.
However, when the system is
>>>>>>>>> done
>>>>>>>> and
>>>>>>>>> boots up, the interfaces are still messed
up. So this is what I
>>>>>>>>> have
>>>>>>>> in
>>>>>>>> the
>>>>>>>>> kickstart file:
>>>>>>>>
>>>>>>>> What version of CentOS 6 is this?
>>>>>>>>
>>>>>>>>> In the PXE config file I have:
>>>>>>>>>
>>>>>>>>> IPAPPEND 2
>>>>>>>>> APPEND ks=http://192.168.x.x/ks/portico.ks
>>>>>>>> initrd=centos/x86_64/initrd.img
>>>>>>>>> ramdisk_size=100000 ksdevice=bootif
>>>>>>>>
>>>>>>>>> As soon as I *remove* the additional
ethernet card, the system
>>>>>>>>> will
>>>>>>>> boot
>>>>>>>> up
>>>>>>>>> with the ports configured correctly (port 1
= eth0, port 2 =
>>>>>>>>> eth1).
>>>>>>>> So
>>>>>>>> why
>>>>>>>>> is it that as soon as there is an
additional one, all things go
>>>>>>>>> to
>>>>>>>> hell?
>>>>>>>>> Why must the boot process shuffle them?
More importantly, how do
>>>>>>>>> I
>>>>>>>> prevent
>>>>>>>>> this so that the system comes up properly
after a kickstart
>>>>>>>>> install?
>>>>>>>>>
>>>>>>>>
>>>>>>>> The reason I ask the version, is this is
exactly the sort of
>>>>>>>> thing that
>>>>>>>> biosdevname is designed to solve. With
biosdevname, you get
>>>>>>>> devices
>>>>>>>> like
>>>>>>>> 'em1, em2, p6p1', which aren't as
friendly as 'eth0' but also keep
>>>>>>>> names
>>>>>>>> sane and avoid the hair-tearing issues
you're experiencing
>>>>>>>> currently.
>>>>>>>> You don't appear to be adding anything via
your append line that
>>>>>>>> would
>>>>>>>> disable biosdevname, so I must assume
you're using a much older 6
>>>>>>>> base
>>>>>>>> install.
>>>>>>>>
>>>>>>>>
>>>>>>>> In my experience biosdevname creates just as
many problems as it
>>>>>> solves.
>>>>>> Dell can keep it.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>>> Jim Perrin
>>>>>>>> The CentOS Project | http://www.centos.org
>>>>>>>> twitter: @BitIntegrity | GPG Key: FA09AD77
>>>>>>>> _______________________________________________
>>>>>>>> CentOS mailing list
>>>>>>>> CentOS at centos.org
>>>>>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>> CentOS mailing list
>>>>>>> CentOS at centos.org
>>>>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>> CentOS mailing list
>>>>> CentOS at centos.org
>>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>>
>>>> _______________________________________________
>>>> CentOS mailing list
>>>> CentOS at centos.org
>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>
>>>>
>>> _______________________________________________