Here is my script for post install if you want to try it.
In order for the shuffling to not occur you do need to create the udev
rules file somehow. I am not sure how mangled this will be in email but
it is worth a try. It should run OK with nothing else. I have a better
version in the works but the enhancements are mainly useful for Fedora
19-21.
I did forget to say I also block NetworkManager from the interfaces.
############################
#!/bin/bash
## BIND MAC address to proper interfaces so they stay persistent
## I want them to stay as they were in kickstart
## This can cause issues with VLAN interfaces for both bond dev's and base
eth dev's.
## The bond one was solved by adding in the "KERNEL="eth?*" as
that will
only apply to physical
## Devices. Once we have VLAN's on a real device instead of just on
BOND's this then applies
## to the hardware devices as well. The core issue is that the MAC
address is inherited
## by all of the children devices and thus the UDEV rule has to be
crafted to only apply
## to the base physical device.
## This one was solved via adding DRIVERS=="?*" as the VLAN int's
wont
have one
echo "[KICKSTART] Binding eth interfaces to the expected MAC address
in UDEV"
echo "## Created by Kickstart to keep network interfaces in an
expected order" > \
/etc/udev/rules.d/70-persistent-net.rules
echo "" >> \
/etc/udev/rules.d/70-persistent-net.rules
cd /sys/class/net/
for NETDEV in $(ls | grep eth | sort)
do
## Create a UDEV rule for each eth interface
echo "## ${NETDEV} interface" >> \
/etc/udev/rules.d/70-persistent-net.rules
## We throw this one in here as it can contain some useful
information
echo "## $(dmesg | grep ${NETDEV} | grep -i -v -e
"console" -e
"Command line" | head -1)" >> \
/etc/udev/rules.d/70-persistent-net.rules
echo -n "SUBSYSTEM==\"net\", ACTION==\"add\",
DRIVERS==\"?*\", " >> \
/etc/udev/rules.d/70-persistent-net.rules
echo -n "ATTR{address}==\"$(cat ${NETDEV}/address)\",
" >> \
/etc/udev/rules.d/70-persistent-net.rules
echo -n "ATTR{dev_id}==\"0x0\",
ATTR{type}==\"1\",
KERNEL==\"eth?*\", " >> \
/etc/udev/rules.d/70-persistent-net.rules
echo -e "NAME=\"${NETDEV}\"\n" >> \
/etc/udev/rules.d/70-persistent-net.rules
## Make a log of the devices present during install
echo -e "${NETDEV} $(cat ${NETDEV}/address)\n" >>
/root/ksnet-devices
## Also remove the HWADDR line from all of the net config files
grep -v -e NAME -e HWADDR -e NM_CONTROLLED \
/etc/sysconfig/network-scripts/ifcfg-${NETDEV} | sed
's/\"//g'
\
> /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
echo "NM_CONTROLLED=no" >>
/etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
/usr/bin/perl -p -i -e 's/dhcp/none/'
/etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
mv -f /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp \
/etc/sysconfig/network-scripts/ifcfg-${NETDEV}
done
###########################
On Wed, 25 Feb 2015 14:53:40 -0600, Ashley M. Kirchner <ashley at
pcraft.com>
wrote:
> Thanks for that Jason but it didn't solve the problem. The system is
> still
> coming up with the interfaces shuffled. It seems to *always* want to use
> the added ethernet card as eth0.
>
> On Wed, Feb 25, 2015 at 1:37 PM, Jason Warr <jason at warr.net>
wrote:
>
>> Starting back in RHEL/Cent 5 I found that the only way to make sure
your
>> interface enumeration was consistent after install with what you had
>> during
>> install was to create a udev rules file using the mac addresses as the
>> key. It is easy to run a short script in postinstall to create it
>> based on
>> how anaconda has seen them.
>>
>> In order for this to work on Cent 6 you have to set biosdevname=0 on
the
>> kernel boot for the installed system.
>>
>> PXE boot options:
>>
>> label c6inst-sda
>> kernel /linux-boot/cent6-x64/vmlinuz
>> append initrd=/linux-boot/cent6-x64/initrd.img ksdevice=bootif
>> ip=dhcp ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg
>> ipappend 2
>>
>> In kickstart:
>>
>> BOOTOPTS="biosdevname=0"
>>
>> Also in kickstart I do not specify the config for ANY network
>> interfaces.
>> I let anaconda pull in only the config for the boot interface from
>> PXE. I
>> manually configure everything else. The only thing I do to non-boot
>> interfaces is set the DHCP and ONBOOT to no.
>>
>>
>>
>> On Wed, 25 Feb 2015 14:21:18 -0600, Ashley M. Kirchner
>> <ashley at pcraft.com>
>> wrote:
>>
>> Version 6.6 ...
>>>
>>> On Wed, Feb 25, 2015 at 1:17 PM, Jim Perrin <jperrin at
centos.org> wrote:
>>>
>>> <overly trimmed>
>>>>
>>>> On 02/25/2015 01:56 PM, Ashley M. Kirchner wrote:
>>>> > Ok, so some of this now works, but I'm still having
problems. With
>>>> the
>>>> > bootif option, the system now correctly configures and
uses the same
>>>> > interface to get its kickstart file. However, when the
system is
>>>> done
>>>> and
>>>> > boots up, the interfaces are still messed up. So this is
what I
>>>> have in
>>>> the
>>>> > kickstart file:
>>>>
>>>> What version of CentOS 6 is this?
>>>>
>>>> > In the PXE config file I have:
>>>> >
>>>> > IPAPPEND 2
>>>> > APPEND ks=http://192.168.x.x/ks/portico.ks
>>>> initrd=centos/x86_64/initrd.img
>>>> > ramdisk_size=100000 ksdevice=bootif
>>>>
>>>> > As soon as I *remove* the additional ethernet card, the
system will
>>>> boot
>>>> up
>>>> > with the ports configured correctly (port 1 = eth0, port 2
= eth1).
>>>> So
>>>> why
>>>> > is it that as soon as there is an additional one, all
things go to
>>>> hell?
>>>> > Why must the boot process shuffle them? More importantly,
how do I
>>>> prevent
>>>> > this so that the system comes up properly after a
kickstart install?
>>>> >
>>>>
>>>> The reason I ask the version, is this is exactly the sort of
thing
>>>> that
>>>> biosdevname is designed to solve. With biosdevname, you get
devices
>>>> like
>>>> 'em1, em2, p6p1', which aren't as friendly as
'eth0' but also keep
>>>> names
>>>> sane and avoid the hair-tearing issues you're experiencing
currently.
>>>> You don't appear to be adding anything via your append line
that would
>>>> disable biosdevname, so I must assume you're using a much
older 6 base
>>>> install.
>>>>
>>>>
>> In my experience biosdevname creates just as many problems as it
solves.
>> Dell can keep it.
>>
>>
>>
>>>> --
>>>> Jim Perrin
>>>> The CentOS Project | http://www.centos.org
>>>> twitter: @BitIntegrity | GPG Key: FA09AD77
>>>> _______________________________________________
>>>> CentOS mailing list
>>>> CentOS at centos.org
>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>
>>>>
>>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS at centos.org
>>> http://lists.centos.org/mailman/listinfo/centos
>>>
>>
>>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
Ok, when I run that, it creates a now "custom" 70-persistent-net.rules, but the interfaces are still out of order, with the added one listed first, and the built-ins 2nd and 3rd. On Wed, Feb 25, 2015 at 2:00 PM, Jason Warr <jason at warr.net> wrote:> Here is my script for post install if you want to try it. > > In order for the shuffling to not occur you do need to create the udev > rules file somehow. I am not sure how mangled this will be in email but it > is worth a try. It should run OK with nothing else. I have a better > version in the works but the enhancements are mainly useful for Fedora > 19-21. > > I did forget to say I also block NetworkManager from the interfaces. > > ############################ > > #!/bin/bash > ## BIND MAC address to proper interfaces so they stay persistent > ## I want them to stay as they were in kickstart > > ## This can cause issues with VLAN interfaces for both bond dev's and base > eth dev's. > ## The bond one was solved by adding in the "KERNEL="eth?*" as that will > only apply to physical > ## Devices. Once we have VLAN's on a real device instead of just on > BOND's this then applies > ## to the hardware devices as well. The core issue is that the MAC > address is inherited > ## by all of the children devices and thus the UDEV rule has to be > crafted to only apply > ## to the base physical device. > ## This one was solved via adding DRIVERS=="?*" as the VLAN int's wont > have one > > echo "[KICKSTART] Binding eth interfaces to the expected MAC address > in UDEV" > echo "## Created by Kickstart to keep network interfaces in an > expected order" > \ > /etc/udev/rules.d/70-persistent-net.rules > echo "" >> \ > /etc/udev/rules.d/70-persistent-net.rules > > cd /sys/class/net/ > for NETDEV in $(ls | grep eth | sort) > do > ## Create a UDEV rule for each eth interface > echo "## ${NETDEV} interface" >> \ > /etc/udev/rules.d/70-persistent-net.rules > > ## We throw this one in here as it can contain some useful > information > echo "## $(dmesg | grep ${NETDEV} | grep -i -v -e "console" -e > "Command line" | head -1)" >> \ > /etc/udev/rules.d/70-persistent-net.rules > > echo -n "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", " > >> \ >>> >> /etc/udev/rules.d/70-persistent-net.rules > echo -n "ATTR{address}==\"$(cat ${NETDEV}/address)\", " >> \ > /etc/udev/rules.d/70-persistent-net.rules > echo -n "ATTR{dev_id}==\"0x0\", ATTR{type}==\"1\", > KERNEL==\"eth?*\", " >> \ > /etc/udev/rules.d/70-persistent-net.rules > echo -e "NAME=\"${NETDEV}\"\n" >> \ > /etc/udev/rules.d/70-persistent-net.rules > > ## Make a log of the devices present during install > echo -e "${NETDEV} $(cat ${NETDEV}/address)\n" >> > /root/ksnet-devices > > ## Also remove the HWADDR line from all of the net config files > grep -v -e NAME -e HWADDR -e NM_CONTROLLED \ > /etc/sysconfig/network-scripts/ifcfg-${NETDEV} | sed > 's/\"//g' \ > > /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp > echo "NM_CONTROLLED=no" >> /etc/sysconfig/network- > scripts/ifcfg-${NETDEV}-tmp > /usr/bin/perl -p -i -e 's/dhcp/none/' /etc/sysconfig/network- > scripts/ifcfg-${NETDEV}-tmp > mv -f /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp \ > /etc/sysconfig/network-scripts/ifcfg-${NETDEV} > done > > ########################### > > > On Wed, 25 Feb 2015 14:53:40 -0600, Ashley M. Kirchner <ashley at pcraft.com> > wrote: > > Thanks for that Jason but it didn't solve the problem. The system is still >> coming up with the interfaces shuffled. It seems to *always* want to use >> the added ethernet card as eth0. >> >> On Wed, Feb 25, 2015 at 1:37 PM, Jason Warr <jason at warr.net> wrote: >> >> Starting back in RHEL/Cent 5 I found that the only way to make sure your >>> interface enumeration was consistent after install with what you had >>> during >>> install was to create a udev rules file using the mac addresses as the >>> key. It is easy to run a short script in postinstall to create it based >>> on >>> how anaconda has seen them. >>> >>> In order for this to work on Cent 6 you have to set biosdevname=0 on the >>> kernel boot for the installed system. >>> >>> PXE boot options: >>> >>> label c6inst-sda >>> kernel /linux-boot/cent6-x64/vmlinuz >>> append initrd=/linux-boot/cent6-x64/initrd.img ksdevice=bootif >>> ip=dhcp ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg >>> ipappend 2 >>> >>> In kickstart: >>> >>> BOOTOPTS="biosdevname=0" >>> >>> Also in kickstart I do not specify the config for ANY network interfaces. >>> I let anaconda pull in only the config for the boot interface from PXE. >>> I >>> manually configure everything else. The only thing I do to non-boot >>> interfaces is set the DHCP and ONBOOT to no. >>> >>> >>> >>> On Wed, 25 Feb 2015 14:21:18 -0600, Ashley M. Kirchner < >>> ashley at pcraft.com> >>> wrote: >>> >>> Version 6.6 ... >>> >>>> >>>> On Wed, Feb 25, 2015 at 1:17 PM, Jim Perrin <jperrin at centos.org> wrote: >>>> >>>> <overly trimmed> >>>> >>>>> >>>>> On 02/25/2015 01:56 PM, Ashley M. Kirchner wrote: >>>>> > Ok, so some of this now works, but I'm still having problems. With >>>>> the >>>>> > bootif option, the system now correctly configures and uses the same >>>>> > interface to get its kickstart file. However, when the system is done >>>>> and >>>>> > boots up, the interfaces are still messed up. So this is what I have >>>>> in >>>>> the >>>>> > kickstart file: >>>>> >>>>> What version of CentOS 6 is this? >>>>> >>>>> > In the PXE config file I have: >>>>> > >>>>> > IPAPPEND 2 >>>>> > APPEND ks=http://192.168.x.x/ks/portico.ks >>>>> initrd=centos/x86_64/initrd.img >>>>> > ramdisk_size=100000 ksdevice=bootif >>>>> >>>>> > As soon as I *remove* the additional ethernet card, the system will >>>>> boot >>>>> up >>>>> > with the ports configured correctly (port 1 = eth0, port 2 = eth1). >>>>> So >>>>> why >>>>> > is it that as soon as there is an additional one, all things go to >>>>> hell? >>>>> > Why must the boot process shuffle them? More importantly, how do I >>>>> prevent >>>>> > this so that the system comes up properly after a kickstart install? >>>>> > >>>>> >>>>> The reason I ask the version, is this is exactly the sort of thing that >>>>> biosdevname is designed to solve. With biosdevname, you get devices >>>>> like >>>>> 'em1, em2, p6p1', which aren't as friendly as 'eth0' but also keep >>>>> names >>>>> sane and avoid the hair-tearing issues you're experiencing currently. >>>>> You don't appear to be adding anything via your append line that would >>>>> disable biosdevname, so I must assume you're using a much older 6 base >>>>> install. >>>>> >>>>> >>>>> In my experience biosdevname creates just as many problems as it >>> solves. >>> Dell can keep it. >>> >>> >>> >>> -- >>>>> Jim Perrin >>>>> The CentOS Project | http://www.centos.org >>>>> twitter: @BitIntegrity | GPG Key: FA09AD77 >>>>> _______________________________________________ >>>>> CentOS mailing list >>>>> CentOS at centos.org >>>>> http://lists.centos.org/mailman/listinfo/centos >>>>> >>>>> >>>>> _______________________________________________ >>>>> >>>> CentOS mailing list >>>> CentOS at centos.org >>>> http://lists.centos.org/mailman/listinfo/centos >>>> >>>> >>> >>> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> http://lists.centos.org/mailman/listinfo/centos >> > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > >
Define out of order in this case just so I know for sure what you mean. What my solution does, or at least does reliably in my case, is make sure the interfaces are in the same order once installed as the install kernel saw them. It won't re-order them to be sequential based on bus, mac or driver. I am working on that but it will also include naming the devices based on the module name, similar to how FreeBSD and Solaris do it. Just to get an idea of what might be going on can you run "dmesg | grep eth" so I can see some of what udev is doing? On Wed, 25 Feb 2015 16:28:31 -0600, Ashley M. Kirchner <ashley at pcraft.com> wrote:> Ok, when I run that, it creates a now "custom" 70-persistent-net.rules, > but > the interfaces are still out of order, with the added one listed first, > and > the built-ins 2nd and 3rd. > > On Wed, Feb 25, 2015 at 2:00 PM, Jason Warr <jason at warr.net> wrote: > >> Here is my script for post install if you want to try it. >> >> In order for the shuffling to not occur you do need to create the udev >> rules file somehow. I am not sure how mangled this will be in email >> but it >> is worth a try. It should run OK with nothing else. I have a better >> version in the works but the enhancements are mainly useful for Fedora >> 19-21. >> >> I did forget to say I also block NetworkManager from the interfaces. >> >> ############################ >> >> #!/bin/bash >> ## BIND MAC address to proper interfaces so they stay persistent >> ## I want them to stay as they were in kickstart >> >> ## This can cause issues with VLAN interfaces for both bond dev's and >> base >> eth dev's. >> ## The bond one was solved by adding in the "KERNEL="eth?*" as that >> will >> only apply to physical >> ## Devices. Once we have VLAN's on a real device instead of just on >> BOND's this then applies >> ## to the hardware devices as well. The core issue is that the MAC >> address is inherited >> ## by all of the children devices and thus the UDEV rule has to be >> crafted to only apply >> ## to the base physical device. >> ## This one was solved via adding DRIVERS=="?*" as the VLAN int's wont >> have one >> >> echo "[KICKSTART] Binding eth interfaces to the expected MAC address >> in UDEV" >> echo "## Created by Kickstart to keep network interfaces in an >> expected order" > \ >> /etc/udev/rules.d/70-persistent-net.rules >> echo "" >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> >> cd /sys/class/net/ >> for NETDEV in $(ls | grep eth | sort) >> do >> ## Create a UDEV rule for each eth interface >> echo "## ${NETDEV} interface" >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> >> ## We throw this one in here as it can contain some useful >> information >> echo "## $(dmesg | grep ${NETDEV} | grep -i -v -e "console" -e >> "Command line" | head -1)" >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> >> echo -n "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", " >> >>> \ >>>> >>> /etc/udev/rules.d/70-persistent-net.rules >> echo -n "ATTR{address}==\"$(cat ${NETDEV}/address)\", " >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> echo -n "ATTR{dev_id}==\"0x0\", ATTR{type}==\"1\", >> KERNEL==\"eth?*\", " >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> echo -e "NAME=\"${NETDEV}\"\n" >> \ >> /etc/udev/rules.d/70-persistent-net.rules >> >> ## Make a log of the devices present during install >> echo -e "${NETDEV} $(cat ${NETDEV}/address)\n" >> >> /root/ksnet-devices >> >> ## Also remove the HWADDR line from all of the net config files >> grep -v -e NAME -e HWADDR -e NM_CONTROLLED \ >> /etc/sysconfig/network-scripts/ifcfg-${NETDEV} | sed >> 's/\"//g' \ >> > /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp >> echo "NM_CONTROLLED=no" >> /etc/sysconfig/network- >> scripts/ifcfg-${NETDEV}-tmp >> /usr/bin/perl -p -i -e 's/dhcp/none/' /etc/sysconfig/network- >> scripts/ifcfg-${NETDEV}-tmp >> mv -f /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp \ >> /etc/sysconfig/network-scripts/ifcfg-${NETDEV} >> done >> >> ########################### >> >> >> On Wed, 25 Feb 2015 14:53:40 -0600, Ashley M. Kirchner >> <ashley at pcraft.com> >> wrote: >> >> Thanks for that Jason but it didn't solve the problem. The system is >> still >>> coming up with the interfaces shuffled. It seems to *always* want to >>> use >>> the added ethernet card as eth0. >>> >>> On Wed, Feb 25, 2015 at 1:37 PM, Jason Warr <jason at warr.net> wrote: >>> >>> Starting back in RHEL/Cent 5 I found that the only way to make sure >>> your >>>> interface enumeration was consistent after install with what you had >>>> during >>>> install was to create a udev rules file using the mac addresses as the >>>> key. It is easy to run a short script in postinstall to create it >>>> based >>>> on >>>> how anaconda has seen them. >>>> >>>> In order for this to work on Cent 6 you have to set biosdevname=0 on >>>> the >>>> kernel boot for the installed system. >>>> >>>> PXE boot options: >>>> >>>> label c6inst-sda >>>> kernel /linux-boot/cent6-x64/vmlinuz >>>> append initrd=/linux-boot/cent6-x64/initrd.img ksdevice=bootif >>>> ip=dhcp ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg >>>> ipappend 2 >>>> >>>> In kickstart: >>>> >>>> BOOTOPTS="biosdevname=0" >>>> >>>> Also in kickstart I do not specify the config for ANY network >>>> interfaces. >>>> I let anaconda pull in only the config for the boot interface from >>>> PXE. >>>> I >>>> manually configure everything else. The only thing I do to non-boot >>>> interfaces is set the DHCP and ONBOOT to no. >>>> >>>> >>>> >>>> On Wed, 25 Feb 2015 14:21:18 -0600, Ashley M. Kirchner < >>>> ashley at pcraft.com> >>>> wrote: >>>> >>>> Version 6.6 ... >>>> >>>>> >>>>> On Wed, Feb 25, 2015 at 1:17 PM, Jim Perrin <jperrin at centos.org> >>>>> wrote: >>>>> >>>>> <overly trimmed> >>>>> >>>>>> >>>>>> On 02/25/2015 01:56 PM, Ashley M. Kirchner wrote: >>>>>> > Ok, so some of this now works, but I'm still having problems. With >>>>>> the >>>>>> > bootif option, the system now correctly configures and uses the >>>>>> same >>>>>> > interface to get its kickstart file. However, when the system is >>>>>> done >>>>>> and >>>>>> > boots up, the interfaces are still messed up. So this is what I >>>>>> have >>>>>> in >>>>>> the >>>>>> > kickstart file: >>>>>> >>>>>> What version of CentOS 6 is this? >>>>>> >>>>>> > In the PXE config file I have: >>>>>> > >>>>>> > IPAPPEND 2 >>>>>> > APPEND ks=http://192.168.x.x/ks/portico.ks >>>>>> initrd=centos/x86_64/initrd.img >>>>>> > ramdisk_size=100000 ksdevice=bootif >>>>>> >>>>>> > As soon as I *remove* the additional ethernet card, the system >>>>>> will >>>>>> boot >>>>>> up >>>>>> > with the ports configured correctly (port 1 = eth0, port 2 = >>>>>> eth1). >>>>>> So >>>>>> why >>>>>> > is it that as soon as there is an additional one, all things go to >>>>>> hell? >>>>>> > Why must the boot process shuffle them? More importantly, how do I >>>>>> prevent >>>>>> > this so that the system comes up properly after a kickstart >>>>>> install? >>>>>> > >>>>>> >>>>>> The reason I ask the version, is this is exactly the sort of thing >>>>>> that >>>>>> biosdevname is designed to solve. With biosdevname, you get devices >>>>>> like >>>>>> 'em1, em2, p6p1', which aren't as friendly as 'eth0' but also keep >>>>>> names >>>>>> sane and avoid the hair-tearing issues you're experiencing >>>>>> currently. >>>>>> You don't appear to be adding anything via your append line that >>>>>> would >>>>>> disable biosdevname, so I must assume you're using a much older 6 >>>>>> base >>>>>> install. >>>>>> >>>>>> >>>>>> In my experience biosdevname creates just as many problems as it >>>> solves. >>>> Dell can keep it. >>>> >>>> >>>> >>>> -- >>>>>> Jim Perrin >>>>>> The CentOS Project | http://www.centos.org >>>>>> twitter: @BitIntegrity | GPG Key: FA09AD77 >>>>>> _______________________________________________ >>>>>> CentOS mailing list >>>>>> CentOS at centos.org >>>>>> http://lists.centos.org/mailman/listinfo/centos >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>> CentOS mailing list >>>>> CentOS at centos.org >>>>> http://lists.centos.org/mailman/listinfo/centos >>>>> >>>>> >>>> >>>> _______________________________________________ >>> CentOS mailing list >>> CentOS at centos.org >>> http://lists.centos.org/mailman/listinfo/centos >>> >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> http://lists.centos.org/mailman/listinfo/centos >> >> > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos