Linux_hpc_swstack - Jun 2008 - Sun HPC Software, Linux Edition 1.0 Preview

Just wanted to send out a note to the list that we have a preview of the 
Linux Edition online now.  In it''s initial form, we''re
delivering it as
a "build your own ISO" package, which basically is a set of tools that
go out and grab the relevant pieces and build an ISO that can install a 
management node.

The main product page (for those interested in reading more) can be 
found at:

http://www.sun.com/software/products/hpcsoftware/

But, for those that just want to get started, go here:

http://dlc.sun.com/linux_hpc/1.0/readme.html

So, please try it out and let the list know how everything is going.

-- 
"A simile is not a lie, unless it is a bad simile."
- Christopher John Francis Boone

Hi,

I had a try of "sunhpc-linux-iso" script and noticed a minor error.

---------------------------------------------------------------------------------
Error MSG
---------------------------------------------------------------------------------
[root at localhost ~]# sunhpc-linux-iso
Running sanity checks on build environment... Done.

NOTICE: Couldn''t find implantisomd5 on your system.

This is not fatal, but it means that the anaconda installer will not
be able to perform media verification on your newly-burnt DVD.

If you really want this functionality, you''ll need to install the
anaconda-runtime package and all its dependencies.

Using wget to download ISO skeleton...
--09:10:36--  http://dlc.sun.com/linux_hpc/iso/base/sunhpc_base.tgz
Resolving dlc.sun.com... 72.5.124.114
Connecting to dlc.sun.com|72.5.124.114|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 157452080 (150M) [text/plain]
Saving to: `sunhpc_base.tgz''

100%[=======================================>] 157,452,080 1.90K/s   in 43m
59s

09:54:38 (58.3 KB/s) - `sunhpc_base.tgz'' saved [157452080/157452080]

Done.
Unpacking ISO skeleton... Done.
Generating XML for installer... Done.
Downloading RPMs for ISO...
usage: "usage: yumdownloader [options] package1 [package2] [package..]

Command line error: no such option: --config
FATAL: yum download for CentOS 5 was incomplete
-------------------------------------------------------------------------------------------------------------

It could be easily fixed with a small change at
"/usr/bin/sunhpc-linux-iso" line 108 of 207

Change from "yumdownloader --config=$CONFIG_DIR/yum.conf \"
to "yumdownloader -c $CONFIG_DIR/yum.conf \"

similar change for centOS 4 at line 116.

------------------------------------------------------------------------------------------------------------
Testing Environment
------------------------------------------------------------------------------------------------------------
CentOS 5.2 fresh installation

yumdownloader - 1.0 (yum - 3.2.8)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/linux_hpc_swstack/attachments/20080703/a4616abf/attachment.html

I want to report another error on
/etc/sunhpc-linux-iso/yum.repos.d/ss12.repo
Neither of these baseurls is accessible.

[sunhpc-ss12]
name=SunStudio 12
baseurl=http://dlc.sun.com/linux_hpc/yum/SunStudio/12
baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
gpgcheck=0
enabled=1

It brings an un-continuable error.
------------------------------------------------------------------------------------------------------------------
Error MSG
------------------------------------------------------------------------------------------------------------------
Unpacking ISO skeleton... Done.
Generating XML for installer... Done.
Downloading RPMs for ISO...
Loading "fastestmirror" plugin
Determining fastest mirrors
 * sunhpc-centos4: dlc.sun.com
 * sunhpc-centos5: dlc.sun.com
 * c4-updates: ftp.oss.eznetsols.org
 * c5-extras: ftp.oss.eznetsols.org
 * c4-base: ftp.oss.eznetsols.org
 * c5-addons: ftp.oss.eznetsols.org
 * c4-addons: ftp.oss.eznetsols.org
 * c5-updates: ftp.oss.eznetsols.org
 * c5-base: ftp.oss.eznetsols.org
 * sunhpc-ss12: giraffe.lustre.org
 * c4-extras: ftp.oss.eznetsols.org
http://giraffe.lustre/dlc_stage/yum/SunStudio/12/repodata/repomd.xml: [Errno
4] IOError: <urlopen error (-2, ''Name or service not
known'')>
Trying other mirror.
Cannot retrieve repository metadata (repomd.xml) for repository:
sunhpc-ss12. Please verify its path and try again
FATAL: yum download for CentOS 5 was incomplete

Regards,
Zhiqi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/linux_hpc_swstack/attachments/20080703/963dd747/attachment-0001.html

Hmm, it looks like the newer versions of yumdownloader changed their  
options; but your observation is correct since both versions will  
accept the "-c" option.

I''m adding this change in, and will look at posting a new version of  
the script shortly.

On Jul 2, 2008, at 9:38 AM, Zhiqi Tao wrote:
> Hi,
>
> I had a try of "sunhpc-linux-iso" script and noticed a minor
error.
>
>
---------------------------------------------------------------------------------
> Error MSG
>
---------------------------------------------------------------------------------
> [root at localhost ~]# sunhpc-linux-iso
> Running sanity checks on build environment... Done.
>
> NOTICE: Couldn''t find implantisomd5 on your system.
>
> This is not fatal, but it means that the anaconda installer will not
> be able to perform media verification on your newly-burnt DVD.
>
> If you really want this functionality, you''ll need to install the
> anaconda-runtime package and all its dependencies.
>
> Using wget to download ISO skeleton...
> --09:10:36--  http://dlc.sun.com/linux_hpc/iso/base/sunhpc_base.tgz
> Resolving dlc.sun.com... 72.5.124.114
> Connecting to dlc.sun.com|72.5.124.114|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 157452080 (150M) [text/plain]
> Saving to: `sunhpc_base.tgz''
>
> 100%[========================> ==============>] 157,452,080 1.90K/s  
in 43m 59s
>
> 09:54:38 (58.3 KB/s) - `sunhpc_base.tgz'' saved
[157452080/157452080]
>
> Done.
> Unpacking ISO skeleton... Done.
> Generating XML for installer... Done.
> Downloading RPMs for ISO...
> usage: "usage: yumdownloader [options] package1 [package2] [package..]
>
> Command line error: no such option: --config
> FATAL: yum download for CentOS 5 was incomplete
>
-------------------------------------------------------------------------------------------------------------
>
> It could be easily fixed with a small change at
> "/usr/bin/sunhpc-linux-iso" line 108 of 207
>
> Change from "yumdownloader --config=$CONFIG_DIR/yum.conf \"
> to "yumdownloader -c $CONFIG_DIR/yum.conf \"
>
> similar change for centOS 4 at line 116.
>
>
------------------------------------------------------------------------------------------------------------
> Testing Environment
>
------------------------------------------------------------------------------------------------------------
> CentOS 5.2 fresh installation
>
> yumdownloader - 1.0 (yum - 3.2.8)
> _______________________________________________
> Linux_hpc_swstack mailing list
> Linux_hpc_swstack at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
--
"A simile is not a lie, unless it is a bad simile."
- Christopher John Francis Boone

This seems to have been overlooked.  I''ve made the change (soon to be  
released) to look like:

[sunhpc-ss12]
name=SunStudio 12
baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
         http://dlc.sun.com/linux_hpc/yum/SunStudio/12
gpgcheck=0
enabled=1

I''ll try and get something tested with these changes and released  
tomorrow.

On Jul 2, 2008, at 9:48 AM, Zhiqi Tao wrote:
> I want to report another error on /etc/sunhpc-linux-iso/yum.repos.d/ 
> ss12.repo
> Neither of these baseurls is accessible.
>
> [sunhpc-ss12]
> name=SunStudio 12
> baseurl=http://dlc.sun.com/linux_hpc/yum/SunStudio/12
> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
> gpgcheck=0
> enabled=1
>
> It brings an un-continuable error.
>
------------------------------------------------------------------------------------------------------------------
> Error MSG
>
------------------------------------------------------------------------------------------------------------------
> Unpacking ISO skeleton... Done.
> Generating XML for installer... Done.
> Downloading RPMs for ISO...
> Loading "fastestmirror" plugin
> Determining fastest mirrors
>  * sunhpc-centos4: dlc.sun.com
>  * sunhpc-centos5: dlc.sun.com
>  * c4-updates: ftp.oss.eznetsols.org
>  * c5-extras: ftp.oss.eznetsols.org
>  * c4-base: ftp.oss.eznetsols.org
>  * c5-addons: ftp.oss.eznetsols.org
>  * c4-addons: ftp.oss.eznetsols.org
>  * c5-updates: ftp.oss.eznetsols.org
>  * c5-base: ftp.oss.eznetsols.org
>  * sunhpc-ss12: giraffe.lustre.org
>  * c4-extras: ftp.oss.eznetsols.org
> http://giraffe.lustre/dlc_stage/yum/SunStudio/12/repodata/ 
> repomd.xml: [Errno 4] IOError: <urlopen error (-2, ''Name or
service
> not known'')>
> Trying other mirror.
> Cannot retrieve repository metadata (repomd.xml) for repository:  
> sunhpc-ss12. Please verify its path and try again
> FATAL: yum download for CentOS 5 was incomplete
>
> Regards,
> Zhiqi
> _______________________________________________
> Linux_hpc_swstack mailing list
> Linux_hpc_swstack at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
--
"A simile is not a lie, unless it is a bad simile."
- Christopher John Francis Boone

Sorry for the spam, apparently I need to think a little bit more  
before I hit send.  At this time we are not supplying sunstudio, and  
therefore that repo is completely unavailable (and the config file  
just needs to be removed).  As such, I''m pulling it from the latest  
RPM; expect a new version available tomorrow.

On Jul 2, 2008, at 8:08 PM, Makia Minich wrote:
> This seems to have been overlooked.  I''ve made the change (soon to
be
> released) to look like:
>
> [sunhpc-ss12]
> name=SunStudio 12
> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>         http://dlc.sun.com/linux_hpc/yum/SunStudio/12
> gpgcheck=0
> enabled=1
>
> I''ll try and get something tested with these changes and released
> tomorrow.
>
> On Jul 2, 2008, at 9:48 AM, Zhiqi Tao wrote:
>
>> I want to report another error on /etc/sunhpc-linux-iso/yum.repos.d/
>> ss12.repo
>> Neither of these baseurls is accessible.
>>
>> [sunhpc-ss12]
>> name=SunStudio 12
>> baseurl=http://dlc.sun.com/linux_hpc/yum/SunStudio/12
>> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>> gpgcheck=0
>> enabled=1
>>
>> It brings an un-continuable error.
>>
------------------------------------------------------------------------------------------------------------------
>> Error MSG
>>
------------------------------------------------------------------------------------------------------------------
>> Unpacking ISO skeleton... Done.
>> Generating XML for installer... Done.
>> Downloading RPMs for ISO...
>> Loading "fastestmirror" plugin
>> Determining fastest mirrors
>> * sunhpc-centos4: dlc.sun.com
>> * sunhpc-centos5: dlc.sun.com
>> * c4-updates: ftp.oss.eznetsols.org
>> * c5-extras: ftp.oss.eznetsols.org
>> * c4-base: ftp.oss.eznetsols.org
>> * c5-addons: ftp.oss.eznetsols.org
>> * c4-addons: ftp.oss.eznetsols.org
>> * c5-updates: ftp.oss.eznetsols.org
>> * c5-base: ftp.oss.eznetsols.org
>> * sunhpc-ss12: giraffe.lustre.org
>> * c4-extras: ftp.oss.eznetsols.org
>> http://giraffe.lustre/dlc_stage/yum/SunStudio/12/repodata/
>> repomd.xml: [Errno 4] IOError: <urlopen error (-2, ''Name or
service
>> not known'')>
>> Trying other mirror.
>> Cannot retrieve repository metadata (repomd.xml) for repository:
>> sunhpc-ss12. Please verify its path and try again
>> FATAL: yum download for CentOS 5 was incomplete
>>
>> Regards,
>> Zhiqi
>> _______________________________________________
>> Linux_hpc_swstack mailing list
>> Linux_hpc_swstack at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
>
> --
> "A simile is not a lie, unless it is a bad simile."
> - Christopher John Francis Boone
>
> _______________________________________________
> Linux_hpc_swstack mailing list
> Linux_hpc_swstack at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
--
"A simile is not a lie, unless it is a bad simile."
- Christopher John Francis Boone

Dear Makia,

Another minor suggestion.

Is there any chance not to create a new temp working directory for
each time? So the download process could continue from a
partially-downloaded file in previous instance. 150MB is considerable
for a slow internet connection.

Changes could be something like:

From
"wget $iso_skel \"
to
"wget -c $iso_skel \"

From
"ISO_WORKSPACE=${ISO_WORKSPACE:-$(mktemp -d)}"
to
"ISO_WORKSPACE=${ISO_WORKSPACE:-$(mktemp -d /tmp/SunHpcIso)}"

Other changes might also be needed.
"
Also beginning with Wget 1.7, if you use ''-c'' on a file which
is of
equal size as the one on the server, Wget will refuse to download the
file and print an explanatory message. The same happens when the file
is smaller on the server than locally (presumably because it was
changed on the server since your last download attempt)?because
"continuing" is not meaningful, no download occurs.
"
http://www.gnu.org/software/wget/manual/wget.html#Download-Options

Regards,
Zhiqi


On Thu, Jul 3, 2008 at 12:08 PM, Makia Minich <makia at sun.com>
wrote:
> This seems to have been overlooked.  I''ve made the change (soon to
be
> released) to look like:
>
> [sunhpc-ss12]
> name=SunStudio 12
> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>        http://dlc.sun.com/linux_hpc/yum/SunStudio/12
> gpgcheck=0
> enabled=1
>
> I''ll try and get something tested with these changes and released
tomorrow.
>
> On Jul 2, 2008, at 9:48 AM, Zhiqi Tao wrote:
>
>> I want to report another error on
>> /etc/sunhpc-linux-iso/yum.repos.d/ss12.repo
>> Neither of these baseurls is accessible.
>>
>> [sunhpc-ss12]
>> name=SunStudio 12
>> baseurl=http://dlc.sun.com/linux_hpc/yum/SunStudio/12
>> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>> gpgcheck=0
>> enabled=1
>>
>> It brings an un-continuable error.
>>
>>
------------------------------------------------------------------------------------------------------------------
>> Error MSG
>>
>>
------------------------------------------------------------------------------------------------------------------
>> Unpacking ISO skeleton... Done.
>> Generating XML for installer... Done.
>> Downloading RPMs for ISO...
>> Loading "fastestmirror" plugin
>> Determining fastest mirrors
>>  * sunhpc-centos4: dlc.sun.com
>>  * sunhpc-centos5: dlc.sun.com
>>  * c4-updates: ftp.oss.eznetsols.org
>>  * c5-extras: ftp.oss.eznetsols.org
>>  * c4-base: ftp.oss.eznetsols.org
>>  * c5-addons: ftp.oss.eznetsols.org
>>  * c4-addons: ftp.oss.eznetsols.org
>>  * c5-updates: ftp.oss.eznetsols.org
>>  * c5-base: ftp.oss.eznetsols.org
>>  * sunhpc-ss12: giraffe.lustre.org
>>  * c4-extras: ftp.oss.eznetsols.org
>> http://giraffe.lustre/dlc_stage/yum/SunStudio/12/repodata/repomd.xml:
>> [Errno 4] IOError: <urlopen error (-2, ''Name or service not
known'')>
>> Trying other mirror.
>> Cannot retrieve repository metadata (repomd.xml) for repository:
>> sunhpc-ss12. Please verify its path and try again
>> FATAL: yum download for CentOS 5 was incomplete
>>
>> Regards,
>> Zhiqi
>> _______________________________________________
>> Linux_hpc_swstack mailing list
>> Linux_hpc_swstack at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
>
> --
> "A simile is not a lie, unless it is a bad simile."
> - Christopher John Francis Boone
>
>

There was some internal discussion on how best to handle this (as this  
is a really good point), and I believe that we have some ideas.  I''m  
going to hold off releasing something until after the (US) holiday  
while I work on some of these changes.

On Jul 2, 2008, at 8:34 PM, Zhiqi Tao wrote:
> Dear Makia,
>
> Another minor suggestion.
>
> Is there any chance not to create a new temp working directory for
> each time? So the download process could continue from a
> partially-downloaded file in previous instance. 150MB is considerable
> for a slow internet connection.
>
> Changes could be something like:
>
> From
> "wget $iso_skel \"
> to
> "wget -c $iso_skel \"
>
> From
> "ISO_WORKSPACE=${ISO_WORKSPACE:-$(mktemp -d)}"
> to
> "ISO_WORKSPACE=${ISO_WORKSPACE:-$(mktemp -d /tmp/SunHpcIso)}"
>
> Other changes might also be needed.
> "
> Also beginning with Wget 1.7, if you use ''-c'' on a file
which is of
> equal size as the one on the server, Wget will refuse to download the
> file and print an explanatory message. The same happens when the file
> is smaller on the server than locally (presumably because it was
> changed on the server since your last download attempt)?because
> "continuing" is not meaningful, no download occurs.
> "
> http://www.gnu.org/software/wget/manual/wget.html#Download-Options
>
> Regards,
> Zhiqi
>
>
> On Thu, Jul 3, 2008 at 12:08 PM, Makia Minich <makia at sun.com>
wrote:
>> This seems to have been overlooked.  I''ve made the change
(soon to be
>> released) to look like:
>>
>> [sunhpc-ss12]
>> name=SunStudio 12
>> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>>       http://dlc.sun.com/linux_hpc/yum/SunStudio/12
>> gpgcheck=0
>> enabled=1
>>
>> I''ll try and get something tested with these changes and
released
>> tomorrow.
>>
>> On Jul 2, 2008, at 9:48 AM, Zhiqi Tao wrote:
>>
>>> I want to report another error on
>>> /etc/sunhpc-linux-iso/yum.repos.d/ss12.repo
>>> Neither of these baseurls is accessible.
>>>
>>> [sunhpc-ss12]
>>> name=SunStudio 12
>>> baseurl=http://dlc.sun.com/linux_hpc/yum/SunStudio/12
>>> baseurl=http://giraffe.lustre/dlc_stage/yum/SunStudio/12
>>> gpgcheck=0
>>> enabled=1
>>>
>>> It brings an un-continuable error.
>>>
>>>
------------------------------------------------------------------------------------------------------------------
>>> Error MSG
>>>
>>>
------------------------------------------------------------------------------------------------------------------
>>> Unpacking ISO skeleton... Done.
>>> Generating XML for installer... Done.
>>> Downloading RPMs for ISO...
>>> Loading "fastestmirror" plugin
>>> Determining fastest mirrors
>>> * sunhpc-centos4: dlc.sun.com
>>> * sunhpc-centos5: dlc.sun.com
>>> * c4-updates: ftp.oss.eznetsols.org
>>> * c5-extras: ftp.oss.eznetsols.org
>>> * c4-base: ftp.oss.eznetsols.org
>>> * c5-addons: ftp.oss.eznetsols.org
>>> * c4-addons: ftp.oss.eznetsols.org
>>> * c5-updates: ftp.oss.eznetsols.org
>>> * c5-base: ftp.oss.eznetsols.org
>>> * sunhpc-ss12: giraffe.lustre.org
>>> * c4-extras: ftp.oss.eznetsols.org
>>> http://giraffe.lustre/dlc_stage/yum/SunStudio/12/repodata/ 
>>> repomd.xml:
>>> [Errno 4] IOError: <urlopen error (-2, ''Name or service
not known'')>
>>> Trying other mirror.
>>> Cannot retrieve repository metadata (repomd.xml) for repository:
>>> sunhpc-ss12. Please verify its path and try again
>>> FATAL: yum download for CentOS 5 was incomplete
>>>
>>> Regards,
>>> Zhiqi
>>> _______________________________________________
>>> Linux_hpc_swstack mailing list
>>> Linux_hpc_swstack at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/linux_hpc_swstack
>>
>> --
>> "A simile is not a lie, unless it is a bad simile."
>> - Christopher John Francis Boone
>>
>>
--
"A simile is not a lie, unless it is a bad simile."
- Christopher John Francis Boone