crossbow discuss - Dec 2006 - dladm vnic-create fails; conf files; command name

I have tried the Crossbow FBUs on top of build 54 for both SPARC (with bge) and
x64 (with e1000g),
and in both cases I get error in fundamental places.

After additional attempts at things, I am able to hang the SPARC system.

Has anyone else tried these bits on 54?

Thanks
Steffen

First error:
# dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
dladm: VNIC creation failed: File exists

Where the file that exists is /etc/vnic.conf, which has the folling in it:
...
# Use the dladm(1m) command to change the contents of this file.

1       bge3    auto    -       -       0

I had plumbed bge3 manually, so I can understand why something might be in
there.

Second error:
After moving /etc/vnic.conf aside,
# dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
dladm: VNIC creation failed: Invalid argument

And an empty /etc/vnic.conf is create, BTW.

After removing /etc/vnic.conf and rebooting, therefore no bge3 configuration:
# dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
dladm: VNIC creation failed: Invalid argument

Plus, on the console:
vnic: WARNING: vnic_dev_create failed with 22


A documentation NIT:

dladm should reference the file /etc/vnic.conf if it uses it.

And in the spirit of consistency, if netrcm uses a file in
/etc/inet/netrcm.conf, then:

1. It should also be documented in netrcm(1M), and

2. The location of the files for dladm and netrcm should be the same directory,
IMHO.


Also, why is netrcm the command name when a lot of other admin command are *adm?
A lot is an
understatement, there are no other ''admin'' commands that I can
see in /usr/sbin that aren''t *adm.

> Also, why is netrcm the command name when a lot of other admin command
 > are *adm? A lot is an understatement, there are no other
''admin''
 > commands that I can see in /usr/sbin that aren''t *adm.

Indeed -- netrcm also makes me think of the network_rcm module, which is
part of our DR framework.  I was talking about this with the team last
week, and some tought that flowadm might be a reasonable alternative.
In any case, having it end with `adm'' would be nice.

-- 
meem

Peter Memishian wrote On 12/17/06 19:51,:
>  > Also, why is netrcm the command name when a lot of other admin
command
>  > are *adm? A lot is an understatement, there are no other
''admin''
>  > commands that I can see in /usr/sbin that aren''t *adm.
> 
> Indeed -- netrcm also makes me think of the network_rcm module, which is
> part of our DR framework.  I was talking about this with the team last
> week, and some tought that flowadm might be a reasonable alternative.
> In any case, having it end with `adm'' would be nice.
> 
Haven''t had a chance to try it yet, since I can''t get anything
to work :(, but wouldn''t it be better
to put the CPU allocation parts of dladm also with this ''adm''
functionality (-c), so that all
resource related things are done in one command? It is interesting that only the
bandwidth
specification is part of both.

Besides consistency, it may also facilitate separation of duties, if that is
required.

Steffen

Steffen Weiberle wrote:
> I have tried the Crossbow FBUs on top of build 54 for both SPARC (with 
> bge) and x64 (with e1000g), and in both cases I get error in fundamental 
> places.
> 
> After additional attempts at things, I am able to hang the SPARC system.
> 
> Has anyone else tried these bits on 54?
The bits are still based on snv_49, we haven''t tried 54. I''d
suggest
bfu''ing Crossbow from an snv_49 install until with finish our resync 
with build 54.
> 
> Thanks
> Steffen
> 
> First error:
> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
> dladm: VNIC creation failed: File exists
> 
> Where the file that exists is /etc/vnic.conf, which has the folling in it:
> ...
> # Use the dladm(1m) command to change the contents of this file.
> 
> 1       bge3    auto    -       -       0
> 
> I had plumbed bge3 manually, so I can understand why something might be 
> in there.
Why do you think this is an error? You have an existing VNIC with VNIC 
id 1, so you cannot create another VNIC with the same id. VNIC ids must 
be unique on a system-wide basis.

What were the steps performed that lead you to that point?
> 
> Second error:
> After moving /etc/vnic.conf aside,
> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
> dladm: VNIC creation failed: Invalid argument
> 
> And an empty /etc/vnic.conf is create, BTW.
> 
> After removing /etc/vnic.conf and rebooting, therefore no bge3 
> configuration:
> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
> dladm: VNIC creation failed: Invalid argument
> 
> Plus, on the console:
> vnic: WARNING: vnic_dev_create failed with 22
By manually moving /etc/vnic.conf aside, you''ve now introduced an 
inconsistency between the content of the configuration and the kernel 
state. We cannot guarantee the behavior of the system from that point on.

If you see an unexpected behavior, please let us know what that behavior 
is, but do not attempt to modify the back-end configuration directly to 
work-around it.

Did you try removing the existing VNIC with ''dladm
delete-vnic'' first?
> 
> 
> A documentation NIT:
> 
> dladm should reference the file /etc/vnic.conf if it uses it.
No. That file is private to dladm(1M) and should not be touched by the 
administrator, so it should not be part of the interface documented by 
dladm(1M).
> 
> And in the spirit of consistency, if netrcm uses a file in 
> /etc/inet/netrcm.conf, then:
> 
> 1. It should also be documented in netrcm(1M), and
See above.
> 
> 2. The location of the files for dladm and netrcm should be the same 
> directory, IMHO.
The files will move. We''re going to move them to the /etc/dladm 
directory as soon as we resync with an ONNV build which includes the 
GLDv3/wifi changes. The files might even be completely replaced by a SMF 
repository in the future, which is another reason why we don''t want to 
document them.
> 
> 
> Also, why is netrcm the command name when a lot of other admin command 
> are *adm? A lot is an understatement, there are no other
''admin''
> commands that I can see in /usr/sbin that aren''t *adm.
We realize that netrcm(1M) is not an ideal name and we''re considering 
other names.

Nicolas.
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss
-- 
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux

Steffen Weiberle wrote:
> Haven''t had a chance to try it yet, since I can''t get
anything to work
> :(, but wouldn''t it be better to put the CPU allocation parts of
dladm
> also with this ''adm'' functionality (-c), so that all
resource related
> things are done in one command? It is interesting that only the 
> bandwidth specification is part of both.
> 
> Besides consistency, it may also facilitate separation of duties, if 
> that is required.
Even though some of the properties set by dladm(1M) and netrcm(1M) are 
the same (e.g. maximum bandwidth), the objects managed by the two 
subcommands are fundamentally different. dladm(1M) manages data-links, 
while netrcm(1M) (name suggest to change) defines flows on top of these 
data-links. For example if I want to create a VNIC and assign it a set 
of CPUs and a maximum bandwidth, I shouldn''t have to know about 
netrcm(1M) and flows.

Nicolas.

-- 
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux

> Even though some of the properties set by dladm(1M) and netrcm(1M) are 
 > the same (e.g. maximum bandwidth), the objects managed by the two 
 > subcommands are fundamentally different. dladm(1M) manages data-links, 
 > while netrcm(1M) (name suggest to change) defines flows on top of these 
 > data-links.

FWIW: I strongly agree, especially with the rationale.

-- 
meem

Hi Nicolas,

Nicolas Droux wrote On 12/18/06 13:35,:
> Steffen Weiberle wrote:
> 
>> I have tried the Crossbow FBUs on top of build 54 for both SPARC (with 
>> bge) and x64 (with e1000g), and in both cases I get error in 
>> fundamental places.
>>
>> After additional attempts at things, I am able to hang the SPARC
system.
>>
>> Has anyone else tried these bits on 54?
> 
> The bits are still based on snv_49, we haven''t tried 54.
I''d suggest
> bfu''ing Crossbow from an snv_49 install until with finish our
resync
> with build 54.
Where would I find a snv_49 image? SX 11/06 is based on 50. I don''t see
anything that old on nana.
When will the 54 based resync be available? I''ll keep that build
around.
>> Thanks
>> Steffen
>>
>> First error:
>> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
>> dladm: VNIC creation failed: File exists
>>
>> Where the file that exists is /etc/vnic.conf, which has the folling in 
>> it:
>> ...
>> # Use the dladm(1m) command to change the contents of this file.
>>
>> 1       bge3    auto    -       -       0
>>
>> I had plumbed bge3 manually, so I can understand why something might 
>> be in there.
> 
> Why do you think this is an error? You have an existing VNIC with VNIC 
> id 1, so you cannot create another VNIC with the same id. VNIC ids must 
> be unique on a system-wide basis.
> 
> What were the steps performed that lead you to that point?
ifconfig bge3 plumb x.y.x.t netmask + broadcast + up

My first call to dladm create-vnic failed, and the I noticed this file.
>> Second error:
>> After moving /etc/vnic.conf aside,
>> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
>> dladm: VNIC creation failed: Invalid argument
>>
>> And an empty /etc/vnic.conf is create, BTW.
>>
>> After removing /etc/vnic.conf and rebooting, therefore no bge3 
>> configuration:
>> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
>> dladm: VNIC creation failed: Invalid argument
>>
>> Plus, on the console:
>> vnic: WARNING: vnic_dev_create failed with 22
> 
> By manually moving /etc/vnic.conf aside, you''ve now introduced an 
> inconsistency between the content of the configuration and the kernel 
> state. We cannot guarantee the behavior of the system from that point on.
Sure. But I had this problem on the x64 system with e1000g on it and the same
issue. I did not move
any files there, and I had an error on my first attempt to explicitly create a
VNIC. But I also had
done an ifconfig there first. After a reboot and no ifconfig, the problem still
persisted.
> If you see an unexpected behavior, please let us know what that behavior 
> is, but do not attempt to modify the back-end configuration directly to 
> work-around it.
> 
> Did you try removing the existing VNIC with ''dladm
delete-vnic'' first?
No, since I had never created a VNIC, at least not explicitly. Is running
ifconfig also creating a VNIC?
>> A documentation NIT:
>>
>> dladm should reference the file /etc/vnic.conf if it uses it.
I understand your points below. I was led to these files only because of the
error message that a
file already exists.
> No. That file is private to dladm(1M) and should not be touched by the 
> administrator, so it should not be part of the interface documented by 
> dladm(1M).
> 
>>
>> And in the spirit of consistency, if netrcm uses a file in 
>> /etc/inet/netrcm.conf, then:
>>
>> 1. It should also be documented in netrcm(1M), and
> 
> 
> See above.
> 
>>
>> 2. The location of the files for dladm and netrcm should be the same 
>> directory, IMHO.
> 
> 
> The files will move. We''re going to move them to the /etc/dladm 
> directory as soon as we resync with an ONNV build which includes the 
> GLDv3/wifi changes. The files might even be completely replaced by a SMF 
> repository in the future, which is another reason why we don''t
want to
> document them.
> 
>>
>>
>> Also, why is netrcm the command name when a lot of other admin command 
>> are *adm? A lot is an understatement, there are no other
''admin''
>> commands that I can see in /usr/sbin that aren''t *adm.
> 
> 
> We realize that netrcm(1M) is not an ideal name and we''re
considering
> other names.
> 
> Nicolas.
> 
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://opensolaris.org/mailman/listinfo/crossbow-discuss
> 
>

Hi Steffen,

On Dec 18, 2006, at 2:09 PM, Steffen Weiberle wrote:
> Where would I find a snv_49 image? SX 11/06 is based on 50. I
don''t
> see anything that old on nana. When will the 54 based resync be  
> available? I''ll keep that build around.
The resync is in progress, but it takes some time to perform due to  
the complexity of the changes involved. We''re hoping to have it done  
before the holiday break, or shortly after the break. I don''t wait to  
wait until then to look into the issue you reported though.
>
>>> Thanks
>>> Steffen
>>>
>>> First error:
>>> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
>>> dladm: VNIC creation failed: File exists
>>>
>>> Where the file that exists is /etc/vnic.conf, which has the  
>>> folling in it:
>>> ...
>>> # Use the dladm(1m) command to change the contents of this file.
>>>
>>> 1       bge3    auto    -       -       0
>>>
>>> I had plumbed bge3 manually, so I can understand why something  
>>> might be in there.
>> Why do you think this is an error? You have an existing VNIC with  
>> VNIC id 1, so you cannot create another VNIC with the same id.  
>> VNIC ids must be unique on a system-wide basis.
>> What were the steps performed that lead you to that point?
>
> ifconfig bge3 plumb x.y.x.t netmask + broadcast + up
>
> My first call to dladm create-vnic failed, and the I noticed this  
> file.
<snip>
> Sure. But I had this problem on the x64 system with e1000g on it  
> and the same issue. I did not move any files there, and I had an  
> error on my first attempt to explicitly create a VNIC. But I also  
> had done an ifconfig there first. After a reboot and no ifconfig,  
> the problem still persisted.
What command did you run for the initial VNIC creation, and which  
error was returned specifically? If a VNIC creation fails, it should  
not leave left over entries in /etc/vnic.conf.

I''d like to go back to the exact sequence that leads to this state.  
One way to start from a sane state at this point is to move /etc/ 
vnic.conf aside _and_ reboot. After doing that, try creating a VNIC,  
and send me the full sequence of commands run and their status.
>> If you see an unexpected behavior, please let us know what that  
>> behavior is, but do not attempt to modify the back-end  
>> configuration directly to work-around it.
>> Did you try removing the existing VNIC with ''dladm
delete-vnic''
>> first?
>
> No, since I had never created a VNIC, at least not explicitly. Is  
> running ifconfig also creating a VNIC?
Plumbing a device does not automatically create a VNIC. So that VNIC  
was somehow created through an invocation of ''dladm
create-vnic''.
>>> A documentation NIT:
>>>
>>> dladm should reference the file /etc/vnic.conf if it uses it.
>
> I understand your points below. I was led to these files only  
> because of the error message that a file already exists.
That particular message comes from the strerror() for EEXIST, it  
indicates that the VNIC already existed. We are planning to move away  
from using errnos and have libdladm provide its own error space in  
the future.

Thanks,
Nicolas.

-- 
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux

> That particular message comes from the strerror() for EEXIST, it  
 > indicates that the VNIC already existed. We are planning to move away  
 > from using errnos and have libdladm provide its own error space in  
 > the future.

WiFi/GLDv3 added an error space for libdladm in build 54 (dladm_status_t)
-- so this should be easy to address.

--
meem

Hi Nicolas,

So now I can''t reproduce the situation with the /etc/vnic.conf file,
but here are some steps that do
lead to the repeatable failure situation.

Fresh install, and bfu, and reboot, then the following...I will reload with 49.

pinebarren# dladm show-vnic
pinebarren# tail /etc/vnic.conf
#
# Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)vnic.conf  1.1     06/10/31 SMI"
#
# DO NOT EDIT OR PARSE THIS FILE!
#
# Use the dladm(1m) command to change the contents of this file.

pinebarren# dladm show-dev
bge0            link: up        speed: 100   Mbps       duplex: full
bge1            link: unknown   speed: 0     Mbps       duplex: unknown
bge2            link: unknown   speed: 0     Mbps       duplex: unknown
bge3            link: unknown   speed: 0     Mbps       duplex: unknown
pinebarren# dladm show-link
bge0            type: non-vlan  mtu: 1500       device: bge0
bge1            type: non-vlan  mtu: 1500       device: bge1
bge2            type: non-vlan  mtu: 1500       device: bge2
bge3            type: non-vlan  mtu: 1500       device: bge3
pinebarren# dladm create-vnic -d bge3 -m 9:7:8:9:10:92 1
dladm: VNIC creation failed: Invalid argument      >>>>>> SEE
CONSOLE OUTPUT BELOW 16:57:52
pinebarren# tail /etc/vnic.conf
#
# Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)vnic.conf  1.1     06/10/31 SMI"
#
# DO NOT EDIT OR PARSE THIS FILE!
#
# Use the dladm(1m) command to change the contents of this file.

pinebarren# ifconfig bge3 plumb
pinebarren# dladm show-dev
bge0            link: up        speed: 100   Mbps       duplex: full
bge1            link: down      speed: 0     Mbps       duplex: unknown
bge2            link: down      speed: 0     Mbps       duplex: unknown
bge3            link: up        speed: 1000  Mbps       duplex: full
pinebarren# ifconfig bge3 10.1.14.146 netmask + broadcast + up
Setting netmask of bge3 to 255.255.255.192
pinebarren# tail /etc/vnic.conf
#
# Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)vnic.conf  1.1     06/10/31 SMI"
#
# DO NOT EDIT OR PARSE THIS FILE!
#
# Use the dladm(1m) command to change the contents of this file.

pinebarren# dladm show-vnic
pinebarren# dladm create-vnic -d bge3 -m 8:0:20:0:10:92 1

NOTE: System was hung at this point!!!


Read from remote host pinebarren: Connection timed out
Connection to pinebarren closed.


here is console output:

Dec 18 16:57:52 pinebarren vnic: WARNING: vnic_dev_create failed with 22
pinebarren# Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(0)
ill/drv/ring=600004617a8 30021dddb48
30022be0030
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(1) ill/drv/ring=600004617a8
30021dddbf8 30022be00d0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(2) ill/drv/ring=600004617a8
30021dddca8 30022be0170
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(3) ill/drv/ring=600004617a8
30021dddd58 30022be0210
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(4) ill/drv/ring=600004617a8
30021ddde08 30022be02b0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(5) ill/drv/ring=600004617a8
30021dddeb8 30022be0350
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(6) ill/drv/ring=600004617a8
30021dddf68 30022be03f0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(7) ill/drv/ring=600004617a8
30021dde018 30022be0490
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(8) ill/drv/ring=600004617a8
30021dde0c8 30022be0530
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(9) ill/drv/ring=600004617a8
30021dde178 30022be05d0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(10) ill/drv/ring=600004617a8
30021dde228 30022be0670
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(11) ill/drv/ring=600004617a8
30021dde2d8 30022be0710
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(12) ill/drv/ring=600004617a8
30021dde388 30022be07b0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(13) ill/drv/ring=600004617a8
30021dde438 30022be0850
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(14) ill/drv/ring=600004617a8
30021dde4e8 30022be08f0
Dec 18 17:36:12 pinebarren ip: bge3: ill_ring(15) ill/drv/ring=600004617a8
30021dde598 30022be0990
Dec 18 17:38:57 pinebarren ip: bge3: ill_ring(0) ill/drv/ring=600004617a8
30021dddb48 30022be0030
Debugging requested; hardware watchdog disabled; reboot to re-enable.

AS A RESULT OF <BREAK> CMD
Type  ''go'' to resume
{1} ok
{1} ok


Nicolas Droux wrote On 12/18/06 19:34,:
> Hi Steffen,
> 
> On Dec 18, 2006, at 2:09 PM, Steffen Weiberle wrote:
> 
>> Where would I find a snv_49 image? SX 11/06 is based on 50. I
don''t
>> see anything that old on nana. When will the 54 based resync be  
>> available? I''ll keep that build around.
> 
> 
> The resync is in progress, but it takes some time to perform due to  the 
> complexity of the changes involved. We''re hoping to have it done 
before
> the holiday break, or shortly after the break. I don''t wait to 
wait
> until then to look into the issue you reported though.
> 
>>
>>>> Thanks
>>>> Steffen
>>>>
>>>> First error:
>>>> # dladm create-vnic -d bge2 -m 9:7:8:9:10:8d 1
>>>> dladm: VNIC creation failed: File exists
>>>>
>>>> Where the file that exists is /etc/vnic.conf, which has the 
folling
>>>> in it:
>>>> ...
>>>> # Use the dladm(1m) command to change the contents of this
file.
>>>>
>>>> 1       bge3    auto    -       -       0
>>>>
>>>> I had plumbed bge3 manually, so I can understand why something
>>>> might be in there.
>>>
>>> Why do you think this is an error? You have an existing VNIC with  
>>> VNIC id 1, so you cannot create another VNIC with the same id. 
VNIC
>>> ids must be unique on a system-wide basis.
>>> What were the steps performed that lead you to that point?
>>
>>
>> ifconfig bge3 plumb x.y.x.t netmask + broadcast + up
>>
>> My first call to dladm create-vnic failed, and the I noticed this 
file.
> 
> 
> <snip>
> 
>> Sure. But I had this problem on the x64 system with e1000g on it  and 
>> the same issue. I did not move any files there, and I had an  error on 
>> my first attempt to explicitly create a VNIC. But I also  had done an 
>> ifconfig there first. After a reboot and no ifconfig,  the problem 
>> still persisted.
> 
> 
> What command did you run for the initial VNIC creation, and which  error 
> was returned specifically? If a VNIC creation fails, it should  not 
> leave left over entries in /etc/vnic.conf.
> 
> I''d like to go back to the exact sequence that leads to this
state.  One
> way to start from a sane state at this point is to move /etc/ vnic.conf 
> aside _and_ reboot. After doing that, try creating a VNIC,  and send me 
> the full sequence of commands run and their status.
> 
>>> If you see an unexpected behavior, please let us know what that  
>>> behavior is, but do not attempt to modify the back-end 
configuration
>>> directly to work-around it.
>>> Did you try removing the existing VNIC with ''dladm
delete-vnic''  first?
>>
>>
>> No, since I had never created a VNIC, at least not explicitly. Is  
>> running ifconfig also creating a VNIC?
> 
> 
> Plumbing a device does not automatically create a VNIC. So that VNIC  
> was somehow created through an invocation of ''dladm
create-vnic''.
> 
>>>> A documentation NIT:
>>>>
>>>> dladm should reference the file /etc/vnic.conf if it uses it.
>>
>>
>> I understand your points below. I was led to these files only  because 
>> of the error message that a file already exists.
> 
> 
> That particular message comes from the strerror() for EEXIST, it  
> indicates that the VNIC already existed. We are planning to move away  
> from using errnos and have libdladm provide its own error space in  the 
> future.
> 
> Thanks,
> Nicolas.
>

On Dec 19, 2006, at 5:56 AM, Steffen Weiberle wrote:
> So now I can''t reproduce the situation with the /etc/vnic.conf  
> file, but here are some steps that do lead to the repeatable  
> failure situation.
Hi Steffen,

Thanks for trying this again and for the detailed description. We  
could reproduce this regression. FYI, 6506500 (UDP bandwidth  
enforcement changes cause hang during VNIC creation) has been filed  
to track this issue, and will be fixed shortly.

Nicolas.

-- 
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux