Xen users - May 2007 - Multiple VMs - one static routable IP address

I have a new server and am looking into using XEN. Looks like everything I 
need is there, and gives me the security and power I''ve found
frustrating
or lacking in VMWare... I have one problem tho... I want to have each VM 
be able to completely host it''s own domain name and it''s own
services. I
only have one routable IP address however. I need to service ports: 22, 
25, 80, 110, and 443

I have no worldly idea how to route the packets to each respective VM when 
everybody is trying to use the same ports, and are all sitting behind the 
firewall effectively running non-routable IP addresses.

Example:
Domain-0 will host mail services on port 25
Domain-1 will also...
Domain-2 as well
Domain-3 again...

For port 80 and 443 traffic, I suppose I could always just stick apache in 
proxy mode and route to the respective VM running on the 10-net behind the 
firewall, and I can run SSHD on a non-standard port for each one I suppose 
as well... but with everybody running their own sendmail and needing to 
receive mail (sending shouldn''t be a problem in this configuration near
as
I can tell anyway) everybody pig piles in on port 25 and only Domain-0 
wins. I could hack up some sort of MTA forwarding I suppose, but there HAS 
to be some way that this is done such that I don''t have to special case
every port. I just don''t know what that is it seems.

So, when a packet comes in to the DHCP as asks for the IP address for one 
(of seven) of the domain names I host. They will all resolve to the same 
single routable IP address I have. I suspect I need some sort of 
soft-router running on Domain 0 to see if the packet is destined for one 
of the VMs and if so route to the 10.0.0.X address accordingly.

I''m completely new to bridging, tho am by no means new to Linux systems
administration. All VMs, including Domain-0, are running CentOS 5.  My 
_guess_ is I need to get a fourth  DomU installed and running and have 
THAT run the routable IP address, and then forward all traffic into 
Domain-0, or the respective other VM domains, switching based on the 
domain name the traffic is trying to go to...  I wouldn''t be able to 
connect to any of the VMs but that forward facing one by using the IP 
address, but then, I seldom to never do that anyway...  plus that way I 
could hide Domain-0 a bit better anyway.  I just have no idea what bits 
I''d need to install and run to get that working.

Any help pointing me in the right direction would be greatly appreciated. 
This can''t be a unique problem... I''m sure the S390 guys are
running
thousands of VMs per system, and suspect there is no way all of those VMs 
are also running routable IP addresses. I just don''t know how to get
the
packets to the right VM when I''ve only got the one externally routable
IP
address.

Help Help Help!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 16, 2007 at 09:01:43AM -0500, xensource@midnightfantasy.com
wrote:
> I have a new server and am looking into using XEN. Looks like everything I 
> need is there, and gives me the security and power I''ve found
frustrating
> or lacking in VMWare... I have one problem tho... I want to have each VM 
> be able to completely host it''s own domain name and it''s
own services. I
> only have one routable IP address however. I need to service ports: 22, 
> 25, 80, 110, and 443
your going to have problems with 22, 110 and 443. You can potentially do it for 
port 80, but yuo would have to service the request on the host.  THis is going 
to be the same for all the virtual machines if you have non routable addresses, 
no real way around it.  You could possible try ipv6 - but then your client 
would have to use ipv6 (both of you can use the ipv4 in ipv6 ability)
> 
> I have no worldly idea how to route the packets to each respective VM when 
> everybody is trying to use the same ports, and are all sitting behind the 
> firewall effectively running non-routable IP addresses.
> 
> Example:
> Domain-0 will host mail services on port 25
> Domain-1 will also...
> Domain-2 as well
> Domain-3 again...
> 
> For port 80 and 443 traffic, I suppose I could always just stick apache in 
> proxy mode and route to the respective VM running on the 10-net behind the 
> firewall, and I can run SSHD on a non-standard port for each one I suppose 
> as well... but with everybody running their own sendmail and needing to 
> receive mail (sending shouldn''t be a problem in this configuration
near as
> I can tell anyway) everybody pig piles in on port 25 and only Domain-0 
> wins. I could hack up some sort of MTA forwarding I suppose, but there HAS 
> to be some way that this is done such that I don''t have to special
case
> every port. I just don''t know what that is it seems.
> 
> So, when a packet comes in to the DHCP as asks for the IP address for one 
> (of seven) of the domain names I host. They will all resolve to the same 
> single routable IP address I have. I suspect I need some sort of 
> soft-router running on Domain 0 to see if the packet is destined for one 
> of the VMs and if so route to the 10.0.0.X address accordingly.
> 
> I''m completely new to bridging, tho am by no means new to Linux
systems
> administration. All VMs, including Domain-0, are running CentOS 5.  My 
> _guess_ is I need to get a fourth  DomU installed and running and have 
> THAT run the routable IP address, and then forward all traffic into 
> Domain-0, or the respective other VM domains, switching based on the 
> domain name the traffic is trying to go to...  I wouldn''t be able
to
> connect to any of the VMs but that forward facing one by using the IP 
> address, but then, I seldom to never do that anyway...  plus that way I 
> could hide Domain-0 a bit better anyway.  I just have no idea what bits 
> I''d need to install and run to get that working.
> 
> Any help pointing me in the right direction would be greatly appreciated. 
> This can''t be a unique problem... I''m sure the S390 guys
are running
> thousands of VMs per system, and suspect there is no way all of those VMs 
> are also running routable IP addresses. I just don''t know how to
get the
> packets to the right VM when I''ve only got the one externally
routable IP
> address.
> 
> Help Help Help!
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>>only have one routable IP address however. I need to service ports: 22, 
>>25, 80, 110, and 443
>your going to have problems with 22, 110 and 443. You can potentially do 
>it for port 80, but yuo would have to service the request on the host. 
>THis is going to be the same for all the virtual machines if you have non 
>routable addresses, no real way around it.  You could possible try ipv6 - 
>but then your client would have to use ipv6 (both of you can use the ipv4 
>in ipv6 ability)
Thanks for the reply Alex!

Ports 80 and 443 I''m not terribly worried about.  Apache in proxy mode 
gets around that simple enough.  It''d mean an additional install of 
Apache, but that''s not a terribly big deal nor a deal breaker for me.

Well, I''m honestly not familiar enough with ipv6 to know how to do 
anything differently.  I''m no stranger to tcp/ip stacks, but I
haven''t
even dabbed a toe in the ipv6 pool.

How does the S390 hosting guys do this sort of thing?  They can''t
really
be using routable IP addresses for everything?  I realize this is more a 
networking question than a VM question, but I figured there would be some 
sort of soft router type functionality built into the solution (just like 
there is for the bridging and such) to address the complication of it now 
being multiple machines.  I can''t be the only guy who does hosting on a
business class DSL line, but with only one routable IP.

Maybe the solution is to spin up a DomU as the firewall and put the apache 
in proxy mode there, as well as a sendmail MTA router to the 10-net behind 
it.  Ports 22 (sshd) and 110 (ipop3) are easy enough to configure around 
and just give a different port to every VM.  The only real sticking point 
was port 25 really.  My sendmail kung-fu just isn''t that strong for a 
multiple machine environment.  Everything I''ve ever done is with one 
server, and multiple backup MX''s.

I just keep coming back to the original question tho, what do the big VM 
environments do when they have hundreds or more VM''s...  are they
really
using up hundreds of routable IP addresses?  Really?

Originally I was planning on putting all my own personal websites and 
email on Domain-0, as well as an iptables based firewall.  Having read 
more, seems like the recommendation is to keep Domain-0 behind a DomU 
where the firewall runs.  Makes sense, and doesn''t seem difficult to
do...
just a new paradigm for me.  I''ve always only had one server, and it
did
everything and anything.  I love the idea of breaking it all up from a 
security and manageability standpoint...  just not sure what to do about 
getting all the bits to the right VMs that need to be routed correctly.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, May 17, 2007 at 10:24:24PM -0500, cyber@Wamal.Com
wrote:
> >>only have one routable IP address however. I need to service ports:
22,
> >>25, 80, 110, and 443
> >your going to have problems with 22, 110 and 443. You can potentially
do
> >it for port 80, but yuo would have to service the request on the host. 
> >THis is going to be the same for all the virtual machines if you have
non
> >routable addresses, no real way around it.  You could possible try ipv6
-
> >but then your client would have to use ipv6 (both of you can use the
ipv4
> >in ipv6 ability)
> 
> Thanks for the reply Alex!
> 
> Ports 80 and 443 I''m not terribly worried about.  Apache in proxy
mode
> gets around that simple enough.  It''d mean an additional install
of
> Apache, but that''s not a terribly big deal nor a deal breaker for
me.
> 
> Well, I''m honestly not familiar enough with ipv6 to know how to do
> anything differently.  I''m no stranger to tcp/ip stacks, but I
haven''t
> even dabbed a toe in the ipv6 pool.
> 
> How does the S390 hosting guys do this sort of thing?  They can''t
really
> be using routable IP addresses for everything?  I realize this is more a 
> networking question than a VM question, but I figured there would be some 
> sort of soft router type functionality built into the solution (just like 
> there is for the bridging and such) to address the complication of it now 
> being multiple machines.  I can''t be the only guy who does hosting
on a
> business class DSL line, but with only one routable IP.
> 
> Maybe the solution is to spin up a DomU as the firewall and put the apache 
> in proxy mode there, as well as a sendmail MTA router to the 10-net behind 
> it.  Ports 22 (sshd) and 110 (ipop3) are easy enough to configure around 
> and just give a different port to every VM.  The only real sticking point 
> was port 25 really.  My sendmail kung-fu just isn''t that strong
for a
> multiple machine environment.  Everything I''ve ever done is with
one
> server, and multiple backup MX''s.
> 
> I just keep coming back to the original question tho, what do the big VM 
> environments do when they have hundreds or more VM''s...  are they
really
> using up hundreds of routable IP addresses?  Really?
most of the uml that i have seen use routeable.

the problem with 22 and 443 is they are encryption and authentication.  how do 
you determine which 22 is the destination, having said that you could assign 
port 23 to one machine for ssh (it doesn''t need to be stuck to port
22).

the mta on the firewall machine - you could as well, but it would leave a trail 
(add its foot print to email travelling through it)
> 
> Originally I was planning on putting all my own personal websites and 
> email on Domain-0, as well as an iptables based firewall.  Having read 
> more, seems like the recommendation is to keep Domain-0 behind a DomU 
> where the firewall runs.  Makes sense, and doesn''t seem difficult
to do...
> just a new paradigm for me.  I''ve always only had one server, and
it did
> everything and anything.  I love the idea of breaking it all up from a 
> security and manageability standpoint...  just not sure what to do about 
> getting all the bits to the right VMs that need to be routed correctly.
if its for security of apps, why not look at chroot ?

> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Alex Samad wrote:
> On Thu, May 17, 2007 at 10:24:24PM -0500, cyber@Wamal.Com wrote:
>
>   
>> Originally I was planning on putting all my own personal websites and 
>> email on Domain-0, as well as an iptables based firewall.  Having read 
>> more, seems like the recommendation is to keep Domain-0 behind a DomU 
>> where the firewall runs.  Makes sense, and doesn''t seem
difficult to do...
>> just a new paradigm for me.  I''ve always only had one server,
and it did
>> everything and anything.  I love the idea of breaking it all up from a 
>> security and manageability standpoint...  just not sure what to do
about
>> getting all the bits to the right VMs that need to be routed correctly.
>>     
>
> if its for security of apps, why not look at chroot ?
>
>   
chroot for OpenSSH has never been well-supported. (I used to be the 
maintainer of that add-on functionality, and it remains rejected by the 
core authors to this day, much to my lament.) WebDAV over HTTPS works 
well for upload/download sites, and avoids the shell access and local 
account problems of SSH.

I''m not a believer in external, hardware firewalls, to avoid the 
complexities and difficulties of maintaining my own software ones.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, May 17, 2007 at 10:24:24PM -0500, cyber@Wamal.Com
wrote:
> I just keep coming back to the original question tho, what do the big VM 
> environments do when they have hundreds or more VM''s...  are they
really
> using up hundreds of routable IP addresses?  Really?
It is best practice to use internal, non-internet-routed IP space
for as much as possible, regardless of virtualisation.

For example, you will have an incredibly hard time trying to pass a
PCI DSS audit when your app servers are on publically routed IP
space, no matter what firewalls you have.

Also what do you consider to be a "big VM environment"?  An
enterprise will tend to have fewer, larger VMs with each one
dedicated to a specific task.  Almost all of that will be on private
IPs.  By contrast a VM hosting company will have vast numbers of
small VMs which will be like mini personal servers often doing
multiple tasks, and they''ll all be on publically routed IPs.  Which
would you say is bigger?

Cheers,
Andy


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I finally have some time over the long weekend to work on my new server. 
I''m trying to put together a DomU as the firewall, which should take
eth0
and eth1.  The intent is to have eth0 connected to the DSL line, and run 
my sole routable IP address.  I will also host named, apache, and sendmail 
on the firewall DomU and route web traffic to the appropriate DomU''s
via
apache reverse proxy configurations.  I''ll create a sendmail MTA config
to
route mail to the correct DomU''s as well.  When I bring up Domain-0 my 
understanding is that the physical eth adapters will be remapped to peth0 
and peth1, and a bridge added for each.  My hope is to have the firewall 
control peth0 and peth1 and via iptables create a pretty straight forward 
firewall.  ETH1 will be plugged into a hub which the rest of the physical 
machines on my home network will connect, and what I hope to have the rest 
of the DomUs and Domain-0 connect to.

So, anything in particular I would need to do so that Domain-0 doesn''t
get
ETH0 and so that the firewall DomU gets both eth0 and eth1?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users