Xen users - May 2009 - can one do traffic accounting on bridged domU''s?

Hi,
I''m just wondering, is it possible to setup the dom0 to perform traffic
account of domU''s, which are setup using bridge mode networking?

-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, May 16, 2009 at 4:51 AM, Rudi Ahlers <rudiahlers@gmail.com>
wrote:
> Hi,
> I''m just wondering, is it possible to setup the dom0 to perform
traffic
> account of domU''s, which are setup using bridge mode networking?
Possible, but not easy. You can name domU interfaces so they have
persistent naming. Once you have that, snmp should work.

One gotcha is that you probably need to restart snmpd after domU
startup/shutdown.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,
I found very useful pmacct tool http://www.pmacct.net/, which is able to log
traffic into database. The collected data may be postprocessed to produce
statistics/plots etc.
-Maciej
http://statnet.pl

On Fri, May 15, 2009 at 11:51 PM, Rudi Ahlers <rudiahlers@gmail.com>
wrote:
> Hi,
> I''m just wondering, is it possible to setup the dom0 to perform
traffic
> account of domU''s, which are setup using bridge mode networking?
>
> --
> Kind Regards
> Rudi Ahlers
> CEO, SoftDux Hosting
> Web: http://www.SoftDux.com
> Office: 087 805 9573
> Cell: 082 554 7532
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Rudi Ahlers wrote:
>I''m just wondering, is it possible to setup the dom0 to perform 
>traffic account of domU''s, which are setup using bridge mode 
>networking?
Depends on what you want to do.

If logging traffic in/out of the real interfaces by IP address is 
enough, then that''s reasonably easy to do - though it won''t
log
traffic between Dom0 and DomU or between DomU''s on the same box. 
Doing it strictly by virtual interface will be a bit more work as the 
list of interfaces will keep changing.

I covered how I do traffic accounting at my border routing in a 
thread titled "Limiting domU network?" on 2nd May. The same approach 
should work on the real interface(s) of a Dom0 - provided "by IP 
address" is accurate enough.
-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Simon Hobson wrote:
> Rudi Ahlers wrote:
> 
>> I''m just wondering, is it possible to setup the dom0 to
perform
>> traffic account of domU''s, which are setup using bridge mode
networking?
> 
> Depends on what you want to do.
> 
> If logging traffic in/out of the real interfaces by IP address is
> enough, then that''s reasonably easy to do - though it
won''t log traffic
> between Dom0 and DomU or between DomU''s on the same box. Doing it
> strictly by virtual interface will be a bit more work as the list of
> interfaces will keep changing.
Well, that''s not so hard to perform. Here''s a quick example,
taken from
our stable version of dtc-xen:

http://git.gplhost.com/gitweb/?p=dtc-xen.git;a=blob;f=panel/graph/xenupdate.py;h=c1eb90f49871a66817cc841155b43f69b993021b;hb=f78e9f8f48920b1ceb39d3ad4b68f3cb5f015633

This sends CPU, IO and Network usage to some RRDs. Here''s the graphing
for the network part:

http://git.gplhost.com/gitweb/?p=dtc-xen.git;a=blob;f=panel/graph/xennetgraph.py;h=53a31be86956fd0f538ead6948988d633b53f00b;hb=f78e9f8f48920b1ceb39d3ad4b68f3cb5f015633

These 2 scripts are of course to set in a cron job. Note that on our
system, all domU are called xenXX, feel free to adapt to your needs.

On our next version of dtc-xen, it saves the values, that are fetched
using SOAP, and then stored on our management server. That''s of course
harder to read / use right away.

Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, May 16, 2009 at 3:37 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:
> On Sat, May 16, 2009 at 4:51 AM, Rudi Ahlers <rudiahlers@gmail.com>
wrote:
> > Hi,
> > I''m just wondering, is it possible to setup the dom0 to
perform traffic
> > account of domU''s, which are setup using bridge mode
networking?
>
> Possible, but not easy. You can name domU interfaces so they have
> persistent naming. Once you have that, snmp should work.
>
> One gotcha is that you probably need to restart snmpd after domU
> startup/shutdown.
>
> Regards,
>
> Fajar
>
Hi Fajar,

This is how I have it currently. Each domU''s interface name is
"hard coded"
in it''s config file, as follows:

vif        = [''ip=1.2.3.4, vifname=vifrudi0,  mac=aa:00:bf:5b:79:01
'']

But the problem I have is that snmp doesn''t always work as expected,
and I''m
thinking of a different approach, perhaps with iptables?


-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Rudi Ahlers wrote:
> Hi Fajar,
> 
> This is how I have it currently. Each domU''s interface name is
"hard
> coded" in it''s config file, as follows:
> 
> vif        = [''ip=1.2.3.4, vifname=vifrudi0, 
mac=aa:00:bf:5b:79:01 '']
> 
> But the problem I have is that snmp doesn''t always work as
expected, and
> I''m thinking of a different approach, perhaps with iptables?
Hi,

As I wrote to you on my previous email, the solution is the byte counter
in /proc (if that was not obvious reading the python scripts).

Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, May 16, 2009 at 11:38 AM, Thomas Goirand <thomas@goirand.fr>
wrote:
> Rudi Ahlers wrote:
> > Hi Fajar,
> >
> > This is how I have it currently. Each domU''s interface name
is "hard
> > coded" in it''s config file, as follows:
> >
> > vif        = [''ip=1.2.3.4, vifname=vifrudi0, 
mac=aa:00:bf:5b:79:01 '']
> >
> > But the problem I have is that snmp doesn''t always work as
expected, and
> > I''m thinking of a different approach, perhaps with iptables?
>
> Hi,
>
> As I wrote to you on my previous email, the solution is the byte counter
> in /proc (if that was not obvious reading the python scripts).
>
> Thomas
>
> _______________________________________________
>
Hi Thomas,

I had a look at your scripts, but it''s doesn''t make any sense
to me. I''m not
a Python programmer, and don''t intend on learning it either.

Thank you for your help though


-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Rudi Ahlers wrote:
>> Hi Fajar,
>>
>> This is how I have it currently. Each domU''s interface name is
"hard
>> coded" in it''s config file, as follows:
>>
>> vif        = [''ip=1.2.3.4, vifname=vifrudi0,
 mac=aa:00:bf:5b:79:01 '']
>>
>> But the problem I have is that snmp doesn''t always work as
expected, and
>> I''m thinking of a different approach, perhaps with iptables?
Did you restart snmpd after domU startup/shutdown? If you didn''t, I
believe snmpd would end up listing two (or more) interface with  the
same name.

There''s also this, which parses xentop''s output into something
suitable for snmpd (and later, cacti) :
http://run.tournament.org.il/xen-vms-performance-collection/comment-page-1/
The posted xen_stats.pl don''t work though, probably copy-paste error
or mangled by the CMS. Here''s my version which should work for Xen 3.1
(tested on RHEL''s Xen) and Xen 3.3.1 : http://pastebin.com/f1d292654

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi
> I''m just wondering, is it possible to setup the dom0 to perform 
> traffic account of domU''s, which are setup using bridge mode
networking?
i would (and do) use pmacct ( http://www.pmacct.net ). Please read the 
FAQ Q5 is quite useful.

Fabian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 17, 2009 at 3:22 PM, Fabian Zimmermann <
xen-users@z-technologies.de> wrote:
> Hi
>
>> I''m just wondering, is it possible to setup the dom0 to
perform traffic
>> account of domU''s, which are setup using bridge mode
networking?
>>
> i would (and do) use pmacct ( http://www.pmacct.net ). Please read the FAQ
> Q5 is quite useful.
>
> Fabian
>
>  _______________________________________________
>
Thanx Fabian,

I''ll check it out. Does your pmacct run totally independant of your XEN
VM''s, or do you need to install extra software on the VM''s?



-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,
You need to install pmmact only on Dom0, the DomUs do not require any
additional software.
-Maciej
http://statnet.pl

On Mon, May 18, 2009 at 9:24 AM, Rudi Ahlers <rudiahlers@gmail.com> wrote:
>
>
> On Sun, May 17, 2009 at 3:22 PM, Fabian Zimmermann <
> xen-users@z-technologies.de> wrote:
>
>> Hi
>>
>>> I''m just wondering, is it possible to setup the dom0 to
perform traffic
>>> account of domU''s, which are setup using bridge mode
networking?
>>>
>> i would (and do) use pmacct ( http://www.pmacct.net ). Please read the
>> FAQ Q5 is quite useful.
>>
>> Fabian
>>
>>  _______________________________________________
>>
>
> Thanx Fabian,
>
> I''ll check it out. Does your pmacct run totally independant of
your XEN
> VM''s, or do you need to install extra software on the
VM''s?
>
>
>
> --
> Kind Regards
> Rudi Ahlers
> CEO, SoftDux Hosting
> Web: http://www.SoftDux.com
> Office: 087 805 9573
> Cell: 082 554 7532
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

You need to install pmacct only on Dom0, the DomUs do not require any
additional software.

-Maciej
http://statnet.pl


On Mon, May 18, 2009 at 9:24 AM, Rudi Ahlers <rudiahlers@gmail.com> wrote:
>
>
> On Sun, May 17, 2009 at 3:22 PM, Fabian Zimmermann <
> xen-users@z-technologies.de> wrote:
>
>> Hi
>>
>>> I''m just wondering, is it possible to setup the dom0 to
perform traffic
>>> account of domU''s, which are setup using bridge mode
networking?
>>>
>> i would (and do) use pmacct ( http://www.pmacct.net ). Please read the
>> FAQ Q5 is quite useful.
>>
>> Fabian
>>
>>  _______________________________________________
>>
>
> Thanx Fabian,
>
> I''ll check it out. Does your pmacct run totally independant of
your XEN
> VM''s, or do you need to install extra software on the
VM''s?
>
>
>
> --
> Kind Regards
> Rudi Ahlers
> CEO, SoftDux Hosting
> Web: http://www.SoftDux.com
> Office: 087 805 9573
> Cell: 082 554 7532
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

You need to install pmacct only on Dom0, the DomUs do not require any
additional software.

-Maciej
http://statnet.pl


On Mon, May 18, 2009 at 9:24 AM, Rudi Ahlers <rudiahlers@gmail.com> wrote:
>
>
> On Sun, May 17, 2009 at 3:22 PM, Fabian Zimmermann <
> xen-users@z-technologies.de> wrote:
>
>> Hi
>>
>>> I''m just wondering, is it possible to setup the dom0 to
perform traffic
>>> account of domU''s, which are setup using bridge mode
networking?
>>>
>> i would (and do) use pmacct ( http://www.pmacct.net ). Please read the
>> FAQ Q5 is quite useful.
>>
>> Fabian
>>
>>  _______________________________________________
>>
>
> Thanx Fabian,
>
> I''ll check it out. Does your pmacct run totally independant of
your XEN
> VM''s, or do you need to install extra software on the
VM''s?
>
>
>
> --
> Kind Regards
> Rudi Ahlers
> CEO, SoftDux Hosting
> Web: http://www.SoftDux.com
> Office: 087 805 9573
> Cell: 082 554 7532
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 17, 2009 at 3:22 PM, Fabian Zimmermann <
xen-users@z-technologies.de> wrote:
> Hi
>
>> I''m just wondering, is it possible to setup the dom0 to
perform traffic
>> account of domU''s, which are setup using bridge mode
networking?
>>
> i would (and do) use pmacct ( http://www.pmacct.net ). Please read the FAQ
> Q5 is quite useful.
>
> Fabian
>
>  _______________________________________________
>
Fabian, on what OS did you install pmacct? Seeing as there''e no rpm,
and I''m
using CentOS, I''m a bit precautious about installing it.

-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Rudi Ahlers,
> Fabian, on what OS did you install pmacct? Seeing as there''e no
rpm,
> and I''m using CentOS, I''m a bit precautious about
installing it.
Debian Etch. I compiled it from source.

If you followed my advice and read FAQ5 you may also compile 
libpcap-mmap from source.

Just to support you a bit:

pmacct.conf
--
daemonize: true
pidfile: /var/run/pmacctd.pid
syslog: daemon
aggregate[out]: src_host
aggregate[in]: dst_host
aggregate_filter[out]: src net 212.X.Y.0/24 or src net 212.X.Y.0/24
aggregate_filter[in]: dst net 212.X.Y.0/24 or dst net 212.X.Y.0/24
pcap_filter: net 212.X.Y.0/24 or net 212.X.Y.0/24
interface: eth1
plugins: mysql[in], mysql[out]
sql_host: 192.168.100.X
sql_passwd: x
sql_refresh_time: 300
sql_optimize_clauses: true
sql_history: 5m
sql_dont_try_update: true
sql_history_roundoff: m
sql_recovery_logfile[in]: /var/lib/pmacct/recovery_log_in
sql_recovery_logfile[out]: /var/lib/pmacct/recovery_log_out

with libpcap-mmap installed I start pmacct with:

PCAP_MEMORY=10000 PCAP_VERBOSE=1 PCAP_TO_MS=0 pmacctd -f pmacctd.conf

Cu,
Fabian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 20, 2009 at 5:20 AM, Rudi Ahlers <rudiahlers@gmail.com>
wrote:
> Fabian, on what OS did you install pmacct? Seeing as there''e no
rpm, and I''m
> using CentOS, I''m a bit precautious about installing it.
If you want something with as little installation as possible, and you
already give your vifs unique name, a simple change to vif-bridge
script should work for snmp. Here''s mine (using Xen 3.3.1''s
vif-bridge
as base) : http://pastebin.com/f7ef8a39f
The only change required is two lines that restarts snmpd (there
should be a better way to do this, but it works). I put the script on
/etc/xen/scripts/vif-bridge-snmp

The vif line in my domU config look like this
vif =	[
	''mac=00:16:3E:F3:AE:C0, bridge=br102, script=vif-bridge-snmp,
vifname=rad-eth0'',
	]

With that, anything that can get snmp data using interface name (e.g.
cacti) should work.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users