On Thursday 27 Mar 2003 09:24, Luman wrote:
> >Assumptions:
> >  Determine and mark 'good traffic' -- i.e. smtp, ftp, http, ssh, etc.=
,
> >  everything which uses well known ports.  Probably most people do it
> >  anyway, at least to some level.
>
> The problem is with that currently P2P soft often use these well known
> ports too. So the assumption that port 80 is only for HTTP is bad.

Yes, but you can then analyze the traffic just on those ports for
verification. HTTP patkets have a certain anatomy to them. This can be
detected. Same goes for FTP, SMTP, etc. The problems start with HTTPS, SS=
H,
IMAPS, POP3S, etc, as they are encrypted, and therefore cannot be analyze=
d.
With those, you simply have a problem. However, for SSH, IMAPS and POP3S =
you
don't need lots of bandwidth. You could therefore throttle them to
low-latency low-bandwidth. P2P networks will not like this.

SFTP runs over SSH so you may have a problem with that. HTTPS is also
problematic.

However, you can scan to verify if SSH/HTTPS is being used. You can simpl=
y
write a bot that scans the ports when your router detects traffic. It can
send valid SSH/HTTPS connection requests and see if it talks back as
SSH/HTTPS should.

Unfortunately, it gets progressively more difficult when P2P clients lear=
n to
masquerade as the real protocols, and there is at least one P2P applicati=
on
out there that can operate over SMTP, sending valid requests. :-(

I hope you are prepared to accept that eventually it all comes down to a
battle of wills between the sysadmins writing filters and the P2P develop=
ers
finding more ways to outsmart the filters.

> The intention of the bringing forward my problem is to open wider
> discussion aimed to find or if need be to build a "tool" (it
might be a
> kernel patch, or whatever), which can thoroughly analyze traffic and it=
s
> content, and on the base of it can take a decision (likely not with 100=
%
> likelihood) what is the content. For instance it can detect that the
> traffic is HTTP even if it is sent to 46723 port, basing on the content
> of data.

How do you deal with HTTPS/SFTP/SSH/IMAPS/POP3S? Automatically do a man i=
n
 the middle attack on everything, at your router?

> Such tool should based on a modular architecture allowing
> adding new testers or new knowledge trying to guess the protocol.

This can generally be done for unencrypted connections, but once things s=
tart
to run over SSL (some already do), the chances of "recognizing"
traffic v=
ery
soon become adjusted to zero. :-(

> Obviously, it should track connections, session and everything what can
> be used to traffic classification.

In order to write a rule for traffic analysis, you must first know what t=
o
look for. If you can come up with a method to analyze SSL traffic, especi=
ally
in real-time (or close to), I am sure a lot of people in the security
industry will want to hear from you.

> As the result packets would be marked by a standardized number
> determining type of a protocol, for instance HTTP, KaZaa, MSN, SSH, SCP
> etc.

If you can get as far as distinguishing packets, then that's great. How t=
o
 get that far is the difficult part.

> This knowledge could be used to traffic shaping and whatever. Can
> you imaging the comfort of administrators' work if at the border router=
,
> or at the firewall configuration, can work with this well determined
> content. Number or rules would be reduced dramatically. Obviously, the
> classification knowledge would be growing day by day.

Sounds great, in theory.

> Whole idea is very similar to Unix 'file' command. For instance I had o=
n
> my system "a.gz" file. The type of this file is obvious this is
gzip.
> However, I changed its name to "a.txt". It should suggest that
this is
> text file, however, when I run file a.txt I get the fallowing answer:
> ~# file a.txt
> a.txt: gzip compressed data, deflated, original filename,
> `ucspi-tcp-0.88.tar', last modified: Sat Mar 18 16:21:39 2000, max
> compression, os: Unix.
> This program doesn't care about extensions it tries to guess the type b=
y
> analyzing content. Of course many times it gives wrong answer, but that
> is related to weak of knowledge.

Your knowledge of encrypted traffic is non-existant. That is the whole po=
int
of encryption. How do you disallow valid encrypted traffic while allowing=
 P2P
encrypted traffic? What happens when the method of using SMTP for P2P bec=
omes
more wide spread? You can send perfectly valid looking emails - that are =
PGP
encrypted, with all the SMTP headers in place to make them indistinguisha=
ble
from real PGP encrypted email.

> Summarizing my pretty long mail, I think our present methods are simila=
r
> to determining the content of file basing only on extension of its name=
=2E
> But I believe we strongly require something more.

And that will work until encryption is used. As Most P2P networks are now
starting to use encryption on the connectiong streams, this very quickly
becomes unworkable.

> >  All what is left are P2P connections and some other misc connections=
=2E
> >  A bit unfair for other protocol using non-standard ports, like
> >  Instant  Messenger style-software, and lots of other stuff.  So here=
 we
> >  introduce
> >  a trick.  IMs and other low bandwidth traffic will use small packets
> >  ( < 512 or even < 256), P2P will use maximum MTA available
(usually
> >  1500, but I've seen some using 576 packets, hence i treat > 512 as
> > P2P).
> >
> >  Probably you've notices that I mention round numbers, as 512 or 1024=
,
> >  that's because I use u32 for marking packets.  How I do it, we leave
> >  as an exercise to the reader. ;-)))
>
> I like your solution very much. I'll try to apply it for my system, as =
a
> temporary solution.

That sounds like an interesting idea, provided you have some real evidenc=
e of
this being the case. And this will only work until P2P network software
starts to randomly change packet sizes to obfuscate itself. :-(

But, I guess we have to work with what we have now, and not worry about t=
he
future advancements before they happen. :-)

I hope you will all forgive me for being... restrained (for want of bette=
r
word) in my expectations of the success of such network traffic analysis.=
 It
is a depressing subject to talk about. :-(

I cannot help but think that this is also starting to get slightly off-to=
pic
for this mailing list...

Gordan

-------------------------------------------------------

Hi,

> -----Original Message-----
> From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]On
> Behalf Of Gordan Bobic
> Sent: Thursday, March 27, 2003 11:17 AM
> To: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Intelligent P2P detection
>

> Unfortunately, it gets progressively more difficult when P2P
> clients learn to
> masquerade as the real protocols, and there is at least one P2P
> application
> out there that can operate over SMTP, sending valid requests. :-(
>
The everlasting battle between creators of swords and shields:) If p2p apps
start to mimick as other protocols and use encription then content based
classificators are of no use.


> I hope you are prepared to accept that eventually it all comes down to a
> battle of wills between the sysadmins writing filters and the P2P
> developers
> finding more ways to outsmart the filters.
Correct. Anyway there are some other solutions.
> That sounds like an interesting idea, provided you have some real
> evidence of
> this being the case. And this will only work until P2P network software
> starts to randomly change packet sizes to obfuscate itself. :-(
I was told that applications doing it exists. I haven't checked it, though.
>
> But, I guess we have to work with what we have now, and not worry
> about the
> future advancements before they happen. :-)
Hehe... yes doing something instead of just talking is a good idea:)
>
> I hope you will all forgive me for being... restrained (for want of better
> word) in my expectations of the success of such network traffic
> analysis. It
> is a depressing subject to talk about. :-(
I think this e-mail is a nice summary. I enjoyed reading it. I could say
that I agree your opinions.

>
> I cannot help but think that this is also starting to get
> slightly off-topic
> for this mailing list...
I think opposite:)

Maybe creating free alternatives to shaping software like those from
www.dyband.com is a way. People using it are very happy actually. They adapt
to network utilization, allow extensive logging, setting different
parameters like max bandwidth, ramps, minimum acceptable rate. The main idea
is to limit aggresive users and give maximum performance and quality
(latency, jitter throughput etc.) to standard users. It looks very well on
paper but I haven't tried dyband yet.... Maybe there is other software like
this I am not aware of.

RK

On Thursday 27 Mar 2003 15:32, Robert Krycza=B3o wrote:

> > Unfortunately, it gets progressively more difficult when P2P
> > clients learn to
> > masquerade as the real protocols, and there is at least one P2P
> > application
> > out there that can operate over SMTP, sending valid requests. :-(
>
> The everlasting battle between creators of swords and shields:) If p2p =
apps
> start to mimick as other protocols and use encription then content base=
d
> classificators are of no use.

Yup. And this is happening right now...

> > That sounds like an interesting idea, provided you have some real
> > evidence of
> > this being the case. And this will only work until P2P network softwa=
re
> > starts to randomly change packet sizes to obfuscate itself. :-(
>
> I was told that applications doing it exists. I haven't checked it, tho=
ugh.

I haven't heard of an application that does it, but I have always felt=20
reasonably sure that it has either already happened or is about to be=20
implemented...

> > But, I guess we have to work with what we have now, and not worry
> > about the
> > future advancements before they happen. :-)
>
> Hehe... yes doing something instead of just talking is a good idea:)

Well, for a little while, anyway, until the new version of the client com=
es=20
out...

> > I hope you will all forgive me for being... restrained (for want of
> > better word) in my expectations of the success of such network traffi=
c
> > analysis. It
> > is a depressing subject to talk about. :-(
>
> I think this e-mail is a nice summary. I enjoyed reading it. I could sa=
y
> that I agree your opinions.

Thank you. :-)

> Maybe creating free alternatives to shaping software like those from
> www.dyband.com is a way. People using it are very happy actually. They
> adapt to network utilization, allow extensive logging, setting differen=
t
> parameters like max bandwidth, ramps, minimum acceptable rate. The main
> idea is to limit aggresive users and give maximum performance and quali=
ty
> (latency, jitter throughput etc.) to standard users. It looks very well=
 on
> paper but I haven't tried dyband yet.... Maybe there is other software =
like
> this I am not aware of.

I haven't heard about any of them. I am a great believer in "home
brewed"=
=20
solutions. :-)

The problem you start getting there is that monitoring and shaping traffi=
c on=20
a 100 Mb pipe will take a huge amount of CPU power, and even that will on=
ly=20
work if the traffic is not encrypted. The only way of attacking the probl=
em I=20
can think of is by actually attempting to connect to the client machine o=
n=20
the suspiciously used well known ports, and seeing if it works. If it doe=
sn't=20
work as expected, you know it's likely to be a P2P application.

I am not sure if you really want to do that, though, as it involves activ=
e=20
port scanning rather than just monitoring, and some of your customers may=
=20
complain...

Gordan

Hi,
> > > That sounds like an interesting idea, provided you have some real
> > > evidence of
> > > this being the case. And this will only work until P2P
> network software
> > > starts to randomly change packet sizes to obfuscate itself. :-(
> >
> > I was told that applications doing it exists. I haven't checked
> it, though.
>
> I haven't heard of an application that does it, but I have always felt
> reasonably sure that it has either already happened or is about to be
> implemented...
Hehe, I agree:)

>
> > > But, I guess we have to work with what we have now, and not worry
> > > about the
> > > future advancements before they happen. :-)
> >
> > Hehe... yes doing something instead of just talking is a good idea:)
>
> Well, for a little while, anyway, until the new version of the
> client comes
> out...
>
The war without end...

> > I think this e-mail is a nice summary. I enjoyed reading it. I could
say
> > that I agree your opinions.
>
> Thank you. :-)
You are welcomed.
>
> > Maybe creating free alternatives to shaping software like those from
> > www.dyband.com is a way. People using it are very happy actually. They
> > adapt to network utilization, allow extensive logging, setting
different
> > parameters like max bandwidth, ramps, minimum acceptable rate. The
main
> > idea is to limit aggresive users and give maximum performance
> and quality
> > (latency, jitter throughput etc.) to standard users. It looks
> very well on
> > paper but I haven't tried dyband yet.... Maybe there is other
> software like
> > this I am not aware of.
>
> I haven't heard about any of them. I am a great believer in "home
brewed"
> solutions. :-)
So am I.
>
> The problem you start getting there is that monitoring and
> shaping traffic on
> a 100 Mb pipe will take a huge amount of CPU power, and even that
> will only
> work if the traffic is not encrypted. The only way of attacking
> the problem I
> can think of is by actually attempting to connect to the client
> machine on
> the suspiciously used well known ports, and seeing if it works.
> If it doesn't
> work as expected, you know it's likely to be a P2P application.
>
> I am not sure if you really want to do that, though, as it
> involves active
> port scanning rather than just monitoring, and some of your customers may
> complain...
Well they will for sure in a scenario described by you. But I think you have
misunderstood me. Dyband don't do any scanning or content analyzing. It
works as a bridge modyfing data rate based on IP addresses. You can set up
complicated scheme of bandwidth sharing. You can even automaticaly limit
some "aggresive users" based on their usage. It happens on the fly and
is
very "smooth" from client point of view. If a client doesn't use his
bandwidth for a while the limit raises (recharges). It allows ISP or
enterprise to FULLY (i mean nearly 100%) utilize their uplink. You don't do
provisioning:).

Maybe it is the only reasonable solution....

RK

On Thursday 27 Mar 2003 16:38, Robert Krycza=B3o wrote:

> > The problem you start getting there is that monitoring and
> > shaping traffic on
> > a 100 Mb pipe will take a huge amount of CPU power, and even that
> > will only
> > work if the traffic is not encrypted. The only way of attacking
> > the problem I
> > can think of is by actually attempting to connect to the client
> > machine on
> > the suspiciously used well known ports, and seeing if it works.
> > If it doesn't
> > work as expected, you know it's likely to be a P2P application.
> >
> > I am not sure if you really want to do that, though, as it
> > involves active
> > port scanning rather than just monitoring, and some of your customers=
 may
> > complain...
>
> Well they will for sure in a scenario described by you. But I think you
> have misunderstood me. Dyband don't do any scanning or content analyzin=
g.
> It works as a bridge modyfing data rate based on IP addresses.

Yes, but in order to detect what the traffic is, as the client software s=
tarts=20
being more clever, you may have to do some pro-active scanning to see whe=
ther=20
the traffic is legit or not. And even then the client software may fake l=
egit=20
server appearance. You would have to mimick the actual P2P connection=20
handshake to be sure. And on some of them you have a real problem, e.g.=20
FastTrack. They use encrypted connection, and the software is closed-sour=
ce,=20
so it's very difficult to get a handle on cracking the protocol.

> You can set
> up complicated scheme of bandwidth sharing. You can even automaticaly l=
imit
> some "aggresive users" based on their usage. It happens on the
fly and =
is
> very "smooth" from client point of view. If a client doesn't use
his
> bandwidth for a while the limit raises (recharges). It allows ISP or
> enterprise to FULLY (i mean nearly 100%) utilize their uplink. You don'=
t do
> provisioning:).
>
> Maybe it is the only reasonable solution....

It sounds like a useful thing to do, but ultimately, you have to detect t=
he=20
traffic you want to throttle before you can throttle it. That is where th=
e=20
biggest problem is.

Gordan

Hi
> >
> > Maybe it is the only reasonable solution....
>
> It sounds like a useful thing to do, but ultimately, you have to
> detect the
> traffic you want to throttle before you can throttle it. That is
> where the
> biggest problem is.
Well in this case w throttle traffic based on IP, IP activity (so aggresive
users are limited on the fly), current uplink utilization etc. In many
sifferent situations this aproach works very well and I am thinking about
shifting to this solution.

RK