Hi, There has been some discussion of evented database adapters. One potential application mentioned was in the EM based analogger. For this reason I thought the following blog might be of interest to some EM users... apologies if it is considered OT. http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ I''m still to use analogger, but it is what I have in mind for logging when I rewrite one library of mine. Its not clear to me if the Analogger client instance just needs to be pointed at the splunk server:port, anyway its on my ''to-investigate-list'' and thought other EM users may be interested. HTH Mark
On Tue, Jun 24, 2008 at 11:09 PM, Mark V <mvyver at gmail.com> wrote:> Hi, > There has been some discussion of evented database adapters. One > potential application mentioned was in the EM based analogger. For > this reason I thought the following blog might be of interest to some > EM users... apologies if it is considered OT. > > http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ > > I''m still to use analogger, but it is what I have in mind for logging > when I rewrite one library of mine. > Its not clear to me if the Analogger client instance just needs to be > pointed at the splunk server:port, anyway its on my > ''to-investigate-list'' and thought other EM users may be interested.I''ll take a look today and see what I can learn. One of the many nice things about Analogger (IMHO) is that it is extremely fast. Analogger Speedtest -- larger messages Entering event loop Testing 500000 messages of 100 bytes each. user system total real 1.900000 0.040000 1.940000 ( 1.943443) Message rate: 257277.611805132/second (1.943426) Real rate (3.088595 seconds): 161885.906051133 Analogger Speedtest -- larger messages Entering event loop Testing 500000 messages of 1000 bytes each. user system total real 2.300000 0.400000 2.700000 ( 2.694631) Message rate: 185555.471437075/second (2.694612) Real rate (4.316182 seconds): 115843.122463325 Message rate is how fast they went from client to server, real rate includes the time taken to write them to disk (which is dependent on your drives and filesystem). That''s with the EM based client, BTW. It''s slower with the standard Ruby networking based client, but is still faster than the standard Ruby Logger. So logging has very little effect on throughput of your application, even when doing a lot of it. In just scanning the Splurk info quickly, the best integration approach would probably be to provide a plugin that, with a config option, allows Analogger logs to be written out to a Splurk destination. Kirk Haines
On Wed, Jun 25, 2008 at 11:38 PM, Kirk Haines <wyhaines at gmail.com> wrote:> On Tue, Jun 24, 2008 at 11:09 PM, Mark V <mvyver at gmail.com> wrote: >> Hi, >> There has been some discussion of evented database adapters. One >> potential application mentioned was in the EM based analogger. For >> this reason I thought the following blog might be of interest to some >> EM users... apologies if it is considered OT. >> >> http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ >> >> I''m still to use analogger, but it is what I have in mind for logging >> when I rewrite one library of mine. >> Its not clear to me if the Analogger client instance just needs to be >> pointed at the splunk server:port, anyway its on my >> ''to-investigate-list'' and thought other EM users may be interested. > > I''ll take a look today and see what I can learn. One of the many nice > things about Analogger (IMHO) is that it is extremely fast. > > Analogger Speedtest -- larger messages > Entering event loop > Testing 500000 messages of 100 bytes each. > user system total real > 1.900000 0.040000 1.940000 ( 1.943443) > > Message rate: 257277.611805132/second (1.943426) > Real rate (3.088595 seconds): 161885.906051133 > > > > Analogger Speedtest -- larger messages > Entering event loop > Testing 500000 messages of 1000 bytes each. > user system total real > 2.300000 0.400000 2.700000 ( 2.694631) > > Message rate: 185555.471437075/second (2.694612) > Real rate (4.316182 seconds): 115843.122463325 > > Message rate is how fast they went from client to server, real rate > includes the time taken to write them to disk (which is dependent on > your drives and filesystem). That''s with the EM based client, BTW. > It''s slower with the standard Ruby > networking based client, but is still faster than the standard Ruby Logger. > > So logging has very little effect on throughput of your application, > even when doing a lot of it. >Yep the performance is impressive and doesn''t have a Ruby competitor as far as I am aware. Splunk it seems to me is about operating on another scale: > 500MB/day. As I remark below, I think a real benefit of Analogger would be to permit an application to really scale without logging becoming an ''early issue''. Aggregated logging would seem to be a very useful (critical?) facility for anyone writing an EM based applications, so I won''t take this off-list. If you''ll indulge me, a couple of thoughts: - It would seem to be very useful to have the Analogger output directed to the local syslog, and then the user set-up their machine to direct that syslog to the splunk server. - An Analogger-Splunk ''direct'' connection would also be useful. I freely admit to being a novice, but unless I misread things the real advantage of aggregated logging is that one have some hint of what was going on in several places at or around the time of an issue. If this is right then logging to syslog would seem to be most useful. Is syslog logging from within Ruby even possible ?>From the blog/screencast below it seems that Splunk can be tweaked to''parsing/processor/indexing'' 5Mbps. It also appears that parsing, etc. might be an issue and perhaps Analogger can format a message that is splunk ''friendly'' http://www.splunk.com/article/2183 For those interested in EC2, some (non-splunk) performance metrics are here: http://rubyurl.com/EeQb Cheers Mark> In just scanning the Splurk info quickly, the best integration > approach would probably be to provide a plugin that, with a config > option, allows Analogger logs to be written out to a Splurk > destination. > > > Kirk Haines >
I''ve been playing with splunk recently, and although the software and web-interface are extremely impressive (easily one of the coolest pieces of software I have used in a while), it is quite the memory and cpu hog (my load average shoots up to 11 and it uses ~500MB of RAM). Aman Gupta On Wed, Jun 25, 2008 at 3:31 PM, Mark V <mvyver at gmail.com> wrote:> On Wed, Jun 25, 2008 at 11:38 PM, Kirk Haines <wyhaines at gmail.com> wrote: > > On Tue, Jun 24, 2008 at 11:09 PM, Mark V <mvyver at gmail.com> wrote: > >> Hi, > >> There has been some discussion of evented database adapters. One > >> potential application mentioned was in the EM based analogger. For > >> this reason I thought the following blog might be of interest to some > >> EM users... apologies if it is considered OT. > >> > >> http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ > >> > >> I''m still to use analogger, but it is what I have in mind for logging > >> when I rewrite one library of mine. > >> Its not clear to me if the Analogger client instance just needs to be > >> pointed at the splunk server:port, anyway its on my > >> ''to-investigate-list'' and thought other EM users may be interested. > > > > I''ll take a look today and see what I can learn. One of the many nice > > things about Analogger (IMHO) is that it is extremely fast. > > > > Analogger Speedtest -- larger messages > > Entering event loop > > Testing 500000 messages of 100 bytes each. > > user system total real > > 1.900000 0.040000 1.940000 ( 1.943443) > > > > Message rate: 257277.611805132/second (1.943426) > > Real rate (3.088595 seconds): 161885.906051133 > > > > > > > > Analogger Speedtest -- larger messages > > Entering event loop > > Testing 500000 messages of 1000 bytes each. > > user system total real > > 2.300000 0.400000 2.700000 ( 2.694631) > > > > Message rate: 185555.471437075/second (2.694612) > > Real rate (4.316182 seconds): 115843.122463325 > > > > Message rate is how fast they went from client to server, real rate > > includes the time taken to write them to disk (which is dependent on > > your drives and filesystem). That''s with the EM based client, BTW. > > It''s slower with the standard Ruby > > networking based client, but is still faster than the standard Ruby > Logger. > > > > So logging has very little effect on throughput of your application, > > even when doing a lot of it. > > > > Yep the performance is impressive and doesn''t have a Ruby competitor > as far as I am aware. > Splunk it seems to me is about operating on another scale: > 500MB/day. > As I remark below, I think a real benefit of Analogger would be to > permit an application to really scale without logging becoming an > ''early issue''. > > Aggregated logging would seem to be a very useful (critical?) facility > for anyone writing an EM based applications, so I won''t take this > off-list. > If you''ll indulge me, a couple of thoughts: > - It would seem to be very useful to have the Analogger output > directed to the local syslog, and then the user set-up their machine > to direct that syslog to the splunk server. > - An Analogger-Splunk ''direct'' connection would also be useful. > > I freely admit to being a novice, but unless I misread things the real > advantage of aggregated logging is that one have some hint of what was > going on in several places at or around the time of an issue. If this > is right then logging to syslog would seem to be most useful. Is > syslog logging from within Ruby even possible ? > > >From the blog/screencast below it seems that Splunk can be tweaked to > ''parsing/processor/indexing'' 5Mbps. It also appears that parsing, > etc. might be an issue and perhaps Analogger can format a message that > is splunk ''friendly'' > http://www.splunk.com/article/2183 > > For those interested in EC2, some (non-splunk) performance metrics are > here: > http://rubyurl.com/EeQb > > Cheers > Mark > > > In just scanning the Splurk info quickly, the best integration > > approach would probably be to provide a plugin that, with a config > > option, allows Analogger logs to be written out to a Splurk > > destination. > > > > > > Kirk Haines > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20080625/d9ba53f4/attachment.html>
On Thu, Jun 26, 2008 at 9:10 AM, Aman Gupta <themastermind1 at gmail.com> wrote:> I''ve been playing with splunk recently, and although the software and > web-interface are extremely impressive (easily one of the coolest pieces of > software I have used in a while), it is quite the memory and cpu hog (my > load average shoots up to 11 and it uses ~500MB of RAM).Yes, don''t underestimate that. In the tuning screencast I referred to you''ll see that at one point the saturated cpu count is between 3 to 6. I _think_ the cores were mentioned as 1.8 GHz, I didn''t pay attention to the memory usage, but that probably will be high too. To compare to Analogger''s throughput to disk I''m still digging to find what Splunk raw throughput figures are without the parsing/aggregating/indexing. Mark> Aman Gupta > On Wed, Jun 25, 2008 at 3:31 PM, Mark V <mvyver at gmail.com> wrote: >> >> On Wed, Jun 25, 2008 at 11:38 PM, Kirk Haines <wyhaines at gmail.com> wrote: >> > On Tue, Jun 24, 2008 at 11:09 PM, Mark V <mvyver at gmail.com> wrote: >> >> Hi, >> >> There has been some discussion of evented database adapters. One >> >> potential application mentioned was in the EM based analogger. For >> >> this reason I thought the following blog might be of interest to some >> >> EM users... apologies if it is considered OT. >> >> >> >> http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ >> >> >> >> I''m still to use analogger, but it is what I have in mind for logging >> >> when I rewrite one library of mine. >> >> Its not clear to me if the Analogger client instance just needs to be >> >> pointed at the splunk server:port, anyway its on my >> >> ''to-investigate-list'' and thought other EM users may be interested. >> > >> > I''ll take a look today and see what I can learn. One of the many nice >> > things about Analogger (IMHO) is that it is extremely fast. >> > >> > Analogger Speedtest -- larger messages >> > Entering event loop >> > Testing 500000 messages of 100 bytes each. >> > user system total real >> > 1.900000 0.040000 1.940000 ( 1.943443) >> > >> > Message rate: 257277.611805132/second (1.943426) >> > Real rate (3.088595 seconds): 161885.906051133 >> > >> > >> > >> > Analogger Speedtest -- larger messages >> > Entering event loop >> > Testing 500000 messages of 1000 bytes each. >> > user system total real >> > 2.300000 0.400000 2.700000 ( 2.694631) >> > >> > Message rate: 185555.471437075/second (2.694612) >> > Real rate (4.316182 seconds): 115843.122463325 >> > >> > Message rate is how fast they went from client to server, real rate >> > includes the time taken to write them to disk (which is dependent on >> > your drives and filesystem). That''s with the EM based client, BTW. >> > It''s slower with the standard Ruby >> > networking based client, but is still faster than the standard Ruby >> > Logger. >> > >> > So logging has very little effect on throughput of your application, >> > even when doing a lot of it. >> > >> >> Yep the performance is impressive and doesn''t have a Ruby competitor >> as far as I am aware. >> Splunk it seems to me is about operating on another scale: > 500MB/day. >> As I remark below, I think a real benefit of Analogger would be to >> permit an application to really scale without logging becoming an >> ''early issue''. >> >> Aggregated logging would seem to be a very useful (critical?) facility >> for anyone writing an EM based applications, so I won''t take this >> off-list. >> If you''ll indulge me, a couple of thoughts: >> - It would seem to be very useful to have the Analogger output >> directed to the local syslog, and then the user set-up their machine >> to direct that syslog to the splunk server. >> - An Analogger-Splunk ''direct'' connection would also be useful. >> >> I freely admit to being a novice, but unless I misread things the real >> advantage of aggregated logging is that one have some hint of what was >> going on in several places at or around the time of an issue. If this >> is right then logging to syslog would seem to be most useful. Is >> syslog logging from within Ruby even possible ? >> >> >From the blog/screencast below it seems that Splunk can be tweaked to >> ''parsing/processor/indexing'' 5Mbps. It also appears that parsing, >> etc. might be an issue and perhaps Analogger can format a message that >> is splunk ''friendly'' >> http://www.splunk.com/article/2183 >> >> For those interested in EC2, some (non-splunk) performance metrics are >> here: >> http://rubyurl.com/EeQb >> >> Cheers >> Mark >> >> > In just scanning the Splurk info quickly, the best integration >> > approach would probably be to provide a plugin that, with a config >> > option, allows Analogger logs to be written out to a Splurk >> > destination. >> > >> > >> > Kirk Haines >> > >> _______________________________________________ >> Eventmachine-talk mailing list >> Eventmachine-talk at rubyforge.org >> http://rubyforge.org/mailman/listinfo/eventmachine-talk > >
On Thu, Jun 26, 2008 at 10:13 AM, Mark V <mvyver at gmail.com> wrote:> On Thu, Jun 26, 2008 at 9:10 AM, Aman Gupta <themastermind1 at gmail.com> wrote: >> I''ve been playing with splunk recently, and although the software and >> web-interface are extremely impressive (easily one of the coolest pieces of >> software I have used in a while), it is quite the memory and cpu hog (my >> load average shoots up to 11 and it uses ~500MB of RAM). > > Yes, don''t underestimate that. > In the tuning screencast I referred to you''ll see that at one point > the saturated cpu count is between 3 to 6. > I _think_ the cores were mentioned as 1.8 GHz, I didn''t pay attention > to the memory usage, but that probably will be high too. > To compare to Analogger''s throughput to disk I''m still digging to find > what Splunk raw throughput figures are without the > parsing/aggregating/indexing.To minimize someones search costs: - Up to 2 Cores, multiple source data types, with full indexing(I think) http://www.splunk.com/doc/3.2.4/deployment/DeployBenchmarks - Up to 4 Cores, syslog, different index densities: http://www.splunk.com/themes/splunk_com/img/assets/pdfs/education/SplunkPerformanceGuide2_1.pdf It would seem that to get Analogger levels of performance with full indexing multiple CPU''s are required, as well as (?) some tuning of the Splunk server. Speculation: It _seems_ that with minimal index density one might get Analogger levels of performance with one CPU, this is speculation on the indicated 150K events per second in the ''up to 4 cores'' set up with an average 347 bytes per message. Analogger-Splunk and Analogger-syslog-Splunk would seem to be worthwhile since it seems a single CPU single Splunk server could conceivably handle (with low index density) several Analogger clients running flatchat. Mark> > Mark > >> Aman Gupta >> On Wed, Jun 25, 2008 at 3:31 PM, Mark V <mvyver at gmail.com> wrote: >>> >>> On Wed, Jun 25, 2008 at 11:38 PM, Kirk Haines <wyhaines at gmail.com> wrote: >>> > On Tue, Jun 24, 2008 at 11:09 PM, Mark V <mvyver at gmail.com> wrote: >>> >> Hi, >>> >> There has been some discussion of evented database adapters. One >>> >> potential application mentioned was in the EM based analogger. For >>> >> this reason I thought the following blog might be of interest to some >>> >> EM users... apologies if it is considered OT. >>> >> >>> >> http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ >>> >> >>> >> I''m still to use analogger, but it is what I have in mind for logging >>> >> when I rewrite one library of mine. >>> >> Its not clear to me if the Analogger client instance just needs to be >>> >> pointed at the splunk server:port, anyway its on my >>> >> ''to-investigate-list'' and thought other EM users may be interested. >>> > >>> > I''ll take a look today and see what I can learn. One of the many nice >>> > things about Analogger (IMHO) is that it is extremely fast. >>> > >>> > Analogger Speedtest -- larger messages >>> > Entering event loop >>> > Testing 500000 messages of 100 bytes each. >>> > user system total real >>> > 1.900000 0.040000 1.940000 ( 1.943443) >>> > >>> > Message rate: 257277.611805132/second (1.943426) >>> > Real rate (3.088595 seconds): 161885.906051133 >>> > >>> > >>> > >>> > Analogger Speedtest -- larger messages >>> > Entering event loop >>> > Testing 500000 messages of 1000 bytes each. >>> > user system total real >>> > 2.300000 0.400000 2.700000 ( 2.694631) >>> > >>> > Message rate: 185555.471437075/second (2.694612) >>> > Real rate (4.316182 seconds): 115843.122463325 >>> > >>> > Message rate is how fast they went from client to server, real rate >>> > includes the time taken to write them to disk (which is dependent on >>> > your drives and filesystem). That''s with the EM based client, BTW. >>> > It''s slower with the standard Ruby >>> > networking based client, but is still faster than the standard Ruby >>> > Logger. >>> > >>> > So logging has very little effect on throughput of your application, >>> > even when doing a lot of it. >>> > >>> >>> Yep the performance is impressive and doesn''t have a Ruby competitor >>> as far as I am aware. >>> Splunk it seems to me is about operating on another scale: > 500MB/day. >>> As I remark below, I think a real benefit of Analogger would be to >>> permit an application to really scale without logging becoming an >>> ''early issue''. >>> >>> Aggregated logging would seem to be a very useful (critical?) facility >>> for anyone writing an EM based applications, so I won''t take this >>> off-list. >>> If you''ll indulge me, a couple of thoughts: >>> - It would seem to be very useful to have the Analogger output >>> directed to the local syslog, and then the user set-up their machine >>> to direct that syslog to the splunk server. >>> - An Analogger-Splunk ''direct'' connection would also be useful. >>> >>> I freely admit to being a novice, but unless I misread things the real >>> advantage of aggregated logging is that one have some hint of what was >>> going on in several places at or around the time of an issue. If this >>> is right then logging to syslog would seem to be most useful. Is >>> syslog logging from within Ruby even possible ? >>> >>> >From the blog/screencast below it seems that Splunk can be tweaked to >>> ''parsing/processor/indexing'' 5Mbps. It also appears that parsing, >>> etc. might be an issue and perhaps Analogger can format a message that >>> is splunk ''friendly'' >>> http://www.splunk.com/article/2183 >>> >>> For those interested in EC2, some (non-splunk) performance metrics are >>> here: >>> http://rubyurl.com/EeQb >>> >>> Cheers >>> Mark >>> >>> > In just scanning the Splurk info quickly, the best integration >>> > approach would probably be to provide a plugin that, with a config >>> > option, allows Analogger logs to be written out to a Splurk >>> > destination. >>> > >>> > >>> > Kirk Haines >>> > >>> _______________________________________________ >>> Eventmachine-talk mailing list >>> Eventmachine-talk at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/eventmachine-talk >> >> >