Hi,
ruby 1.9.3p125 (both mingw and mswin)
Is this a bug? I''ve got a file, test.txt, that contains nothing but
this:
This is line 1
This is line 2
This is line 3
I ran these snippets of code:
IO.read(''test.txt'') # => "This is line
1\nThis is line
2\nThis is line 3\n"
IO.read(''test.txt'', nil) # => "This is line
1\nThis is line
2\nThis is line 3\n"
IO.read(''test.txt'', nil, 0) # => "This is line
1\r\nThis is line
2\r\nThis is line 3\r\n"
Why are the native line endings retained if I specify an offset? Ruby
1.8.x doesn''t do that, btw.
Regards,
Dan
On Wed, May 2, 2012 at 9:48 PM, Daniel Berger <djberg96 at gmail.com> wrote:> Hi, > > ruby 1.9.3p125 (both mingw and mswin) > > Is this a bug? I''ve got a file, test.txt, that contains nothing but this: > > This is line 1 > This is line 2 > This is line 3 > > I ran these snippets of code: > > IO.read(''test.txt'') ? ? ? ? ?# => "This is line 1\nThis is line > 2\nThis is line 3\n" > IO.read(''test.txt'', nil) ? ? # => "This is line 1\nThis is line > 2\nThis is line 3\n" > IO.read(''test.txt'', nil, 0) # => "This is line 1\r\nThis is line > 2\r\nThis is line 3\r\n" > > Why are the native line endings retained if I specify an offset? Ruby > 1.8.x doesn''t do that, btw. >A regression fixed in patch level 194 Root issue is caused by newline universal: https://bugs.ruby-lang.org/issues/6271 Along other like File#pos Cheers, -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry
On Wed, May 2, 2012 at 7:02 PM, Luis Lavena <luislavena at gmail.com> wrote:> On Wed, May 2, 2012 at 9:48 PM, Daniel Berger <djberg96 at gmail.com> wrote: >> Hi, >> >> ruby 1.9.3p125 (both mingw and mswin) >> >> Is this a bug? I''ve got a file, test.txt, that contains nothing but this: >> >> This is line 1 >> This is line 2 >> This is line 3 >> >> I ran these snippets of code: >> >> IO.read(''test.txt'') ? ? ? ? ?# => "This is line 1\nThis is line >> 2\nThis is line 3\n" >> IO.read(''test.txt'', nil) ? ? # => "This is line 1\nThis is line >> 2\nThis is line 3\n" >> IO.read(''test.txt'', nil, 0) # => "This is line 1\r\nThis is line >> 2\r\nThis is line 3\r\n" >> >> Why are the native line endings retained if I specify an offset? Ruby >> 1.8.x doesn''t do that, btw. >> > > A regression fixed in patch level 194 > > Root issue is caused by newline universal: > > https://bugs.ruby-lang.org/issues/6271 > > Along other like File#posThanks, I upgraded and it''s working now. I noticed another problem I think, though. According to the docs I should be able to pass open_args and those should be passed along to open(). However, this doesn''t work: IO.read(''test.txt'', open_args: [''rS'']) # Try to force a sequential scan But that blows up with an invalid access mode error, even though ''S'' is a legal mode on Windows. It looks to me like io.c is explicitly checking mode values instead of just letting them pass through to the open function and letting it fail on its own if they''re invalid. Or am I doing it wrong? Regards, Dan
Hi, 2012/5/3 Daniel Berger <djberg96 at gmail.com>> On Wed, May 2, 2012 at 7:02 PM, Luis Lavena <luislavena at gmail.com> wrote: > > On Wed, May 2, 2012 at 9:48 PM, Daniel Berger <djberg96 at gmail.com> > wrote: > >> Hi, > >> > >> ruby 1.9.3p125 (both mingw and mswin) > >> > >> Is this a bug? I''ve got a file, test.txt, that contains nothing but > this: > >> > >> This is line 1 > >> This is line 2 > >> This is line 3 > >> > >> I ran these snippets of code: > >> > >> IO.read(''test.txt'') # => "This is line 1\nThis is line > >> 2\nThis is line 3\n" > >> IO.read(''test.txt'', nil) # => "This is line 1\nThis is line > >> 2\nThis is line 3\n" > >> IO.read(''test.txt'', nil, 0) # => "This is line 1\r\nThis is line > >> 2\r\nThis is line 3\r\n" > >> > >> Why are the native line endings retained if I specify an offset? Ruby > >> 1.8.x doesn''t do that, btw. > >> > > > > A regression fixed in patch level 194 > > > > Root issue is caused by newline universal: > > > > https://bugs.ruby-lang.org/issues/6271 > > > > Along other like File#pos > > Thanks, I upgraded and it''s working now. > > I noticed another problem I think, though. According to the docs I > should be able to pass open_args and those should be passed along to > open(). However, this doesn''t work: > > IO.read(''test.txt'', open_args: [''rS'']) # Try to force a sequential scan > > But that blows up with an invalid access mode error, even though ''S'' > is a legal mode on Windows. It looks to me like io.c is explicitly > checking mode values instead of just letting them pass through to the > open function and letting it fail on its own if they''re invalid. > > Or am I doing it wrong? > >I think that is a nice feature. Why don''t you make a feature request on http://bugs.ruby-lang.org/ ? Even if it is rejected, you can make your own IO with the WinRuby :) Regards, Park Heesob -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/win32utils-devel/attachments/20120503/b668e341/attachment.html>
On Wed, May 2, 2012 at 11:01 PM, Heesob Park <phasis at gmail.com> wrote:> Hi, > > 2012/5/3 Daniel Berger <djberg96 at gmail.com> >> >> On Wed, May 2, 2012 at 7:02 PM, Luis Lavena <luislavena at gmail.com> wrote: >> > On Wed, May 2, 2012 at 9:48 PM, Daniel Berger <djberg96 at gmail.com> >> > wrote: >> >> Hi, >> >> >> >> ruby 1.9.3p125 (both mingw and mswin) >> >> >> >> Is this a bug? I''ve got a file, test.txt, that contains nothing but >> >> this: >> >> >> >> This is line 1 >> >> This is line 2 >> >> This is line 3 >> >> >> >> I ran these snippets of code: >> >> >> >> IO.read(''test.txt'') ? ? ? ? ?# => "This is line 1\nThis is line >> >> 2\nThis is line 3\n" >> >> IO.read(''test.txt'', nil) ? ? # => "This is line 1\nThis is line >> >> 2\nThis is line 3\n" >> >> IO.read(''test.txt'', nil, 0) # => "This is line 1\r\nThis is line >> >> 2\r\nThis is line 3\r\n" >> >> >> >> Why are the native line endings retained if I specify an offset? Ruby >> >> 1.8.x doesn''t do that, btw. >> >> >> > >> > A regression fixed in patch level 194 >> > >> > Root issue is caused by newline universal: >> > >> > https://bugs.ruby-lang.org/issues/6271 >> > >> > Along other like File#pos >> >> Thanks, I upgraded and it''s working now. >> >> I noticed another problem I think, though. According to the docs I >> should be able to pass open_args and those should be passed along to >> open(). However, this doesn''t work: >> >> IO.read(''test.txt'', open_args: [''rS'']) # Try to force a sequential scan >> >> But that blows up with an invalid access mode error, even though ''S'' >> is a legal mode on Windows. It looks to me like io.c is explicitly >> checking mode values instead of just letting them pass through to the >> open function and letting it fail on its own if they''re invalid. >> >> Or am I doing it wrong? >> > > I think that is a nice feature. > > Why don''t you make a feature request on?http://bugs.ruby-lang.org/ ?Maybe I will.> Even if it is rejected, you can make your own IO with the WinRuby :)Oh, I plan on it. In fact I was playing with the ffi branch of the win32-nio library again today. The NIO.read benchmarks are pretty good. Much faster than the default IO.read method. I couldn''t get NIO.readlines to work, though. I''m doing something wrong with the FileSegment pointer in FFI, but I''m not sure what. Though, if memory serves, the ReadFileScatter approach was actually slower than the default implementation. But, if you''re up for taking a look, I would appreciate it. Regards, Dan
On Thu, May 3, 2012 at 2:25 AM, Daniel Berger <djberg96 at gmail.com> wrote:> > Oh, I plan on it. In fact I was playing with the ffi branch of the > win32-nio library again today. The NIO.read benchmarks are pretty > good. Much faster than the default IO.read method. >Please note that NIO.read might not be doing all the legwork IO.read is already doing in relation to encodings. Which, along with others, I think is overkill. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry
On Thu, May 3, 2012 at 5:58 AM, Luis Lavena <luislavena at gmail.com> wrote:> On Thu, May 3, 2012 at 2:25 AM, Daniel Berger <djberg96 at gmail.com> wrote: >> >> Oh, I plan on it. In fact I was playing with the ffi branch of the >> win32-nio library again today. The NIO.read benchmarks are pretty >> good. Much faster than the default IO.read method. >> > > Please note that NIO.read might not be doing all the legwork IO.read > is already doing in relation to encodings. > > Which, along with others, I think is overkill.True, I thought that option was a bit silly, since users can always encode after the fact if they want. Performance is being hampered somewhat by having to do newline conversions, but the results are good: c:\Users\djberge\Repositories\win32-nio>rake bench ruby -Ilib benchmarks/win32_nio_benchmarks.rb Small file created (588k) Medium file created (6mb) Large file created (58mb) Huge file created (618mb) user system total real IO.read(small) 0.016000 0.016000 0.032000 ( 0.027001) IO.read(medium) 0.234000 0.062000 0.296000 ( 0.295017) IO.read(large) 2.184000 0.484000 2.668000 ( 2.687154) IO.read(huge) 22.245000 5.007000 27.252000 ( 27.283561) NIO.read(small) 0.016000 0.000000 0.016000 ( 0.017001) NIO.read(medium) 0.172000 0.188000 0.360000 ( 0.617035) NIO.read(large) 1.263000 1.014000 2.277000 ( 4.820276) NIO.read(huge) 13.822000 7.909000 21.731000 ( 32.729872) Minor improvements for smaller files, but the savings gets better as the files get bigger. It''s a shame there''s no text mode for CreateFile. BTW, using FILE_FLAG_NO_BUFFERING with a sector aligned buffer didn''t yield any noticeable improvements that I could see. Regards, Dan
I recommend you use some ramdisk like imdisk to avoid hdd interfere with the results. Sorry for top posting. Sent from mobile. On May 3, 2012 11:48 AM, "Daniel Berger" <djberg96 at gmail.com> wrote:> On Thu, May 3, 2012 at 5:58 AM, Luis Lavena <luislavena at gmail.com> wrote: > > On Thu, May 3, 2012 at 2:25 AM, Daniel Berger <djberg96 at gmail.com> > wrote: > >> > >> Oh, I plan on it. In fact I was playing with the ffi branch of the > >> win32-nio library again today. The NIO.read benchmarks are pretty > >> good. Much faster than the default IO.read method. > >> > > > > Please note that NIO.read might not be doing all the legwork IO.read > > is already doing in relation to encodings. > > > > Which, along with others, I think is overkill. > > True, I thought that option was a bit silly, since users can always > encode after the fact if they want. > > Performance is being hampered somewhat by having to do newline > conversions, but the results are good: > > c:\Users\djberge\Repositories\win32-nio>rake bench > ruby -Ilib benchmarks/win32_nio_benchmarks.rb > Small file created (588k) > Medium file created (6mb) > Large file created (58mb) > Huge file created (618mb) > > user system total real > IO.read(small) 0.016000 0.016000 0.032000 ( 0.027001) > IO.read(medium) 0.234000 0.062000 0.296000 ( 0.295017) > IO.read(large) 2.184000 0.484000 2.668000 ( 2.687154) > IO.read(huge) 22.245000 5.007000 27.252000 ( 27.283561) > > NIO.read(small) 0.016000 0.000000 0.016000 ( 0.017001) > NIO.read(medium) 0.172000 0.188000 0.360000 ( 0.617035) > NIO.read(large) 1.263000 1.014000 2.277000 ( 4.820276) > NIO.read(huge) 13.822000 7.909000 21.731000 ( 32.729872) > > Minor improvements for smaller files, but the savings gets better as > the files get bigger. It''s a shame there''s no text mode for > CreateFile. > > BTW, using FILE_FLAG_NO_BUFFERING with a sector aligned buffer didn''t > yield any noticeable improvements that I could see. > > Regards, > > Dan > _______________________________________________ > win32utils-devel mailing list > win32utils-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/win32utils-devel >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/win32utils-devel/attachments/20120503/79ba1c9d/attachment.html>
On Thu, May 3, 2012 at 8:48 AM, Daniel Berger <djberg96 at gmail.com> wrote:> On Thu, May 3, 2012 at 5:58 AM, Luis Lavena <luislavena at gmail.com> wrote: >> On Thu, May 3, 2012 at 2:25 AM, Daniel Berger <djberg96 at gmail.com> wrote: >>> >>> Oh, I plan on it. In fact I was playing with the ffi branch of the >>> win32-nio library again today. The NIO.read benchmarks are pretty >>> good. Much faster than the default IO.read method. >>> >> >> Please note that NIO.read might not be doing all the legwork IO.read >> is already doing in relation to encodings. >> >> Which, along with others, I think is overkill. > > True, I thought that option was a bit silly, since users can always > encode after the fact if they want. > > Performance is being hampered somewhat by having to do newline > conversions, but the results are good: > > c:\Users\djberge\Repositories\win32-nio>rake bench > ruby -Ilib benchmarks/win32_nio_benchmarks.rb > Small file created (588k) > Medium file created (6mb) > Large file created (58mb) > Huge file created (618mb) > > ? ? ? ? ? ? ? ? ? ? ? ? ? user ? ? system ? ? ?total ? ? ? ?real > IO.read(small) ? ? ? ? 0.016000 ? 0.016000 ? 0.032000 ( ?0.027001) > IO.read(medium) ? ? ? ?0.234000 ? 0.062000 ? 0.296000 ( ?0.295017) > IO.read(large) ? ? ? ? 2.184000 ? 0.484000 ? 2.668000 ( ?2.687154) > IO.read(huge) ? ? ? ? 22.245000 ? 5.007000 ?27.252000 ( 27.283561) > > NIO.read(small) ? ? ? ?0.016000 ? 0.000000 ? 0.016000 ( ?0.017001) > NIO.read(medium) ? ? ? 0.172000 ? 0.188000 ? 0.360000 ( ?0.617035) > NIO.read(large) ? ? ? ?1.263000 ? 1.014000 ? 2.277000 ( ?4.820276) > NIO.read(huge) ? ? ? ?13.822000 ? 7.909000 ?21.731000 ( 32.729872) > > Minor improvements for smaller files, but the savings gets better as > the files get bigger. It''s a shame there''s no text mode for > CreateFile. > > BTW, using FILE_FLAG_NO_BUFFERING with a sector aligned buffer didn''t > yield any noticeable improvements that I could see.If I ignore the line ending transformation, things get *much* better: c:\Users\djberge\Repositories\win32-nio>rake bench ruby -Ilib benchmarks/win32_nio_benchmarks.rb Small file created (588k) Medium file created (6mb) Large file created (58mb) Huge file created (618mb) user system total real IO.read(small) 0.032000 0.016000 0.048000 ( 0.034001) IO.read(medium) 0.202000 0.109000 0.311000 ( 0.318018) IO.read(large) 2.200000 0.484000 2.684000 ( 2.682153) IO.read(huge) 22.449000 4.914000 27.363000 ( 27.389567) NIO.read(small) 0.000000 0.000000 0.000000 ( 0.005000) NIO.read(medium) 0.031000 0.046000 0.077000 ( 0.072004) NIO.read(large) 0.156000 0.484000 0.640000 ( 0.639037) NIO.read(huge) 1.919000 5.164000 7.083000 ( 8.560490) Though to be fair, things are much better if you pass the binary flag with plain Ruby. In fact, the difference is negligible until you get into large files. user system total real IO.read(small) 0.016000 0.016000 0.032000 ( 0.032002) IO.read(small) with rb 0.015000 0.000000 0.015000 ( 0.010001) IO.read(medium) 0.250000 0.078000 0.328000 ( 0.321018) IO.read(medium) with rb 0.015000 0.062000 0.077000 ( 0.084005) IO.read(large) 2.216000 0.468000 2.684000 ( 2.679153) IO.read(large) with rb 0.171000 0.499000 0.670000 ( 0.671039) IO.read(huge) 22.324000 4.945000 27.269000 ( 27.425568) IO.read(huge) 3.276000 8.986000 12.262000 ( 30.654754) Regards, Dan
On Thu, May 3, 2012 at 9:16 AM, Luis Lavena <luislavena at gmail.com> wrote:> I recommend you use some ramdisk like imdisk to avoid hdd interfere with the > results.I don''t really believe in doing stuff like that because it doesn''t reflect real world usage. Besides, my laptop has a solid state drive. :) Regards, Dan
Hmm, then what you state is your performance gain wouldn''t be real for those not running ssd and still on spinning drives. Real World <> your world By using a ramdisk you remove the rpm, inner disk caching and seek time variables from the equation, which makes more easy to determine bottlenecks. Sorry for top posting. Sent from mobile. On May 3, 2012 12:43 PM, "Daniel Berger" <djberg96 at gmail.com> wrote:> On Thu, May 3, 2012 at 9:16 AM, Luis Lavena <luislavena at gmail.com> wrote: > > I recommend you use some ramdisk like imdisk to avoid hdd interfere with > the > > results. > > I don''t really believe in doing stuff like that because it doesn''t > reflect real world usage. Besides, my laptop has a solid state drive. > :) > > Regards, > > Dan > _______________________________________________ > win32utils-devel mailing list > win32utils-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/win32utils-devel >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/win32utils-devel/attachments/20120503/3d960cf5/attachment.html>
On Thu, May 3, 2012 at 8:48 AM, Daniel Berger <djberg96 at gmail.com> wrote:> On Thu, May 3, 2012 at 5:58 AM, Luis Lavena <luislavena at gmail.com> wrote: >> On Thu, May 3, 2012 at 2:25 AM, Daniel Berger <djberg96 at gmail.com> wrote: >>> >>> Oh, I plan on it. In fact I was playing with the ffi branch of the >>> win32-nio library again today. The NIO.read benchmarks are pretty >>> good. Much faster than the default IO.read method. >>> >> >> Please note that NIO.read might not be doing all the legwork IO.read >> is already doing in relation to encodings. >> >> Which, along with others, I think is overkill. > > True, I thought that option was a bit silly, since users can always > encode after the fact if they want. > > Performance is being hampered somewhat by having to do newline > conversions, but the results are good: > > c:\Users\djberge\Repositories\win32-nio>rake bench > ruby -Ilib benchmarks/win32_nio_benchmarks.rb > Small file created (588k) > Medium file created (6mb) > Large file created (58mb) > Huge file created (618mb) > > ? ? ? ? ? ? ? ? ? ? ? ? ? user ? ? system ? ? ?total ? ? ? ?real > IO.read(small) ? ? ? ? 0.016000 ? 0.016000 ? 0.032000 ( ?0.027001) > IO.read(medium) ? ? ? ?0.234000 ? 0.062000 ? 0.296000 ( ?0.295017) > IO.read(large) ? ? ? ? 2.184000 ? 0.484000 ? 2.668000 ( ?2.687154) > IO.read(huge) ? ? ? ? 22.245000 ? 5.007000 ?27.252000 ( 27.283561) > > NIO.read(small) ? ? ? ?0.016000 ? 0.000000 ? 0.016000 ( ?0.017001) > NIO.read(medium) ? ? ? 0.172000 ? 0.188000 ? 0.360000 ( ?0.617035) > NIO.read(large) ? ? ? ?1.263000 ? 1.014000 ? 2.277000 ( ?4.820276) > NIO.read(huge) ? ? ? ?13.822000 ? 7.909000 ?21.731000 ( 32.729872) > > Minor improvements for smaller files, but the savings gets better as > the files get bigger. It''s a shame there''s no text mode for > CreateFile. > > BTW, using FILE_FLAG_NO_BUFFERING with a sector aligned buffer didn''t > yield any noticeable improvements that I could see.Hm, I ran the numbers again today and suddenly NIO is slower. Dunno what to make of that. Regards, Dan