win32utils devel - Aug 2007 - Some more win32-changenotify analysis

Hi all,

I decided to check the responsiveness of the pure Ruby vs C extension 
versions of win32-changenotify. I setup this little file generator program:

a = []

10.times{ |n|
    a << Thread.new{
       File.open("File_#{n}", ''w''){ |fh| fh.puts
"test #{n}" }
    }
}

a.each{ |t| t.join }

The pure Ruby version did not do so well. In some cases it only picked 
up one event. In most cases, it didn''t pick up *any* events!

The C extension did much better, though the behavior was somewhat odd. 
Most of the time it picked up multiple notifications on some files, 
while missing others entirely:

[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_0">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_4">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_4">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_5">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_5">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_6">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_6">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_7">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_7">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_8">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_8">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_9">]
[#<struct Struct::ChangeNotifyStruct action="modified", 
file_name="test1\\test2\\File_9">]

I also tried with a C version of win32-changenotify using a pure Ruby 
version of win32-ipc. Although the results were mixed, it did seem to 
get a little worse in terms of the notifications it returned, typically 
only returning 4 or 5 notifications, though sometimes it did as well or 
better than using the C version of win32-ipc, so I''m not sure what to 
make of that.

I''m not sure what to make of the fact that the pure Ruby version
doesn''t
pick up Threaded events most (all?) of the time. Maybe there''s some way
to use threads within changenotify.rb to make it more responsive?

Otherwise, I think I''ll have to declare this a failed experiment.

Regards,

Dan

Hi,

2007/8/8, Daniel Berger <djberg96 at gmail.com>:
>
> Hi all,
>
> I decided to check the responsiveness of the pure Ruby vs C extension
> versions of win32-changenotify. I setup this little file generator
> program:
>
> a = []
>
> 10.times{ |n|
>    a << Thread.new{
>       File.open("File_#{n}", ''w''){ |fh| fh.puts
"test #{n}" }
>    }
> }
>
> a.each{ |t| t.join }
>
> The pure Ruby version did not do so well. In some cases it only picked
> up one event. In most cases, it didn''t pick up *any* events!
>
> The C extension did much better, though the behavior was somewhat odd.
> Most of the time it picked up multiple notifications on some files,
> while missing others entirely:
>
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_0">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_4">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_4">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_5">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_5">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_6">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_6">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_7">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_7">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_8">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_8">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_9">]
> [#<struct Struct::ChangeNotifyStruct action="modified",
> file_name="test1\\test2\\File_9">]
>
> I also tried with a C version of win32-changenotify using a pure Ruby
> version of win32-ipc. Although the results were mixed, it did seem to
> get a little worse in terms of the notifications it returned, typically
> only returning 4 or 5 notifications, though sometimes it did as well or
> better than using the C version of win32-ipc, so I''m not sure what
to
> make of that.
>
> I''m not sure what to make of the fact that the pure Ruby version
doesn''t
> pick up Threaded events most (all?) of the time. Maybe there''s
some way
> to use threads within changenotify.rb to make it more responsive?
>
> Otherwise, I think I''ll have to declare this a failed experiment.
>
> Regards,
>
> Dan

Don''t give up yet :)

I googled and found a Delphi version.
I ported it into Ruby and in my test, it can catch all events.

It requires CreateIoCompletionPort  and GetQueuedCompletionStatus API
function.
I also found a bug releated with aquiring next offset of fni at
get_file_action.

Here is another version of wait and get_file_action:

      CreateIoCompletionPort = Win32API.new(''kernel32'',
''CreateIoCompletionPort'', ''LLPL'',
''L'')
      GetQueuedCompletionStatus = Win32API.new(''kernel32'',
''GetQueuedCompletionStatus'', ''LPPPL'',
''I'')

      def CreateIoCompletionPort(handle,port,key,threads)
        CreateIoCompletionPort.call(handle,port,key,threads)
      end
      def GetQueuedCompletionStatus(handle,bytes,key,overlap,millisec)
       GetQueuedCompletionStatus.call(handle,bytes,key,overlap,millisec) > 0
      end

      # Waits up to ''seconds'' for a notification to occur, or
infinitely
      # if no value is specified.
      #
      # If a block is provided, yields a ChangeNotifyStruct that contains
two
      # members: file_name and action.
      #
      def wait(seconds = INFINITE)
         seconds *= 1000 unless seconds == INFINITE

         fni   = 0.chr * 65536 # FILE_NOTIFY_INFORMATION struct buffer
         bytes = [0].pack(''L'')
         num_bytes = [0].pack(''L'')

         subtree    = @recursive ? 1 : 0
         dir_handle = get_dir_handle(@path)

         completion_key = [12345].pack(''L'')
         completion_port = CreateIoCompletionPort(dir_handle, 0,
completion_key, 0);

         bool = ReadDirectoryChangesW(dir_handle,fni,fni.size
,subtree, at filter,bytes, at overlap,0)
         unless bool
            raise Error, get_last_error
         end

         while true
           GetQueuedCompletionStatus(completion_port, num_bytes,
completion_key, @overlap, seconds)
           break if completion_key.unpack(''L'').first == 0
           yield get_file_action(fni) if block_given?
           ReadDirectoryChangesW(dir_handle,fni,fni.size
,subtree, at filter,bytes, at overlap,0)
        end
      end


      def get_file_action(fni2)
         fni = fni2.dup
         array  = []

         while true
            int_action = fni[4,4].unpack(''L'')[0]

            str_action = ''unknown''

            case int_action
               when FILE_ACTION_ADDED
                  str_action = ''added''
               when FILE_ACTION_REMOVED
                  str_action = ''removed''
               when FILE_ACTION_MODIFIED
                  str_action = ''modified''
               when FILE_ACTION_RENAMED_OLD_NAME
                  str_action = ''renamed old name''
               when FILE_ACTION_RENAMED_NEW_NAME
                  str_action = ''renamed new name''
            end

            len  = fni[8,4].unpack(''L'').first # FileNameLength
struct member
            file = fni[12,len] + "\0\0"       # FileName struct member
+
null
            buf  = 0.chr * 260

            WideCharToMultiByte(CP_ACP, 0, file, -1, buf, 260, 0, 0)

            file = File.join(@path, buf.unpack(''A*'')[0])

            struct = ChangeNotifyStruct.new(str_action, file)
            array.push(struct)
            break if fni[0,4].unpack(''L'')[0] == 0
            fni = fni[fni[0,4].unpack(''L'').first .. -1] # Next
offset
            break if fni.nil?
         end

         array
      end
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070809/1b1b6e5e/attachment.html

On 8/8/07, Heesob Park <phasis at gmail.com>
wrote:
> Hi,
<snip>
> I googled and found a Delphi version.
> I ported it into Ruby and in my test, it can catch all events.
>
> It requires CreateIoCompletionPort  and GetQueuedCompletionStatus API
> function.
> I also found a bug releated with aquiring next offset of fni at
> get_file_action.
<snip code>

Very nice. I''ll try this out soon.

On a side note, after looking at the docs for CreateIoCompletionPort
and GetQueuedCompletionStatus, isn''t that how Kernel#select ought to
be implemented for MS Windows? I don''t understand the sockets API
they''re using.

Perhaps it''s a backwards compatibility issue (95/98/ME). Mega bonus
points if you can solve the Kernel#select blocking issue on MS
Windows, since we only care about NT. :)

Dan

On 8/8/07, Heesob Park <phasis at gmail.com> wrote:

<snip>
> Don''t give up yet :)
>
> I googled and found a Delphi version.
> I ported it into Ruby and in my test, it can catch all events.
>
> It requires CreateIoCompletionPort  and GetQueuedCompletionStatus API
> function.
> I also found a bug releated with aquiring next offset of fni at
> get_file_action.
>
> Here is another version of wait and get_file_action:
<snip>

I tried this on my Windows XP (Home) SP2 laptop at home but it didn''t
pick up any events when I ran the threaded file generator with an
infinite wait period. When I specified a wait time of 10 seconds, it
seemed to pick up one event every 10 seconds, but with the file name
chopped off.

I''ll try it on my XP Pro box tomorrow at work and see what happens.

Is there a way to do a FileIOCompletionRoutine via Win32API btw?

Regards,

Dan

Hi,


2007/8/9, Daniel Berger <djberg96 at gmail.com>:
>
> On 8/8/07, Heesob Park <phasis at gmail.com> wrote:
>
> <snip>
>
> > Don''t give up yet :)
> >
> > I googled and found a Delphi version.
> > I ported it into Ruby and in my test, it can catch all events.
> >
> > It requires CreateIoCompletionPort  and GetQueuedCompletionStatus API
> > function.
> > I also found a bug releated with aquiring next offset of fni at
> > get_file_action.
> >
> > Here is another version of wait and get_file_action:
>
> <snip>
>
> I tried this on my Windows XP (Home) SP2 laptop at home but it
didn''t
> pick up any events when I ran the threaded file generator with an
> infinite wait period. When I specified a wait time of 10 seconds, it
> seemed to pick up one event every 10 seconds, but with the file name
> chopped off.
>
> I''ll try it on my XP Pro box tomorrow at work and see what
happens.

In my test with XP Home SP2 , it works fine.
Do you modified both wait and get_file_aciton with my version?

Is there a way to do a FileIOCompletionRoutine via Win32API btw?


FileIOCompletionRoutine is application defined callback function.
As far as I know Win32API don''t support callback function.

> Regards,
>
> Dan

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070809/ac76ec9f/attachment-0001.html

On 8/9/07, Heesob Park <phasis at gmail.com>
wrote:
>
> Hi,
>
>
> 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > On 8/8/07, Heesob Park <phasis at gmail.com> wrote:
> >
> > <snip>
> >
> > > Don''t give up yet :)
> > >
> > > I googled and found a Delphi version.
> > > I ported it into Ruby and in my test, it can catch all events.
> > >
> > > It requires CreateIoCompletionPort  and GetQueuedCompletionStatus
API
> > > function.
> > > I also found a bug releated with aquiring next offset of fni at
> > > get_file_action.
> > >
> > > Here is another version of wait and get_file_action:
> >
> > <snip>
> >
> > I tried this on my Windows XP (Home) SP2 laptop at home but it
didn''t
> > pick up any events when I ran the threaded file generator with an
> > infinite wait period. When I specified a wait time of 10 seconds, it
> > seemed to pick up one event every 10 seconds, but with the file name
> > chopped off.
> >
> > I''ll try it on my XP Pro box tomorrow at work and see what
happens.
>
>
> In my test with XP Home SP2 , it works fine.
> Do you modified both wait and get_file_aciton with my version?
Yes, though your version does do a much better job at picking up
multiple file events that aren''t threaded in my limited testing. For
example, opening and saving a file via gvim now picks up four separate
events (tempfile generation, deletion, etc).

It just seems to be the threaded file creation it doesn''t pick up.
I''m
not sure why, but I''ll try at work and see if I see the same behavior
on XP Pro.
> > Is there a way to do a FileIOCompletionRoutine via Win32API btw?
>
>
> FileIOCompletionRoutine is application defined callback function.
> As far as I know Win32API don''t support callback function.
Any way we could add it? I know the core team won''t take patches, but
I can always release something separately.

Dan

PS - I uploaded Windows::NIO into CVS (in the windows-pr project),
that contains the functions we need for this. I think I''ll put out
windows-pr 0.7.0 today.

> On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
<snip>
> > In my test with XP Home SP2 , it works fine.
> > Do you modified both wait and get_file_aciton with my version?
BTW, I committed the changes (with some extra error checking) into CVS
so you can take a look.

Regards,

Dan

Hi,

2007/8/9, Daniel Berger <djberg96 at gmail.com>:
>
> > On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
>
> <snip>
>
> > > In my test with XP Home SP2 , it works fine.
> > > Do you modified both wait and get_file_aciton with my version?
>
> BTW, I committed the changes (with some extra error checking) into CVS
> so you can take a look.

Now I noticed some events ocurred between GetQueuedCompletionStatus and
ReadDirectoryChangesW might be missed. The faster CPU machine can detect the
more events. And this version cannot detect the whole events.

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070809/12fae252/attachment.html

On 8/9/07, Daniel Berger <djberg96 at gmail.com>
wrote:
> On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> >
> > Hi,
> >
> >
> > 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > On 8/8/07, Heesob Park <phasis at gmail.com> wrote:
> > >
> > > <snip>
> > >
> > > > Don''t give up yet :)
> > > >
> > > > I googled and found a Delphi version.
> > > > I ported it into Ruby and in my test, it can catch all
events.
> > > >
> > > > It requires CreateIoCompletionPort  and
GetQueuedCompletionStatus API
> > > > function.
> > > > I also found a bug releated with aquiring next offset of fni
at
> > > > get_file_action.
> > > >
> > > > Here is another version of wait and get_file_action:
> > >
> > > <snip>
> > >
> > > I tried this on my Windows XP (Home) SP2 laptop at home but it
didn''t
> > > pick up any events when I ran the threaded file generator with an
> > > infinite wait period. When I specified a wait time of 10 seconds,
it
> > > seemed to pick up one event every 10 seconds, but with the file
name
> > > chopped off.
> > >
> > > I''ll try it on my XP Pro box tomorrow at work and see
what happens.
On my XP Pro box at work it worked fine - picked up every change and
the file names were fine. Must be something about my XP Home laptop.
Good thing we only support Pro officially. :)

I just realized that I forgot to manually set the event to a signaled
state, so I''ll do that, then release assuming all else is well.

Regards,

Dan

On 8/9/07, Heesob Park <phasis at gmail.com>
wrote:
> Hi,
>
>
> 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> >
> > <snip>
> >
> > > > In my test with XP Home SP2 , it works fine.
> > > > Do you modified both wait and get_file_aciton with my
version?
> >
> > BTW, I committed the changes (with some extra error checking) into CVS
> > so you can take a look.
>
>
> Now I noticed some events ocurred between GetQueuedCompletionStatus and
> ReadDirectoryChangesW might be missed. The faster CPU machine can detect
the
> more events. And this version cannot detect the whole events.
Anything we can do to improve it? I think we''re still doing better
than the current C version in any case. :)

Thanks,

Dan

2007/8/9, Daniel Berger <djberg96 at gmail.com>:
>
> On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> > Hi,
> >
> >
> > 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > > On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> > >
> > > <snip>
> > >
> > > > > In my test with XP Home SP2 , it works fine.
> > > > > Do you modified both wait and get_file_aciton with my
version?
> > >
> > > BTW, I committed the changes (with some extra error checking)
into CVS
> > > so you can take a look.
> >
> >
> > Now I noticed some events ocurred between GetQueuedCompletionStatus
and
> > ReadDirectoryChangesW might be missed. The faster CPU machine can
detect
> the
> > more events. And this version cannot detect the whole events.
>
> Anything we can do to improve it? I think we''re still doing better
> than the current C version in any case. :)

In case of C version, I guess It could be improved by using thread. But in
case of pure Ruby version, I have no idea.
If someday Ruby support native windows thread, it would be much better.

Thanks,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070809/7883fea5/attachment.html

On 8/9/07, Heesob Park <phasis at gmail.com>
wrote:
>
>
>
> 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> > > Hi,
> > >
> > >
> > > 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > > > On 8/9/07, Heesob Park <phasis at gmail.com>
wrote:
> > > >
> > > > <snip>
> > > >
> > > > > > In my test with XP Home SP2 , it works fine.
> > > > > > Do you modified both wait and get_file_aciton with
my version?
> > > >
> > > > BTW, I committed the changes (with some extra error
checking) into CVS
> > > > so you can take a look.
> > >
> > >
> > > Now I noticed some events ocurred between
GetQueuedCompletionStatus and
> > > ReadDirectoryChangesW might be missed. The faster CPU machine can
detect
> the
> > > more events. And this version cannot detect the whole events.
> >
> > Anything we can do to improve it? I think we''re still doing
better
> > than the current C version in any case. :)
>
>
> In case of C version, I guess It could be improved by using thread. But in
> case of pure Ruby version, I have no idea.
> If someday Ruby support native windows thread, it would be much better.
Would wrapping that section of code in a Ruby thread help at all? Or
not really due to the nature of the code?

One other thing I''ve noticed is that it doesn''t appear
possible to
Ctrl-C out of that code, even if I setup a separate sleeper thread in
the main program. I have to kill it with the task manager. This is a
minor issue, but I thought I''d mention it in case you had any ideas.

Thanks,

Dan

2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> >
> >
> >
> > 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > On 8/9/07, Heesob Park <phasis at gmail.com> wrote:
> > > > Hi,
> > > >
> > > >
> > > > 2007/8/9, Daniel Berger <djberg96 at gmail.com>:
> > > > > > On 8/9/07, Heesob Park <phasis at gmail.com>
wrote:
> > > > >
> > > > > <snip>
> > > > >
> > > > > > > In my test with XP Home SP2 , it works fine.
> > > > > > > Do you modified both wait and get_file_aciton
with my version?
> > > > >
> > > > > BTW, I committed the changes (with some extra error
checking) into
> CVS
> > > > > so you can take a look.
> > > >
> > > >
> > > > Now I noticed some events ocurred between
GetQueuedCompletionStatus
> and
> > > > ReadDirectoryChangesW might be missed. The faster CPU
machine can
> detect
> > the
> > > > more events. And this version cannot detect the whole
events.
> > >
> > > Anything we can do to improve it? I think we''re still
doing better
> > > than the current C version in any case. :)
> >
> >
> > In case of C version, I guess It could be improved by using thread.
But
> in
> > case of pure Ruby version, I have no idea.
> > If someday Ruby support native windows thread, it would be much
better.
>
> Would wrapping that section of code in a Ruby thread help at all? Or
> not really due to the nature of the code?

I have tried several Ruby threading, but it didn''t help at all.
The basic idea is like this:

         arr = []
         Thread.new {
         while true
            bool = GetQueuedCompletionStatus(
               comp_port,
               qbytes,
               comp_key,
               @overlap,
               seconds
            )

            unless bool
              raise Error, get_last_error
            end

            break if comp_key.unpack(''L'').first == 0

            #yield get_file_action(fni) if block_given?
            arr.push(fni.dup)
            bool = ReadDirectoryChangesW(
               dir_handle,
               fni,
               fni.size,
               subtree,
               @filter,
               rbytes,
               @overlap,
               0
            )

            unless bool
               raise Error, get_last_error
            end
         end
         }
        while true
           while arr.length>0
             yield get_file_action(arr.pop) if block_given?
           end
           sleep 0.5
        end



> One other thing I''ve noticed is that it doesn''t appear
possible to
> Ctrl-C out of that code, even if I setup a separate sleeper thread in
> the main program. I have to kill it with the task manager. This is a
> minor issue, but I thought I''d mention it in case you had any
ideas.

Or press Ctrl-C and delete some monitoring file :)

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070810/b4335f79/attachment-0001.html