win32utils devel - Aug 2007 - Need some help with pure Ruby win32-changenotify

Hi all,

I know it''s deprecated, but people seem to still be using 
win32-changenotify, so I thought I''d see if I could make it pure Ruby. 
Also, I thought it would be a good opportunity to test passing a custom 
Win32::Event object.

I''m mostly done I think, but I''m having trouble unraveling the
FILE_NOTIFY_INFORMATION struct buffer. Please check out the latest 
lib/win32/changenotify.rb from CVS and take a look at both the custom 
wait method and the get_file_action private method.

The notification is clearly getting picked up, but I''m just not 
unraveling the struct properly.

Here''s a little sample program you can use to test:

require ''win32/changenotify''
include Win32

filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
cn = ChangeNotify.new("C:\\", true, filter)
cn.wait(15){ |x|
    p x
} while true

Then, just make sure to modify some file within the 15 second wait period.

Thanks,

Dan

Hi,

2007/8/4, Daniel Berger <djberg96 at gmail.com>:
> Hi all,
>
> I know it''s deprecated, but people seem to still be using
> win32-changenotify, so I thought I''d see if I could make it pure
Ruby.
> Also, I thought it would be a good opportunity to test passing a custom
> Win32::Event object.
>
> I''m mostly done I think, but I''m having trouble
unraveling the
> FILE_NOTIFY_INFORMATION struct buffer. Please check out the latest
> lib/win32/changenotify.rb from CVS and take a look at both the custom
> wait method and the get_file_action private method.
>
> The notification is clearly getting picked up, but I''m just not
> unraveling the struct properly.
>
> Here''s a little sample program you can use to test:
>
> require ''win32/changenotify''
> include Win32
>
> filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> cn = ChangeNotify.new("C:\\", true, filter)
> cn.wait(15){ |x|
>    p x
> } while true
>
> Then, just make sure to modify some file within the 15 second wait period.
>
> Thanks,
>
> Dan

You may overlooked the API document that says the FileName field is a
variable-length field.

Here is working version of get_file_action:

     def get_file_action(fni)
         array  = []
         while true
            break if fni.nil? || fni[0,4].unpack(''L'')[0] == 0
            int_action = fni[4,4].unpack(''L'')[0]
            str_action = ''unknown''
            case int_action
               when FILE_ACTION_ADDED
                  str_action = ''added''
               when FILE_ACTION_REMOVED
                  str_action = ''removed''
               when FILE_ACTION_MODIFIED
                  str_action = ''modified''
               when FILE_ACTION_RENAMED_OLD_NAME
                  str_action = ''renamed old name''
               when FILE_ACTION_RENAMED_NEW_NAME
                  str_action = ''renamed new name''
            end
            len = fni[8,4].unpack(''L'').first
            buf = fni[12,len]
            file = wide_to_multi(buf)
            struct = ChangeNotifyStruct.new(str_action, file)
            array.push(struct)
            fni = fni[fni[0,4].unpack(''L'').first, -1] # Next
offset
         end

         array
      end

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070805/c95798f4/attachment.html

Hi again,

Yep, forgot about the variable length, thanks. I''ve committed your
changes. I also increased the buffer to 64k.

The problem now seems to be that it detects an event, but the
NextEntryOffset is 0. The result is that I see an empty array yielded
back most of the time when an event occurs. The C version works fine.

Here''s a simple script I''m using for testing:

Thread.new{ loop { sleep 0.1 } } # So ctrl-c works (might take 10 seconds)

filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
cn = ChangeNotify.new("C:\\", true, filter)
cn.wait(10){ |x|
   p x
} while true

Any ideas?

Thanks,

Dan

On 8/5/07, Heesob Park <phasis at gmail.com>
wrote:
> Hi,
>
>
> 2007/8/4, Daniel Berger <djberg96 at gmail.com>:
>
> > Hi all,
> >
> > I know it''s deprecated, but people seem to still be using
> > win32-changenotify, so I thought I''d see if I could make it
pure Ruby.
> > Also, I thought it would be a good opportunity to test passing a
custom
> > Win32::Event object.
> >
> > I''m mostly done I think, but I''m having trouble
unraveling the
> > FILE_NOTIFY_INFORMATION struct buffer. Please check out the latest
> > lib/win32/changenotify.rb from CVS and take a look at both the custom
> > wait method and the get_file_action private method.
> >
> > The notification is clearly getting picked up, but I''m just
not
> > unraveling the struct properly.
> >
> > Here''s a little sample program you can use to test:
> >
> > require ''win32/changenotify''
> > include Win32
> >
> > filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> > cn = ChangeNotify.new("C:\\", true, filter)
> > cn.wait(15){ |x|
> >    p x
> > } while true
> >
> > Then, just make sure to modify some file within the 15 second wait
period.
> >
> > Thanks,
> >
> > Dan
>
>
> You may overlooked the API document that says the FileName field is a
> variable-length field.
>
> Here is working version of get_file_action:
>
>      def get_file_action(fni)
>          array  = []
>          while true
>             break if fni.nil? || fni[0,4].unpack(''L'')[0]
== 0
>             int_action = fni[4,4].unpack(''L'')[0]
>             str_action = ''unknown''
>             case int_action
>                when FILE_ACTION_ADDED
>                   str_action = ''added''
>                when FILE_ACTION_REMOVED
>                    str_action = ''removed''
>                when FILE_ACTION_MODIFIED
>                   str_action = ''modified''
>                when FILE_ACTION_RENAMED_OLD_NAME
>                   str_action = ''renamed old name''
>                when FILE_ACTION_RENAMED_NEW_NAME
>                   str_action = ''renamed new name''
>             end
>             len = fni[8,4].unpack(''L'').first
>             buf = fni[12,len]
>             file = wide_to_multi(buf)
>             struct = ChangeNotifyStruct.new(str_action, file)
>             array.push(struct)
>             fni = fni[fni[0,4].unpack(''L'').first, -1] #
Next offset
>          end
>
>          array
>       end
>
> Regards,
>
> Park Heesob
>
> _______________________________________________
> win32utils-devel mailing list
> win32utils-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/win32utils-devel
>

Nevermind. The break condition needs to be moved to the end of the
loop. But, I think there''s another issue. Researching further....

Dan

On 8/5/07, Daniel Berger <djberg96 at gmail.com>
wrote:
> Hi again,
>
> Yep, forgot about the variable length, thanks. I''ve committed your
> changes. I also increased the buffer to 64k.
>
> The problem now seems to be that it detects an event, but the
> NextEntryOffset is 0. The result is that I see an empty array yielded
> back most of the time when an event occurs. The C version works fine.
>
> Here''s a simple script I''m using for testing:
>
> Thread.new{ loop { sleep 0.1 } } # So ctrl-c works (might take 10 seconds)
>
> filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> cn = ChangeNotify.new("C:\\", true, filter)
> cn.wait(10){ |x|
>    p x
> } while true
>
> Any ideas?
>
> Thanks,
>
> Dan
>
> On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
> > Hi,
> >
> >
> > 2007/8/4, Daniel Berger <djberg96 at gmail.com>:
> >
> > > Hi all,
> > >
> > > I know it''s deprecated, but people seem to still be
using
> > > win32-changenotify, so I thought I''d see if I could make
it pure Ruby.
> > > Also, I thought it would be a good opportunity to test passing a
custom
> > > Win32::Event object.
> > >
> > > I''m mostly done I think, but I''m having trouble
unraveling the
> > > FILE_NOTIFY_INFORMATION struct buffer. Please check out the
latest
> > > lib/win32/changenotify.rb from CVS and take a look at both the
custom
> > > wait method and the get_file_action private method.
> > >
> > > The notification is clearly getting picked up, but I''m
just not
> > > unraveling the struct properly.
> > >
> > > Here''s a little sample program you can use to test:
> > >
> > > require ''win32/changenotify''
> > > include Win32
> > >
> > > filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> > > cn = ChangeNotify.new("C:\\", true, filter)
> > > cn.wait(15){ |x|
> > >    p x
> > > } while true
> > >
> > > Then, just make sure to modify some file within the 15 second
wait period.
> > >
> > > Thanks,
> > >
> > > Dan
> >
> >
> > You may overlooked the API document that says the FileName field is a
> > variable-length field.
> >
> > Here is working version of get_file_action:
> >
> >      def get_file_action(fni)
> >          array  = []
> >          while true
> >             break if fni.nil? ||
fni[0,4].unpack(''L'')[0] == 0
> >             int_action = fni[4,4].unpack(''L'')[0]
> >             str_action = ''unknown''
> >             case int_action
> >                when FILE_ACTION_ADDED
> >                   str_action = ''added''
> >                when FILE_ACTION_REMOVED
> >                    str_action = ''removed''
> >                when FILE_ACTION_MODIFIED
> >                   str_action = ''modified''
> >                when FILE_ACTION_RENAMED_OLD_NAME
> >                   str_action = ''renamed old name''
> >                when FILE_ACTION_RENAMED_NEW_NAME
> >                   str_action = ''renamed new name''
> >             end
> >             len = fni[8,4].unpack(''L'').first
> >             buf = fni[12,len]
> >             file = wide_to_multi(buf)
> >             struct = ChangeNotifyStruct.new(str_action, file)
> >             array.push(struct)
> >             fni = fni[fni[0,4].unpack(''L'').first,
-1] # Next offset
> >          end
> >
> >          array
> >       end
> >
> > Regards,
> >
> > Park Heesob
> >
> > _______________________________________________
> > win32utils-devel mailing list
> > win32utils-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/win32utils-devel
> >
>

Ok, solved the remaining issues. I had to move the CloseHandle call in
the wait method above the yield/return. Also, I had to increase the
buffer size for the final string name.

Now I''m wondering if I should automatically prepend the @path to the
file name in the final struct. We could use File.expand_path for
relative paths.

What do you think?

Thanks,

Dan

On 8/5/07, Daniel Berger <djberg96 at gmail.com>
wrote:
> Nevermind. The break condition needs to be moved to the end of the
> loop. But, I think there''s another issue. Researching further....
>
> Dan
>
> On 8/5/07, Daniel Berger <djberg96 at gmail.com> wrote:
> > Hi again,
> >
> > Yep, forgot about the variable length, thanks. I''ve committed
your
> > changes. I also increased the buffer to 64k.
> >
> > The problem now seems to be that it detects an event, but the
> > NextEntryOffset is 0. The result is that I see an empty array yielded
> > back most of the time when an event occurs. The C version works fine.
> >
> > Here''s a simple script I''m using for testing:
> >
> > Thread.new{ loop { sleep 0.1 } } # So ctrl-c works (might take 10
seconds)
> >
> > filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> > cn = ChangeNotify.new("C:\\", true, filter)
> > cn.wait(10){ |x|
> >    p x
> > } while true
> >
> > Any ideas?
> >
> > Thanks,
> >
> > Dan
> >
> > On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
> > > Hi,
> > >
> > >
> > > 2007/8/4, Daniel Berger <djberg96 at gmail.com>:
> > >
> > > > Hi all,
> > > >
> > > > I know it''s deprecated, but people seem to still be
using
> > > > win32-changenotify, so I thought I''d see if I could
make it pure Ruby.
> > > > Also, I thought it would be a good opportunity to test
passing a custom
> > > > Win32::Event object.
> > > >
> > > > I''m mostly done I think, but I''m having
trouble unraveling the
> > > > FILE_NOTIFY_INFORMATION struct buffer. Please check out the
latest
> > > > lib/win32/changenotify.rb from CVS and take a look at both
the custom
> > > > wait method and the get_file_action private method.
> > > >
> > > > The notification is clearly getting picked up, but
I''m just not
> > > > unraveling the struct properly.
> > > >
> > > > Here''s a little sample program you can use to test:
> > > >
> > > > require ''win32/changenotify''
> > > > include Win32
> > > >
> > > > filter = ChangeNotify::FILE_NAME | ChangeNotify::DIR_NAME
> > > > cn = ChangeNotify.new("C:\\", true, filter)
> > > > cn.wait(15){ |x|
> > > >    p x
> > > > } while true
> > > >
> > > > Then, just make sure to modify some file within the 15
second wait period.
> > > >
> > > > Thanks,
> > > >
> > > > Dan
> > >
> > >
> > > You may overlooked the API document that says the FileName field
is a
> > > variable-length field.
> > >
> > > Here is working version of get_file_action:
> > >
> > >      def get_file_action(fni)
> > >          array  = []
> > >          while true
> > >             break if fni.nil? ||
fni[0,4].unpack(''L'')[0] == 0
> > >             int_action =
fni[4,4].unpack(''L'')[0]
> > >             str_action = ''unknown''
> > >             case int_action
> > >                when FILE_ACTION_ADDED
> > >                   str_action = ''added''
> > >                when FILE_ACTION_REMOVED
> > >                    str_action = ''removed''
> > >                when FILE_ACTION_MODIFIED
> > >                   str_action = ''modified''
> > >                when FILE_ACTION_RENAMED_OLD_NAME
> > >                   str_action = ''renamed old
name''
> > >                when FILE_ACTION_RENAMED_NEW_NAME
> > >                   str_action = ''renamed new
name''
> > >             end
> > >             len = fni[8,4].unpack(''L'').first
> > >             buf = fni[12,len]
> > >             file = wide_to_multi(buf)
> > >             struct = ChangeNotifyStruct.new(str_action, file)
> > >             array.push(struct)
> > >             fni =
fni[fni[0,4].unpack(''L'').first, -1] # Next offset
> > >          end
> > >
> > >          array
> > >       end
> > >
> > > Regards,
> > >
> > > Park Heesob
> > >
> > > _______________________________________________
> > > win32utils-devel mailing list
> > > win32utils-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/win32utils-devel
> > >
> >
>

Hi,

2007/8/6, Daniel Berger <djberg96 at gmail.com>:
>
> Ok, solved the remaining issues. I had to move the CloseHandle call in
> the wait method above the yield/return. Also, I had to increase the
> buffer size for the final string name.
>
> Now I''m wondering if I should automatically prepend the @path to
the
> file name in the final struct. We could use File.expand_path for
> relative paths.
>
> What do you think?

I think prepending the @path to the file name is better than printing the
relative paths.
And appliyng File.expand_path gives wrong result when running on different
path from monitoring path.

Thanks,
>
> Dan

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070806/82092a5a/attachment.html

On 8/5/07, Heesob Park <phasis at gmail.com>
wrote:
> Hi,
>
>
> 2007/8/6, Daniel Berger <djberg96 at gmail.com>:
> > Ok, solved the remaining issues. I had to move the CloseHandle call in
> > the wait method above the yield/return. Also, I had to increase the
> > buffer size for the final string name.
> >
> > Now I''m wondering if I should automatically prepend the @path
to the
> > file name in the final struct. We could use File.expand_path for
> > relative paths.
> >
> > What do you think?
>
>
> I think prepending the @path to the file name is better than printing the
> relative paths.
> And appliyng File.expand_path gives wrong result when running on different
> path from monitoring path.
Ah, that''s right, thanks. Ok, I''ve simply join''d the
@path to the file
name. That''s now in CVS.

Oh, one thing that I''ve noticed is that it sometimes returns the
filename as "4913". I know I''ve seen this issue come up at
some point
before (with the C code, too), but I can''t remember what the answer
was, and my Google skills are failing me.

Do you happen to remember what causes that?

Thanks,

Dan

Hi,
----- Original Message ----- 
From: "Daniel Berger" <djberg96 at gmail.com>
To: "Development and ideas for win32utils projects"
<win32utils-devel at rubyforge.org>
Sent: Monday, August 06, 2007 1:46 PM
Subject: Re: [Win32utils-devel] Need some help with pure Rubywin32-changenotify

> On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
>> Hi,
>>
>>
>> 2007/8/6, Daniel Berger <djberg96 at gmail.com>:
>> > Ok, solved the remaining issues. I had to move the CloseHandle
call in
>> > the wait method above the yield/return. Also, I had to increase
the
>> > buffer size for the final string name.
>> >
>> > Now I''m wondering if I should automatically prepend the
@path to the
>> > file name in the final struct. We could use File.expand_path for
>> > relative paths.
>> >
>> > What do you think?
>>
>>
>> I think prepending the @path to the file name is better than printing
the
>> relative paths.
>> And appliyng File.expand_path gives wrong result when running on
different
>> path from monitoring path.
> 
> Ah, that''s right, thanks. Ok, I''ve simply join''d
the @path to the file
> name. That''s now in CVS.
> 
> Oh, one thing that I''ve noticed is that it sometimes returns the
> filename as "4913". I know I''ve seen this issue come up
at some point
> before (with the C code, too), but I can''t remember what the
answer
> was, and my Google skills are failing me.
> 
> Do you happen to remember what causes that?
> 
Sorry, I don''t remember that issue. and I never met such case.

But I noticed other two issues.
First, sometimes filename has trailing garbage characters.
It can be fixed by modifing changenotify.rb line # 171 to
  file = fni[12,len] + "\0\0"   # add null char

Second, sometimes some events missed when two or more files were removed or
added at the same time.
I''m not sure it is a known issue.

Regards,
Park Heesob

Hi,
 
> > On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
<snip>
 
> > Oh, one thing that I''ve noticed is that it sometimes returns
the
> > filename as "4913". I know I''ve seen this issue
come up at
> some point 
> > before (with the C code, too), but I can''t remember what the
answer
> > was, and my Google skills are failing me.
> > 
> > Do you happen to remember what causes that?
> > 
> Sorry, I don''t remember that issue. and I never met such case.
I was able to see this just by setting up monitoring on the current
directory, creating a file via gvim, and editing that file while that
directory was being watched.
 
> But I noticed other two issues.
> First, sometimes filename has trailing garbage characters.
> It can be fixed by modifing changenotify.rb line # 171 to
>   file = fni[12,len] + "\0\0"   # add null char
Thanks, I''ll commit that change today.
 
> Second, sometimes some events missed when two or more files 
> were removed or added at the same time. I''m not sure it is a 
> known issue.
I seem to recall that win32-changenotify could never handle a large
number of simultaneous events. I think this was a limitation of the
ReadDirectoryChangesW function, but I''d have to do more research.

That being said, it does seem like the C version picks up more events
than the pure Ruby version does. So, there''s a problem with the pure
Ruby win32-changenotify code, a problem with the pure Ruby win32-ipc
code, or Win32API is just not quick enough to pick up more than a few
events at a time.

Regards,

Dan


This communication is the property of Qwest and may contain confidential or
privileged information. Unauthorized use of this communication is strictly 
prohibited and may be unlawful.  If you have received this communication 
in error, please immediately notify the sender by reply e-mail and destroy 
all copies of the communication and any attachments.

Hi,

2007/8/7, Berger, Daniel <Daniel.Berger at
qwest.com>:
>
> Hi,
>
> > > On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
>
> <snip>
>
> > > Oh, one thing that I''ve noticed is that it sometimes
returns the
> > > filename as "4913". I know I''ve seen this
issue come up at
> > some point
> > > before (with the C code, too), but I can''t remember what
the answer
> > > was, and my Google skills are failing me.
> > >
> > > Do you happen to remember what causes that?
> > >
> > Sorry, I don''t remember that issue. and I never met such
case.
>
> I was able to see this just by setting up monitoring on the current
> directory, creating a file via gvim, and editing that file while that
> directory was being watched.

If "4913" file already exists, the filename would be "5036".

I investigate the gvim source code, and I found "4913" in fileio.c.
It generate temporary file starting from 4913 increasing by 123.

So the "4913" is just a filename generated by gvim.

Regards,

Park Heesob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/win32utils-devel/attachments/20070807/af2307c2/attachment.html

On 8/6/07, Heesob Park <phasis at gmail.com>
wrote:
> Hi,
>
>
> 2007/8/7, Berger, Daniel <Daniel.Berger at qwest.com>:
> > Hi,
> >
> > > > On 8/5/07, Heesob Park <phasis at gmail.com> wrote:
> >
> > <snip>
> >
> > > > Oh, one thing that I''ve noticed is that it
sometimes returns the
> > > > filename as "4913". I know I''ve seen this
issue come up at
> > > some point
> > > > before (with the C code, too), but I can''t remember
what the answer
> > > > was, and my Google skills are failing me.
> > > >
> > > > Do you happen to remember what causes that?
> > > >
> > > Sorry, I don''t remember that issue. and I never met such
case.
> >
> > I was able to see this just by setting up monitoring on the current
> > directory, creating a file via gvim, and editing that file while that
> > directory was being watched.
>
>
> If "4913" file already exists, the filename would be
"5036".
>
> I investigate the gvim source code, and I found "4913" in
fileio.c.
> It generate temporary file starting from 4913 increasing by 123.
>
> So the "4913" is just a filename generated by gvim.
Oh, heh, good to know.

Thanks,

Dan