Good afternoon all, I'm a web developer (c#) who also manages a small hosting environment. Roughly, our configuration is: Two IIS7 (Windows Server 2008 Web) servers which share configuration data and site files from a SMB share located on a ubuntu server. The ubuntu server is currently running 3.0.28a which is quite old, but read on. We've been having a problem with the change notifications from the ubuntu filesystem reaching IIS. The symptoms are that if a file is changed on disk (such as a binary/dll or web.config or similar) are changed, IIS does not flush the appdomain and reload these files, it continues on using the old files until either 1) an app pool reset is performed (typically bad for performance and not something you should do all the time) 2) samba is restarted (definitely not recommended) 3) the file that was changed is deleted and then added again My test case is a simple .net based application which uses code to set the colour of a box. I compiled three different binaries, one for red/green/blue. I can then switch the binaries out with a simple cp (e.g. cp red/bin/* test/bin - where test is the wwwroot for my test site) When I update the binary, IIS doesn't reload it. I have tested many different configurations. IIS7 accessing XP file share - works IIS7 accessing Win7 file share - works IIS7 accessing 2003 file share - works IIS7 accessing 2008 file share - works IIS7 accessing ubuntu samba 3.0.28a share - doesn't work IIS7 accessing ubuntu samba 4.?? beta share - doesn't work IIS7 accessing ubuntu samba 4 running SMB2 experimental - doesn't work IIS7 accessing ubuntu 3.5.6 (built from source) share - doesn't work Basically, if the share provider is windows (SMB1 or SMB2), it works - if it's linux/samba it doesn't. I've done a lot of reading on this topic, and I'm pretty sure that IIS uses a different change notification mechanism to that of your standard file browser window; that is, if I have a explorer window looking at the share, I see the modify date change immediately, however IIS doesn't realise and load the new binary. I've done some debugging and at one point I saw messages from inside notify.c; Client only wanted %d bytes, trying to marshall %d bytes\n; which results in notify_marshall_changes returning false. As far as I can tell, this bug has existed for quite some time and probably hasn't been diagnosed correctly. I tested explicitly today using a windows 7 desktop (IIS7 installed from programs and features) with both a local folder (to first confirm that my binaries would work as expected then a freshly installed 10.10 ubuntu with samba 3.5.4 (from launchpad/aptitude) followed by a built from source 3.5.6. I tested with 3.5.6 to confirm it wasn't the 4 byte boundry bug recently fixed. The ubuntu server is running on a VM locally. IIS7 works much smoother with UNC paths than IIS6, except you have to use caspol to trust the UNC path if you want to execute asp.net binaries. I'm not very familiar with C code, especially on linux environments so i'm not sure how to test/provide the information you need to diagnose this bug. I am happy to file a bug report, but without a working IIS7 installation, it will be very hard to replicate. It's not reasonable to expect the samba developers to install and test an IIS7 install so I would be happy to work with someone on the problem. I can verify the problem in the most simple test case, and I have tested a reasonable subset of samba installs. My thoughts are it's something to do with watching a folder of files, and this not propogating correctly - as in my testing the notifications reached samba and appeared to be sent to windows, just IIS didn't respond to them. This thread from 2003 describes a very similar issue with classic ASP (the precursor to .net) http://www.mail-archive.com/samba at lists.samba.org/msg12479.html This issue would be great to fix, as it's pretty much the only stumbling block in using SMB as a file share for IIS, we've been doing this for about a year now - and it works perfectly once you set it up - with the exception of the change notifications. We've been working around it by recycling application pools but as mentioned above, this isn't really a long-term solution and it means 3rd party developers need way more access to the hosting environment (to recycle pools etc) than I consider practical. Appreciate any advice/help - and happy to provide more debugging information when required, Gareth Evans Sniper Systems Ltd New Zealand
Good afternoon all, I'm a web developer (c#) who also manages a small hosting environment. Roughly, our configuration is: Two IIS7 (Windows Server 2008 Web) servers which share configuration data and site files from a SMB share located on a ubuntu server. The ubuntu server is currently running 3.0.28a which is quite old, but read on. We've been having a problem with the change notifications from the ubuntu filesystem reaching IIS. The symptoms are that if a file is changed on disk (such as a binary/dll or web.config or similar) are changed, IIS does not flush the appdomain and reload these files, it continues on using the old files until either 1) an app pool reset is performed (typically bad for performance and not something you should do all the time) 2) samba is restarted (definitely not recommended) 3) the file that was changed is deleted and then added again My test case is a simple .net based application which uses code to set the colour of a box. I compiled three different binaries, one for red/green/blue. I can then switch the binaries out with a simple cp (e.g. cp red/bin/* test/bin - where test is the wwwroot for my test site) When I update the binary, IIS doesn't reload it. I have tested many different configurations. IIS7 accessing XP file share - works IIS7 accessing Win7 file share - works IIS7 accessing 2003 file share - works IIS7 accessing 2008 file share - works IIS7 accessing ubuntu samba 3.0.28a share - doesn't work IIS7 accessing ubuntu samba 4.?? beta share - doesn't work IIS7 accessing ubuntu samba 4 running SMB2 experimental - doesn't work IIS7 accessing ubuntu 3.5.6 (built from source) share - doesn't work Basically, if the share provider is windows (SMB1 or SMB2), it works - if it's linux/samba it doesn't. I've done a lot of reading on this topic, and I'm pretty sure that IIS uses a different change notification mechanism to that of your standard file browser window; that is, if I have a explorer window looking at the share, I see the modify date change immediately, however IIS doesn't realise and load the new binary. I've done some debugging and at one point I saw messages from inside notify.c; Client only wanted %d bytes, trying to marshall %d bytes\n; which results in notify_marshall_changes returning false. As far as I can tell, this bug has existed for quite some time and probably hasn't been diagnosed correctly. I tested explicitly today using a windows 7 desktop (IIS7 installed from programs and features) with both a local folder (to first confirm that my binaries would work as expected then a freshly installed 10.10 ubuntu with samba 3.5.4 (from launchpad/aptitude) followed by a built from source 3.5.6. I tested with 3.5.6 to confirm it wasn't the 4 byte boundry bug recently fixed. The ubuntu server is running on a VM locally. IIS7 works much smoother with UNC paths than IIS6, except you have to use caspol to trust the UNC path if you want to execute asp.net binaries. I'm not very familiar with C code, especially on linux environments so i'm not sure how to test/provide the information you need to diagnose this bug. I am happy to file a bug report, but without a working IIS7 installation, it will be very hard to replicate. It's not reasonable to expect the samba developers to install and test an IIS7 install so I would be happy to work with someone on the problem. I can verify the problem in the most simple test case, and I have tested a reasonable subset of samba installs. My thoughts are it's something to do with watching a folder of files, and this not propogating correctly - as in my testing the notifications reached samba and appeared to be sent to windows, just IIS didn't respond to them. This thread from 2003 describes a very similar issue with classic ASP (the precursor to .net) http://www.mail-archive.com/samba at lists.samba.org/msg12479.html This issue would be great to fix, as it's pretty much the only stumbling block in using SMB as a file share for IIS, we've been doing this for about a year now - and it works perfectly once you set it up - with the exception of the change notifications. We've been working around it by recycling application pools but as mentioned above, this isn't really a long-term solution and it means 3rd party developers need way more access to the hosting environment (to recycle pools etc) than I consider practical. Appreciate any advice/help - and happy to provide more debugging information when required, Gareth Evans Sniper Systems Ltd New Zealand
Since I posted the above, i've done more testing and research.
Using my binary test files, I've captured a log from smbd as I change files
on disk - these changes don't reach IIS.
I'm pretty sure this is the relevant portion where the notification fires:
inotify_dispatch called with mask=2, name=[ChangeNotifyTesting.dll]
inotify_dispatch: ne.action = 3, ne.path = ChangeNotifyTesting.dll
filter_match: e->mask=2, w->mask=210003c6, w->filter=17
sys_notify_callback called with action=3, for ChangeNotifyTesting.dll
notify_callback called for client/site.co.nz/htdocs/Test/bin
0000 offset: 00000000
0004 action: 00000003
0008 namelen: 0000002e
000c name: C.h.a.n.g.e.N.o.t.i.f.y.T.e.s.t.i.n.g...d.l.l...
Client only wanted 32 bytes, trying to marshall 58 bytes
size=71
--snip-- (looks like the notification is set up again after it's fired)
[0000] 17 00 00 00 D8 2A 00 00 .....*..
call_nt_transact_notify_change
call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|ATTRIBUTES|LAST_WRITE,
recursive = 0
Locking key 6E6F7469667920617272
Allocated locked data 0x0x7f9cb4c5e450
inotify_add_watch for
/srv/web_store/microsoft/client/site.co.nz/htdocs/Testmask 210003c6
returned wd 19
Unlocking key 6E6F7469667920617272
change_notify_add_request: Adding request for client/site.co.nz/htdocs/Test:
max_param = 32
Taking this a bit further, i've eliminated IIS entirely for the time being,
with a C# application using FileSystemWatcher (available if a samba
developer would like a copy to test with)
watching the UNC path;
I find the following:
Modifications of files show information in the samba log, but do not reach
the FileSystemMonitor
Deleting a file successfully reaches FileSystemMonitor
Creating a file successfully reaches FileSystemMonitor
Moving a file into another (overwrite) does not reach FileSystemMonitor
(same as modification)
Here's the output;
commands over SSH are prefixed with >,
comments with //,
log output with *,
Monitor App with $
Registered ChangeNotifications for
\\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test;
Recursive=True> touch fish
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
* Transaction 24 of length 88 (0 toread)
* switch message SMBnttrans (pid 1366) conn 0x7f0f88645290
* setting sec ctx (1001, 1001) - sec_ctx_stack_ndx = 0
* call_nt_transact_notify_change
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish Created
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish
Changed> nano fish
//edit file and press ctrl-o, ctrl-x
$ No Output
* No Output> cp ../Green/bin/ChangeNotifyTesting.dll fish
$ No Output
* No Output> rm fish
* Transaction 25 of length 88 (0 toread)
* switch message SMBnttrans (pid 1366) conn 0x7f0f88645290
* setting sec ctx (1001, 1001) - sec_ctx_stack_ndx = 0
* call_nt_transact_notify_change
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish Deleted
Recursive true/false makes no difference (I was making the changes in the
folder I had the notification registered for)
(-r argument on testing app)
FileSystemWatcher internally uses the ReadDirectoryChangesW API function:
http://msdn.microsoft.com/en-us/library/aa365465%28v=vs.85%29.aspx
And some googling reveals that ReadDirectoryChangesW doesn't work with
samba, and quite a few people describe this issue.
Some more googling yeilded that IIS makes heavy use of ReadDirectoryChangesW
internally, so this is quite likely the root cause of the issue.
This blog post suggests that there are some errors returned, and also
describes something I saw in testing with IIS which is after a while, the
notifications just stopped firing completely- I think this is due to
repeated errors being returned, and as a result, the notification system is
shut down:
http://leontheprogrammer.blogspot.com/2010/04/readdirectorychangesw-returns.html
This is a post from 2003, where it was reported that ReadDirectoryChangesW
wasn't working and a workaround, but unfortunately, because I can't
modify
anything in IIS itself (or FileSystemWatcher for that matter) it's not the
correct solution here.
http://lists.samba.org/archive/samba-technical/2003-July/030902.html
I don't think there's much more testing I can do here- it seems like the
_changes_ to the folder never show up in samba, and as a result, don't get
pushed to the watcher; i'm only using debug 3; 10 is a bit hard to read :)
I used inotify-tools and manually verified that the expected inotify events
fire when I perform the various operations.
That is, I performed the above commands and found that the inotifywait (I
also used inotifywatch but I couldn't get line by line output as changes
happened with this one) and observed the expected sequence of CREATE, OPEN,
MODIFY, DELETE was raised.
This makes me think that when Samba receives the change notification from
ReadDirectoryChangesW, it doesn't successfully either
a) decode the request parameters
b) translate the request parameters to the underlying inotify arguments (and
thus those changes don't get monitored)
For completeness' sake, I forced recursive to true in nttrans.c and then
tested - I can confirm that the behaviour/output did not change.
I hope this is enough information for someone to either investigate the
issue or to ask for more information regarding a specific test
Gareth
Sniper Systems Ltd
New Zealand
On Sun, Jan 9, 2011 at 3:13 PM, Gareth Evans <agrath at gmail.com> wrote:
> Good afternoon all,
>
> I'm a web developer (c#) who also manages a small hosting environment.
> Roughly, our configuration is:
>
> Two IIS7 (Windows Server 2008 Web) servers which share configuration data
> and site files from a SMB share located on a ubuntu server.
>
> The ubuntu server is currently running 3.0.28a which is quite old, but read
> on.
>
> We've been having a problem with the change notifications from the
ubuntu
> filesystem reaching IIS.
>
> The symptoms are that if a file is changed on disk (such as a binary/dll or
> web.config or similar) are changed, IIS does not flush the appdomain and
> reload these files, it continues on using the old files until either
>
> 1) an app pool reset is performed (typically bad for performance and not
> something you should do all the time)
> 2) samba is restarted (definitely not recommended)
> 3) the file that was changed is deleted and then added again
>
> My test case is a simple .net based application which uses code to set the
> colour of a box.
> I compiled three different binaries, one for red/green/blue.
>
> I can then switch the binaries out with a simple cp
> (e.g. cp red/bin/* test/bin - where test is the wwwroot for my test site)
>
> When I update the binary, IIS doesn't reload it.
>
> I have tested many different configurations.
>
> IIS7 accessing XP file share - works
> IIS7 accessing Win7 file share - works
> IIS7 accessing 2003 file share - works
> IIS7 accessing 2008 file share - works
> IIS7 accessing ubuntu samba 3.0.28a share - doesn't work
> IIS7 accessing ubuntu samba 4.?? beta share - doesn't work
> IIS7 accessing ubuntu samba 4 running SMB2 experimental - doesn't work
> IIS7 accessing ubuntu 3.5.6 (built from source) share - doesn't work
>
> Basically, if the share provider is windows (SMB1 or SMB2), it works - if
> it's linux/samba it doesn't.
>
> I've done a lot of reading on this topic, and I'm pretty sure that
IIS uses
> a different change notification mechanism to that of your standard file
> browser window; that is, if I have a explorer window looking at the share,
I
> see the modify date change immediately, however IIS doesn't realise and
load
> the new binary.
>
> I've done some debugging and at one point I saw messages from inside
> notify.c; Client only wanted %d bytes, trying to marshall %d bytes\n; which
> results in notify_marshall_changes returning false.
>
> As far as I can tell, this bug has existed for quite some time and probably
> hasn't been diagnosed correctly.
>
> I tested explicitly today using a windows 7 desktop (IIS7 installed from
> programs and features) with both a local folder (to first confirm that my
> binaries would work as expected then a freshly installed 10.10 ubuntu with
> samba 3.5.4 (from launchpad/aptitude) followed by a built from source
3.5.6.
>
> I tested with 3.5.6 to confirm it wasn't the 4 byte boundry bug
recently
> fixed.
>
> The ubuntu server is running on a VM locally.
>
> IIS7 works much smoother with UNC paths than IIS6, except you have to use
> caspol to trust the UNC path if you want to execute asp.net binaries.
>
> I'm not very familiar with C code, especially on linux environments so
i'm
> not sure how to test/provide the information you need to diagnose this bug.
>
> I am happy to file a bug report, but without a working IIS7 installation,
> it will be very hard to replicate.
> It's not reasonable to expect the samba developers to install and test
an
> IIS7 install so I would be happy to work with someone on the problem.
>
> I can verify the problem in the most simple test case, and I have tested a
> reasonable subset of samba installs.
>
> My thoughts are it's something to do with watching a folder of files,
and
> this not propogating correctly - as in my testing the notifications reached
> samba and appeared to be sent to windows, just IIS didn't respond to
them.
> This thread from 2003 describes a very similar issue with classic ASP (the
> precursor to .net)
> http://www.mail-archive.com/samba at lists.samba.org/msg12479.html
>
> This issue would be great to fix, as it's pretty much the only
stumbling
> block in using SMB as a file share for IIS, we've been doing this for
about
> a year now - and it works perfectly once you set it up - with the exception
> of the change notifications.
>
> We've been working around it by recycling application pools but as
> mentioned above, this isn't really a long-term solution and it means
3rd
> party developers need way more access to the hosting environment (to
recycle
> pools etc) than I consider practical.
>
> Appreciate any advice/help - and happy to provide more debugging
> information when required,
>
> Gareth Evans
> Sniper Systems Ltd
> New Zealand
>
On Sun, Jan 09, 2011 at 11:00:07AM +0100, Volker Lendecke wrote:> On Sun, Jan 09, 2011 at 10:49:22PM +1300, Gareth Evans wrote: > > Sorry, what command should I use to start smbd and capture the log? > > If I use -F > filename.log, it seems to just daemonize and not log anything > > (at least not to the piped log file or the log file in /var/log/samba) > > Just start is as a normal daemon. Set "debug level = 10" and > "max log size = 100000" in the smb.conf file and just start > "smbd -D". Then it should log into the log file you set. If > you compiled just with ./configure;make, it would show up > under /usr/local/samba/var.Ah, "debug hires timestamp = yes" might also help correlating the network trace and the log. Thanks, Volker Lendecke -- SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen phone: +49-551-370000-0, fax: +49-551-370000-9 AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
sorry, accidentally hit reply instead of reply all On Fri, Jan 14, 2011 at 10:05 AM, Gareth Evans <agrath at gmail.com> wrote:> Okay I guess the recursive thing makes sense, I hope IIS establishes > watches on the specific directories it's interested in. > The tree could indeed be very deep. > I guess the only way to know is to test with it once I can repeat the same > operations as you and get the same output. > > I don't know what i'm doing differently, but it certainly didn't detect the > move/copy/modify operations when I tested it. > Would the local server filesystem make a difference? I doubt it, but worth > asking > > There has to be something different between our two tests, there can't be > anything I'm doing wrong specifically that causes it to not work, it's a > pretty simple test. > I was definitely making the changes in the directory that I had the > notify/watch established for > > Do you see the 'client only requested 32 bytes marshalling 58 (?) bytes' > message in your log output? I always saw that error when I was testing and > it didn't work. > From reading the source, that error is generated immediately before a > return false from the notify response. > How long is your UNC path? Mine was a reasonable length: > That's 52 bytes by itself, plus a filename could be where that ~58 bytes > number came from > \\smbtest\mwh-webstore\clientname\site.co.nz\htdocs > > I think that if I triggered the changes from a samba share that it was > handled correctly, but the changes on the shell didn't. > I'd need to test this specific scenario to confirm if this was the case. > > In our scenario, our FTP servers are modifying files via a mounted NFS > share, or things are changed directly on the console (e.g. unzipping an > archive of site files) > > I'm at the office right now, and my test ubuntu server with 3.5.6 is on it > is at home - I can go there if you need more testing info or want me to try > specific scenarios. > > > Gareth > > > On Fri, Jan 14, 2011 at 9:55 AM, Volker Lendecke < > Volker.Lendecke at sernet.de> wrote: > >> On Fri, Jan 14, 2011 at 09:39:13AM +1300, Gareth Evans wrote: >> > Looks like you're using a mapped drive letter rather than a UNC path >> -that >> > could be contributing. >> >> Yes. Just tried with a UNC path. Same result. >> >> > It sounds like what you're doing to test is fairly equivilent to what I >> was >> > doing but in this case, i'm seeing what I would have expected to see in >> my >> > testing. >> > >> > What OS is your client machine? XP? Win7? >> >> Win7 for this test. >> >> > If a smb client requests a recursive watch, such as IIS would (because >> it >> > would register the notify on the root folder and expect to be told if >> > something in bin subfolder changes) does samba register notifications in >> a >> > recursive way (i.e. subfolders) >> >> It does internally. This means that we get recursive change >> notifies that are triggered via other CIFS clients. >> >> For example the >> >> File: z:\mput\file00000168.exe Deleted >> >> from my last message was triggered by smbclient connected to >> the same share. Doing a "rm /data/tmp/mput/xxx" on the Linux >> server shell would not have triggered it. >> >> We do not get recursive changes triggered by local Unix >> clients. Given the non-recursive nature of the inotify API, >> his would be potentially prohibitively expensive. We might >> have to walk a directory tree arbitrarily large. >> >> With best regards, >> >> Volker Lendecke >> >> -- >> SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen >> phone: +49-551-370000-0, fax: +49-551-370000-9 >> AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen >> > >