disk polling

dangph · Post by **dangph** » Thu Apr 30, 2009 9:22 am

I was looking at file system activity with Process Monitor from Sysinternals. I noticed that Zeus polls the currently active file every half a second. Shouldn't it be using FindFirstChangeNotification instead?

Post by **jussij** » Fri May 01, 2009 12:03 am

Zeus dates back to Windows 3.0 and back then functions like FindFirstChangeNotification did not exist in the Win16 API

dangph · Post by **dangph** » Fri May 01, 2009 2:33 am

That makes sense. It's not a good way to do it though. This is what Mark Russinovich, who knows about such things, has to say:

Polling is undesirable because it degrades performance. A thread that polls sleeps for the polling intervals, wakes up to perform its queries, and then goes back to sleep.

When the thread wakes up and the kernel schedules it the system must context-switch away from potentially useful work to the polling thread. The context switch results in an invalidation of the translation lookaside buffer (TLB – a high-speed cache of virtual to physical memory translations) of the processor and consumption of CPU resources as the thread executes. In addition, the thread references code and data as it executes, which the memory manager must make resident in physical memory, thus consuming memory resources.

Post by **jussij** » Fri May 01, 2009 3:19 am

Polling is undesirable because it degrades performance.

This Zeus code had to not degrade performance on a 40 Mhz machines with only 60 megs of memory, so on today’s modern machine it should run just fine

A thread that polls sleeps for the polling intervals, wakes up to perform its queries, and then goes back to sleep.

Zeus does not actually use this approach in it's polling code. Instead Zeus uses the WM_TIMER message so there is no need to call a sleep function.

I think this is one reason the performance is not degraded, since the Windows timer message is a low priority message and that means if Windows is busy, this message just doesn't get sent.

Having said all of this, just recently a bug was found in this file checking code that did cause a loss of performance:

http://www.zeusedit.com/forum/viewtopic.php?t=2417

Basically there was a bug in this file checked code where by in some situations, the Zeus WM_TIMER code actually took a long time to complete and this did degrade the performance of the machine.

What is strange is I suspect the this bug has probably been in Zeus for many years and it probably explains a why there have been the occasional Zeus is slow editing network files e-mails.

Cheers Jussi

dangph · Post by **dangph** » Sat May 02, 2009 1:57 am

jussij wrote:Zeus does not actually use this approach in it's polling code. Instead Zeus uses the WM_TIMER message so there is no need to call a sleep function.

I don't believe he was talking about a sleep function. When Zeus is sitting there doing nothing, waiting for input, the message-loop thread is in some sort of a waiting state. When it receives a message (keyboard input, WM_TIMER, etc), it "wakes up".

I think this is one reason the performance is not degraded, since the Windows timer message is a low priority message and that means if Windows is busy, this message just doesn't get sent.

The point is not that performance of Zeus itself is degraded--it probably isn't--the point is that performance of the system could be degraded. Russinovich gives his reasons for that. I think he must be right. I don't know what the size of the effect is though. I tried some benchmarking, and I couldn't find any noticeable effect.

Post by **jussij** » Mon May 04, 2009 5:55 am

When Zeus is sitting there doing nothing, waiting for input, the message-loop thread is in some sort of a waiting state. When it receives a message (keyboard input, WM_TIMER, etc), it "wakes up".

I not convinced

But I was curious so I ran a simple test

I create this batch file to start up 15 copies of Zeus with nothing open but one test.cpp file:

Code: Select all

cd "c:\Program Files\Zeus\"
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp
start zeus.exe -s -n -d c:\temp\test.cpp

Next I brough up the task manager and sure enough all 15 copies of Zeus where showing 00% CPU usage.

Next in one of the Zeus instances, I opened the Zeus workspace and ran a full tag rebuild and measured the time it took to complete.

With 15 copies of Zeus running the tag rebuild for three rebuilds took 45, 45, 46 seconds to complete.

I then closed all but one of the Zeus instances and re-ran the tags rebuild and sure enough three runs rebuilds took 45, 46, 45 seconds to complete.

So with 15 copies of Zeus all checking the same file 4 times a second I could not measure any slowdown in the time it took tag rebuild to complete.

But if there is any system degredation, it must be less than a second, which means it is of the order of 1/15 seconds per instance of Zeus, which from a user prespective is effectively zero

Cheers Jussi

PS: I also tried a larger number of instances of Zeus but with about 100 instances, Windows starts to run out of Windows handles and Zeus runs into errors when starting, making it a little hard to test

dangph · Post by **dangph** » Mon May 04, 2009 2:35 pm

I ran three three trials of three tests:

1. Without Zeus running.
2. With 15 instances of Zeus running, polling a single file.
3. With 15 instances of Zeus running with no files open.

Results:

1. 0% slowdown (baseline)
2. 9.1% slowdown
3. 5.5% slowdown

I can't explain result #3. It's probably just "noise".

I don't claim that these tests are conclusive. I think that they are merely interesting. If there is a real effect, then it must be very small. (Less than 1% slowdown per instance of Zeus, and possibly only for intensive memory copying operations.)

Here's my data:

no zeus:

Code: Select all

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2685.02 Mb/s
INTEGER   Scale:     2798.77 Mb/s
INTEGER   Add:       2941.82 Mb/s
INTEGER   Triad:     3081.24 Mb/s
---
INTEGER   AVERAGE:   2876.71 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2648.56 Mb/s
INTEGER   Scale:     2846.42 Mb/s
INTEGER   Add:       3102.25 Mb/s
INTEGER   Triad:     2911.16 Mb/s
---
INTEGER   AVERAGE:   2877.10 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2646.42 Mb/s
INTEGER   Scale:     2854.36 Mb/s
INTEGER   Add:       2978.91 Mb/s
INTEGER   Triad:     3060.90 Mb/s
---
INTEGER   AVERAGE:   2885.15 Mb/s

15 instances of zeus, polling a single file:

Code: Select all

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2562.00 Mb/s
INTEGER   Scale:     2576.10 Mb/s
INTEGER   Add:       2828.73 Mb/s
INTEGER   Triad:     2686.20 Mb/s
---
INTEGER   AVERAGE:   2663.26 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2434.47 Mb/s
INTEGER   Scale:     2484.31 Mb/s
INTEGER   Add:       2688.55 Mb/s
INTEGER   Triad:     2824.50 Mb/s
---
INTEGER   AVERAGE:   2607.96 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2400.94 Mb/s
INTEGER   Scale:     2428.34 Mb/s
INTEGER   Add:       2697.10 Mb/s
INTEGER   Triad:     2798.45 Mb/s
---
INTEGER   AVERAGE:   2581.21 Mb/s

15 instances of zeus, no polling:

Code: Select all

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2483.18 Mb/s
INTEGER   Scale:     2417.95 Mb/s
INTEGER   Add:       2976.02 Mb/s
INTEGER   Triad:     2707.50 Mb/s
---
INTEGER   AVERAGE:   2646.16 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2541.73 Mb/s
INTEGER   Scale:     2548.85 Mb/s
INTEGER   Add:       3104.99 Mb/s
INTEGER   Triad:     2809.33 Mb/s
---
INTEGER   AVERAGE:   2751.22 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32> C:\Users\dan\Downloads\ramspeed-win32\ramspeed-win32.exe -b 3
RAMspeed (Win32) v1.0.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-08

8Gb per pass mode

INTEGER   Copy:      2558.00 Mb/s
INTEGER   Scale:     2661.04 Mb/s
INTEGER   Add:       2861.33 Mb/s
INTEGER   Triad:     2988.69 Mb/s
---
INTEGER   AVERAGE:   2767.27 Mb/s

PS C:\Users\dan\Downloads\ramspeed-win32>

Post by **jussij** » Tue May 05, 2009 12:04 am

1. 0% slowdown (baseline)
2. 9.1% slowdown
3. 5.5% slowdown

These are the sort of numbers I would expect to see. I like the way your numbers show how Zeus is about twice as busy with the single file open, since that clearly shows effect of the polling.

Now let’s assume at worst the 15 copies are creating a 10% and 6% slowdown for the two cases.

So with no files open each copy is contributing 6/15 or a 0.4% to the slowdown and with one file open each copy is contributing 10/15 or a 0.666% to the slowdown.

That indicates the Zeus polling is adding 0.266% to the system slowdown and I can certainly believe that number.

What would be interesting to see is if just two versions of Zeus are running is the expected 1.2% slowdown still be measured

But if the Zeus polling is only adding 0.266% as measurable load on the system then I think that is a very good number

I've got all sorts of programs running on my computer that are nowhere near as generous

I can't explain result #3. It's probably just "noise".

I think this number is correct. Even with no files open the WM_TIMER event will still be firing and taking up CPU time. Zeus actually uses the timer event to do several other things and I think it is the load of these other things that this number is measuring.

Cheers Jussi
PS: Thank you for taking the time to start what has been a very interesting thread. I think your numbers are probably a very good indication of how Zeus does affect the system.