There are few options for sharing webroots among multiple webnodes. This is especially true for Drupal, which has a files directory that could be constantly diverging on each webnode. Usually people decide to go with NFSv3. This is what we do on drupal.org and most of my clients do at Tag1. Sadly, this is not really a decision on merit, but on a lack of other options. NFSv3 is pretty much like communicating with two cans and a string.
A major problem with running Apache/PHP/Drupal on NFS in particular is lstat. Drupal uses a PHP function called include_once and uses it often. This function includes other PHP files, but does it only once. (Truly brilliance in naming) It differentiates between files with their full path and it finds their full path with lstat. Doesn't seem like much, but this leads to a lot of lstats and these calls are not at all cached using NFSv3. Using strace to count them on one Apache run leads to results like this:
47.71 0.847578 3 267280 3192 lstat
19.86 0.352825 10 35063 4101 open
8.28 0.147049 2 66478 16474 access
6.83 0.121374 4 34190 1707 stat
5.21 0.092609 3 33295 260 read
The important columns are the first (percentage of time in syscall) and the 4th (number of calls).
Enter PHP 5.1's realpath cache. Again, PHP developers' naming genius shines through. This feature caches the real path of PHP files and it is used to cache the checks that include_once performs. Sadly, the default php.ini sets this to 16K and makes it close to useless. It is a fairly new feature, so isn't that well known yet, but it shows very real results. Changing this setting to 800K on drupal.org's webnodes gave us this strace result:
58.80 5.179350 31 166219 12408 open
12.51 1.101776 5 223492 4308 read
5.49 0.483280 3 158441 20494 stat
4.90 0.431528 3 168837 36562 access
3.72 0.327620 2 154757 write
2.76 0.242768 1 233763 20582 lstat
2.39 0.210757 46 4595 writev
That is a massive difference. In my testing on another site, this setting increased the pages served by 3-4 per second.
Long Live The Real Path Cache
Twitter
RSS
Comments
Its a very useful
Its a very useful information Thanks for sharing.Good job.
opcode cache?
Just curious, are you using an opcode cache? I recently discovered something similar with APC.
Setting
apc.stat = 0disabled stat checking entirely, which made huge performance difference. As long as you set the cache size to a large enough value, the filesystem is almost never touched (I found 64M is about right for a complex Drupal site).APC and include_once
Actually, someone else just mentioned this to me as well. Yes, we do use APC and something like it or eaccel is absolutely required for a performance-oriented drupal site. However, there is a problem with the stat setting and the include_once_override setting (which APC has developed specifically at this problem). You can see a description of what these settings do here: http://t3.dotgnu.info/blog/php/include_once-mostly-harmless.html
To summarize, APC tries to rewrite the include_once lines to be full path includes and get around this problem. However, Drupal uses dynamic includes and as such APC cannot know at compile time what to rewrite the includes to, so it cannot optimize drupal's include_once calls. The stat setting does help, but in our testing doesn't really prevent the problem and lstat is still far and away the biggest problem with APC's optimizations turned on.
Drupal 7 will fix this
Drupal 7 will fix this, because it defines a DRUPAL_ROOT in index.php and sets it to getcwd(). It then uses this to prefix all the include_once/require_once.
So, that will help.
Another APC configuration parameter that is promising, is apc.include_once_override. If you can measure that too, that would be awesome! ;-)
D7
Yes, d7 should help a lot with this. I was actually talking about the include_once override in the post above yours. For various reasons (dyn includes), Drupal doesn't benefit from it at all :(. Was disapointing to discover.