Apache woes and Drupal

Our server recently died a few times (swap exploded and didn't serve any requests), but since then we've changed a few figures in the configuration files and everything seems fine (for now).

But the apache2 processes seem to be extremely big, each process takes about 200-300MB, and we don't know how to reduce this or even if it is possible (or necessary) to reduce this figure?

So, there are currently 3 sites on this server having 3 separate drupal 6 installations:

the public site (using memcache 6.x-1.8)
the development site (only known to a hand full of people)
and another site (only known to a hand full of people)

And because only a few people know the other two sites I don't think it is a problem having these extra code bases on the same server. But, perhaps we're wrong about this? We have checked the enabled modules and couldn't find any to disable. We also disabled the UIs for some modules (e.g. views). What else can we do?

Some data:

top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
www-data  20   0  548m 316m 228m S    0  7.6  28:43.80 apache2
www-data  20   0  505m 307m 262m S    0  7.4   7:29.57 apache2
www-data  20   0  538m 295m 217m S    0  7.1  28:24.97 apache2
www-data  20   0  491m 292m 261m S    0  7.0  11:52.78 apache2
www-data  20   0  521m 290m 229m S    0  7.0   9:46.53 apache2
www-data  20   0  515m 290m 234m S    0  7.0   9:52.10 apache2
www-data  20   0  486m 287m 261m S    0  6.9  11:03.25 apache2
www-data  20   0  486m 287m 261m S    0  6.9  10:10.30 apache2
www-data  20   0  483m 285m 261m S    0  6.9  10:28.34 apache2
www-data  20   0  484m 284m 258m S    0  6.8   3:56.59 apache2
...

apache mods enabled

alias
auth_basic
authn_file
authz_default
authz_groupfile
authz_host
authz_user
autoindex
cgi
deflate
dir
env
mime
negotiation
php5
rewrite
setenvif
status

php mods enabled

apc
curl
gd
mcrypt
memcache
mysql
mysqli
pdo
pdo_mysql
suhosin
xapian

APC

Uptime 3 days, 8 hours and 38 minutes
Free: 3.7 MBytes (1.4%)
Used: 252.3 MBytes (98.6%)
Fragmentation: 0%

Cached Files 2314 (220.8 MBytes)
Hits 64020298
Misses 16860
Request Rate (hits, misses) 220.59 cache requests/second
Hit Rate 220.53 cache requests/second
Miss Rate 0.06 cache requests/second
Insert Rate 0.06 cache requests/second
Cache full count 6

memcached

Uptime 5 days, 18 hours and 5 minutes
Free: 25.4 MBytes (19.9%)
Used: 102.6 MBytes (80.1%)
Hits: 7828688 (36.7%)
Misses: 13507506 (63.3%)

memory at devel_shutdown

30 - 40 MB

This post based on drupal.org post here by our programmer.

Posted on: 8 June 2011

Tags:

28 Feb19:08

Maybe a problem with your

By apanag (not verified)

Maybe a problem with your apache configuration?

28 Feb19:26

Yes, we know, but what?

By mavergames

Which part of the Apache config is not optimally configured? Thanks

28 Feb20:18

Apache / Drupal Optimization

By dale brendan (not verified)

Are you able to disable any of those Apache modules? Disable any that you do not need. Also disable any and all Drupal modules that are not in use.

Perhaps try to reduce the number of MaxClients so that you are not exceeding your RAM with too many simultaneous connections.

Take a look at the Apache mod_expires module (Drupal knows how to use it automatically) to cache the pages in the User's browser for 2 weeks or unless updated.

I like to use Yahoo's YSlow extension also. That might help point you in the right direction as to what can be improved (i.e. too many DOM requests, etc, etc)

Move your .htaccess directives to httpd.conf - that way those directives aren't parsed on every request, but only when the server is started. Remember to than disable .htaccess lookups.

Enable MySQL caching or consider changing some of the tables to InnoDB from MyISAM (watchdog, cache tables, accesslog) so that you are row level locking instead of table locking--especially if your site is being heavily commented, has a lot of forum usage or is a social network install. Also, make sure Cron is being run on a regular basis (I like 6 times a day). Some of those previously mentioned tables can get out of hand very quickly.

Consider throttling portions of the site? Those are some of the things I learned from the Pro Dev book and are worth a try... Hope that helps!

28 Feb20:45

What happens with memcache?

By Anonymous (not verified)

What happens with memcache? The hit/miss rate is horrible.

28 Feb22:33

We're not exactly sure...

By mavergames

We're trying to figure this out. Any tips?

Thanks

28 Feb23:37

Varnish

By Anonymous (not verified)

I don't know how to reduce the memory usage by Apache, but I have read that Nginx is a much more memory friendly web server.

I would strongly recommend adding Varnish to the mix. It will require using the Pressflow variant of Drupal. Varnish is very impressive. It'll reduce the load on the web server substanially as all cached pages are served from it as well as all image and other assets. That frees up the web server for processing drupal requests as needed.

01 Mar01:40

Even more confused now

By sb (not verified)

The more I dig into this the more I get confused.
After a reload of apache top was showing this:
VIRT: 463m - 523m
RES: 11m - 120m
SHR: 5m - 53m

Could it be that the processes get that high because they never get killed?
One hour later ps was showing 60 apache processes with an average (calculated) RSS of 111MB but 'free -m' was still showing about 600MB of free memory of 4GB (and ~10MB swap).

@apanag
Yes it probably is, but could you give us an advise on how to set MinSpareServers, MaxSpareServers and MaxClients based on the data above as I don't know how 60 processes of 111MB each fit into 4GB.

@dale brendan
Thanks, I enabled mod_expires. We also want to reduce the MaxClients but I'm not sure which value as calculating it based on top or ps always gives me a value <35 but we already have 70 running (because MaxSpareServers is set to 70) without swapping and I'm confused.
MySQL is not tweaked so far but it is on a separate server and cron is running once every hour.

@Anonymous
It seems to be a problem with memcache 1.18 as with v1.15 we had a hit rate of about 90-95%.

01 Mar04:15

Apache is dead.

By mike (not verified)

nginx + php-fpm.

that's all you need. :)

01 Mar06:46

Memcache

By Trump Network (not verified)

I'd take a look at Memcache, this gave me more CPU performance boosed than anything else in my Drupal install.

01 Mar08:26

there might not be a problem

By dalin (not verified)

I think most of the comments here are bad advice - random suggestions that are vaguely related to performance but not the problem at hand. Figure out the problem before you begin surgery.

I do agree about reducing apache modules though. If you don't need cgi or all those auth* modules then remove them.

I think the main problem here is that there is in fact no problem. It looks to me like you are misreading `top`. When it comes to apache the VIRT, RES, and SHR columns don't give you very much useful information as they include memory shared between all apache processes (you'll notice that if you add up all the processes you get a total that is far more memory than you actually have). The main thing that makes those numbers look so big is APC. What you need to see is the DATA column. Press 'O' then 's'. You should see most apache processes using < 30m. There's probably some big ones that have served up a complex admin page, and some small ones that have served up static assets. Then shut down apache and see how much free memory you have. For ease of math lets say it's 1500MB.

(1500 free - 300 shared) / 30m per process = 40 processes

means you can set *Servers in your apache config to 40.

This is all assuming that the database runs on another machine so you don't have to try and juggle them both at once. If that's not the case it gets a little trickier.

If you need more assistance let us know.