httpd eating up resorces!!!

[dperez]

Hi,
I'm running an Apache webserver, and after I do a reboot, I'm up to only about 25% of physical memory usage. By the end of the day, the server is up to 99% and the swap file kicks in. Is there a reason for this? The server has 512M of ram and it runs on an AMD Duron 1Ghz with a 1G swap partition.

Any info would be appreciated.

Thanks
Dennys

[dperez]

Anyone has any tips on what could be going on?

Thanks
Dennys

[Seminole]

What flavor of Linux and what version of Apache?

I had some issues with my comp eating resources with RH8 and Apache 2.0. My solution was to move my website from my 1.3Ghz Athlon with 512M of RAM over to my 433MHz Celery with 512M of RAM. Don't know why, but I haven't had any issues since.

Maybe because it isn't my main machine? I don't do anything on the 433 except surf the net and run the server. On my main machine, I do pretty much everything.

Perhaps someone here knows but as soon as I stopped running Apache, sendmail, and MySQL on my main comp, the resources have been just fine...

[dperez]

I'm running RedHat 8 and Apache 2.0, but the server is also running qmail, mysql and proftpd services.

Thanks
Dennys

[theN]

Originally posted by dperez
Hi,
I'm running an Apache webserver, and after I do a reboot, I'm up to only about 25% of physical memory usage. By the end of the day, the server is up to 99% and the swap file kicks in. Is there a reason for this? The server has 512M of ram and it runs on an AMD Duron 1Ghz with a 1G swap partition.
* Are you absolutely sure that its Apache which's hogging 99% of RAM and nothing else?
* When you talk about 'server', are you talking about the machine or Apache?
* If you are sure that it's Apache, then look in httpd.conf and try to calculate the memory usage, for example -
>> how many child processes are being spawned?
>> how many concurrent connections are being accepted?
and stuff like that.

Hope that one of the elder's respond with a more informative answer to your problem.

hth
theN

ps: I've heard that - Linux seems to use all the RAM, but frees it as and when requested by an application.

[dperez]

Well, I tried stopping apache, and that freed up some memory, but was still up to 90% of utilization. These are the process running on the box...

PID TTY STAT TIME COMMAND
1 ? S 0:05 init
2 ? SW 0:00 [keventd]
3 ? SW 0:00 [kapmd]
4 ? SWN 0:00 [ksoftirqd_CPU0]
5 ? SW 0:00 [kswapd]
6 ? SW 0:00 [bdflush]
7 ? SW 0:00 [kupdated]
8 ? SW 0:00 [mdrecoveryd]
12 ? SW 0:00 [kjournald]
108 ? SW 0:00 [kjournald]
330 ? SW 0:00 [eth0]
379 ? S 0:00 syslogd -m 0
383 ? S 0:00 klogd -x
436 ? S 0:00 /usr/sbin/sshd
450 ? S 0:00 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
462 ? SL 0:00 [ntpd]
488 ? S 0:00 crond
506 ? S 0:00 [atd]
526 ? S 0:00 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/u
541 ? S 0:03 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/we
545 tty1 S 0:00 /sbin/mingetty tty1
546 tty2 S 0:00 /sbin/mingetty tty2
547 tty3 S 0:00 /sbin/mingetty tty3
548 tty4 S 0:00 /sbin/mingetty tty4
549 tty5 S 0:00 /sbin/mingetty tty5
550 tty6 S 0:00 /sbin/mingetty tty6
551 ? S 0:00 /bin/sh /command/svscanboot
555 ? S 0:00 svscan /service
556 ? S 0:00 readproctitle service errors: .......................
557 ? S 0:00 supervise qmail-send
558 ? S 0:00 supervise log
559 ? S 0:00 supervise qmail-smtpd
560 ? S 0:00 supervise log
561 ? S 0:00 supervise qmail-pop3d
562 ? S 0:00 supervise log
563 ? S 0:00 /usr/local/bin/tcpserver -v -R -H -l 0 0 110 /var/qma
564 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail
565 ? S 0:00 [tcpserver]
567 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail/smtpd
570 ? S 0:00 [qmail-send]
571 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail/pop3d
574 ? S 0:00 qmail-lspawn ./Maildir/
575 ? S 0:00 [qmail-rspawn]
576 ? S 0:00 [qmail-clean]
656 ? S 0:00 [proftpd]
854 ? S 0:00 bash
907 ? S 0:01 ./sc_serv sc_serv.conf2
950 ? S 0:00 bash
977 ? S 186:13 ./sc_trans
3311 ? S 0:00 /usr/sbin/sshd
3313 ? S 0:00 [sshd]
3314 pts/2 S 0:00 -bash
3348 pts/2 S 0:00 [su]
3349 pts/2 S 0:00 bash
3774 ? S 0:00 /usr/sbin/httpd
3777 ? S 0:00 [httpd]
3778 ? S 0:00 [httpd]
3779 ? S 0:00 [httpd]
3780 ? S 0:00 [httpd]
3781 ? S 0:00 [httpd]
3788 ? S 0:00 [httpd]

Is there something I don't need running here?

Thanks
Dennys

[chrism01]

Are you sure you have only 1 webserver running? Are you sure you shut it down properly?
Because, you have multiple httpd processes running: httpd=webserver (normally apache on linux).
Did you just try to 'kill' the process or use eg
service httpd stop
?
HTH

[theN]

Originally posted by chrism01
Are you sure you have only 1 webserver running? Are you sure you shut it down properly?
Because, you have multiple httpd processes running: httpd=webserver (normally apache on linux).
Did you just try to 'kill' the process or use eg
service httpd stop
?
HTH Apache2 starts more than 1 process and each process in turn spawns threads which handle requests. These can be controlled from httpd.conf.

AFAIK there's nothing wrong with dperez's stats. They don't indicate more than 1 Apache running. Please correct me if I'm mistaken.

dperez there *isn't* anything out-of-sorts in the services you are running (typical of a web+mail server). Maybe you could do a ps and find out which process is hogging the RAM.

I am no Tux expert, hopefully one of the elder's will come up with a more conclusive and helpful advise.

regards
theN

[chrism01]

The reason i made the piont was he said he'd "tried stopping apache", so i assumed he'd run top afterwards.... If he ran top first, then his stats are indeed normal. I said multiple 'processes' because they are; I didn't say multiple copies of apache.
As you say, there's one master Apache process and typically 5 or 6 child processes that do all the work, but they all have their own pid.

[dperez]

I did shut it down properly (i think). I used #service httpd stop. After that, I did another ps ax and there was no httpd services running. Still the memory doesn't get flushed, only if I reboot the server it will go down again. How do I check which process is hogging the memory?

Thanks
Dennys

EDIT: Is really weird, because after I reboot, I have the exact same process running, yet memory utilization goes down to 14% :confused:

[chrism01]

well, it is true that Linux will try to use all avail mem and then swap as needed. This is perfectly normal, so that as much stuff is in mem as possible.
Unless you've got performance probs, i wouldn't worry too much.
The normal top cmd should show the mem usage of each process as well.

[stiles]

Originally posted by dperez
I did shut it down properly (i think). I used #service httpd stop. After that, I did another ps ax and there was no httpd services running. Still the memory doesn't get flushed, only if I reboot the server it will go down again. How do I check which process is hogging the memory?

EDIT: Is really weird, because after I reboot, I have the exact same process running, yet memory utilization goes down to 14% :confused:

It's normal, it's the filesystem cache. IIRC the default was to use 80% of system memory to cache the filesystem, but that was back in the day when the 2.4 VM was documented (somewhere around 2.4.9 the RM VM got riped out and the VM documentation no longer applies. I saw that the 2.4 AA tree had some VM tuning patches so that may be in the main tree by now but I haven't looked latelly.

[stiles]

Originally posted by dperez
How do I check which process is hogging the memory?


the u flag to ps will show memory usage in the output of ps, typically you will use something like "ps aux" to see all process in this format.

But in your case it's not a process that's hoging the memory it's the VM, but there are a few commands that can gleam what's going on with the VM.

free

vmstat

the cache line in both of these programs is the fs cache.

I looked at an idle box and it looks like the default is ~ 70% but my server has ~35% of memory cached which indicates to me that the VM is pretty good at aging the cache when there are memory pressures. That being said it would be very nice to beable to tune the VM cache, there is no reason to have a server tuned like a desktop no doubt.

[unixtool]

I have only worked with older versions of apache i.e. 1.36. But I am curious to see an output from vmstat and top.

I think you may be chewing up all your swap space thus leading to your resources being high all the time. If you do not have an acceptable amount of memory, and this includes swap, your flush subsystems are running all the time and eating itself.

Give us the output from vmstat;

#vmstat -2 2

and the output from top

#top

This will give us a better idea of what is going on.


HTH

[dperez]

This is the output from vmstat:

[root@server1 admin]# vmstat -n 2 2
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 7040 6540 14052 399500 0 0 119 141 741 144 6 2 92
0 0 0 7040 6540 14052 399828 0 0 162 0 590 55 0 0 100
[root@server1 admin]#

from #top:


count is the number of updates.
12:03am up 14:49, 1 user, load average: 0.01, 0.02, 0.00
72 processes: 68 sleeping, 4 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 1.1% system, 0.0% nice, 98.6% idle
Mem: 505940K av, 500508K used, 5432K free, 0K shrd, 14204K buff
Swap: 1052248K av, 7040K used, 1045208K free 400064K cached

and from #free:

[root@server1 admin]# free
total used free shared buffers cached
Mem: 505940 500500 5440 0 14288 400068
-/+ buffers/cache: 86144 419796
Swap: 1052248 7040 1045208
[root@server1 admin]#

Looks like almost all the memory is being chached. Is there a way to safely clear it?

Thanks much,
Dennys

[stiles]

Originally posted by dperez
Looks like almost all the memory is being chached. Is there a way to safely clear it?


Not that I know of, with the exception of rebooting, which isn't an option. It doesn't matter anyways if you could clear the page cache it would just grow back to it's current size without the ability to tune the VM. Ever sence the RMAP VM was ripped out of 2.4 (somewhere around 2.4.9 iirc) the /proc/sys/vm/ tunables have changed (somewhat) and the new entries are undocumented. Both /proc/sys/vm/pagecache and /proc/sys/vm/buffermem are gone, but if you take the time to read Documentation/sysctl/vm/vm.txt it says it's still there. OK I'm pissed, it's been over a damn year and there is still no updated documentation on the VM tunables, nor has the AA VM tunable patches been merged into the main kernel tree as of 2.4.20 (and it's all your fault for making me look, I've resisted sence 2.4.16 or so, I'm just kidding about it being your fault :P ). For the most part it's not a big deal on a webserver from what I know (who knows a real apache expert may drop some knowlage on apache's own caches and how they work, and then it may be a big deal), but on an oracle server it is so redundant it's sick.

[Magueta]

A couple things. The first is whether you're having any serious performance issues, if not then I would just let Linux handle the memory, simply because memory management is one of Linux's strengths. Open source software is not prone to memory leaks because the programming is so tight. For example, Apache has a configuration option that actually tells a child process to kill itself after a user-specified number connections to avoid memory leak issues (which as I've said are very rare in Linux and open source software). Which brings me to the second point, Apache 1.3 is a multiprocess server so when you do a ps -ax you'll get (lets say) 5 httpd processes running. This is not the case with Apache 2. Apache 2 is a multithreaded server so what you get is a single process handling all the requests and connections, so I would say based on your output from ps that you in fact have Apache 1.3 rather than Apache 2. Anyway, Apache 1.3 is the preferred server right now for a bunch of reasons so stick with it, that's what I'm running.

Joe

[stiles]

Originally posted by Magueta
Which brings me to the second point, Apache 1.3 is a multiprocess server so when you do a ps -ax you'll get (lets say) 5 httpd processes running. This is not the case with Apache 2. Apache 2 is a multithreaded server so what you get is a single process handling all the requests and connections, so I would say based on your output from ps that you in fact have Apache 1.3 rather than Apache 2. Anyway, Apache 1.3 is the preferred server right now for a bunch of reasons so stick with it, that's what I'm running.


The default MPM in apache2 on a UNIX platform is prefork, which is the same behavior as apache1.3.x Second, every version of ps I have seen on LINUX is broken when it comes to displaying threads, ever seen a java program run and your process table jumps up by like 30 processes, well thoes are threads even though ps will tell you they are processes. Actually I think there is a new version of ps that doesn't do this.

[Magueta]

I must say that I've never seen that behaviour that you're talking about with ps but, I'll take your word for it. As far as the default behaviour for Apache, I'm not sure what it is for Apache 2.x, but the reason that Apache 2.x is version 2.x and not 1.x is that it is supposed to be a multithreaded server rather than the prefork model. I'm not sure why the default behaviour for Apache 2 is multiprocess for Unix systems, but I would guess that the default for Windows is multithreaded. Apparently, there's a huge difference in speed between the two versions on Windows.
I'll keep my eye out for that funny ps behaviour, and thanks for letting us know about the default behaviour for Apache, I had no idea.

[dperez]

Not sure why think I'm using Apace 1.3x.. I have redhat 8 with Apache 2.0.40. :confused:

Dennys

[Magueta]

Originally posted by dperez
Not sure why think I'm using Apace 1.3x.. I have redhat 8 with Apache 2.0.40. :confused:

Dennys

What Stiles is saying is that you're probably using Apache 2.0 but it's acting like 1.3 because that's the default behaviour for Apache 2 on unix systems.

Joe

[stiles]

The apache 2 MPM docs (http://httpd.apache.org/docs-2.0/mpm.html)

here is a few ps things.

try "ps m"

ps: error: Thread display not implemented.
usage: ps -[Unix98 options]
ps [BSD-style options]
ps --[GNU-style long options]
ps --help for a command summary


here is the output of ps xf with a java applet running inside galeon (btw the f flag is very kewl, try it). Also notice how many galeon "processes" are running, not all of them are processes ;)

PID TTY STAT TIME COMMAND
31683 vc/2 S 0:00 -su
31701 vc/2 S 0:00 /bin/sh /usr/X11R6/bin/startx
31708 vc/2 S 0:00 \_ xinit /home/stiles/.xinitrc --
31837 vc/2 S 0:00 \_ fluxbox
31839 vc/2 S 13:10 \_ gkrellm -w
27006 pts/1 S 0:00 bash
1023 pts/1 R 0:00 ps xf
26991 ? S 0:00 /usr/bin/gconfd-1 6
26988 ? S 1:25 /usr/bin/galeon-bin
26994 ? S 0:00 /usr/bin/galeon-bin
26995 ? S 0:00 \_ /usr/bin/galeon-bin
26996 ? S 0:00 \_ /usr/bin/galeon-bin
26997 ? S 0:01 \_ /usr/bin/galeon-bin
981 ? S 0:01 java_vm
982 ? S 0:00 \_ java_vm
983 ? S 0:00 \_ java_vm
984 ? S 0:00 \_ java_vm
985 ? S 0:00 \_ java_vm
986 ? S 0:00 \_ java_vm
987 ? S 0:00 \_ java_vm
988 ? S 0:00 \_ java_vm
989 ? S 0:00 \_ java_vm
990 ? S 0:00 \_ java_vm
991 ? S 0:00 \_ java_vm
992 ? S 0:00 \_ java_vm
993 ? S 0:00 \_ java_vm
994 ? S 0:00 \_ java_vm
995 ? S 0:00 \_ java_vm
996 ? S 0:00 \_ java_vm
997 ? S 0:00 \_ java_vm
1001 ? S 0:00 \_ java_vm
1002 ? S 0:00 \_ java_vm
1003 ? S 0:01 \_ java_vm
1006 ? S 0:00 \_ java_vm
1009 ? S 0:00 \_ java_vm
1014 ? S 0:00 \_ java_vm
1015 ? S 0:00 \_ java_vm
1016 ? S 0:00 \_ java_vm
1017 ? S 0:00 \_ java_vm
1018 ? S 0:00 \_ java_vm
2325 ? S 0:00 oafd --ac-activate --ior-output-fd=10

[Magueta]

How can a thread have a pid?

Joe

[bastard23]

Joe,
It depends on the type of threading you are doing. Linux (man 2 clone) supports creation of "light-weight" processes (aka threads). It assigns a new PID to each thread. This is called 1:1 threading (the kernel schedules each thread).

If threading were in userspace (the userspace process schedules the threads) then there would only be one PID. There are libraries available that do this on Linux.

Here is a decent overview of where Linux is on threading: http://www.onlamp.com/pub/a/onlamp/2002/11/07/linux_threads.html

Have fun,
chris

[Magueta]

Thanks bastard, the article cleared up a great deal. Much of what was going on was over my head but I got enough to understand what was going on. What cleared up a great deal was the following little tidbit on the first page about the problems associated with Linux's "lightweight threading" and "clone()" problems.


Signal handling is incorrect.
An extra management thread is created by the pthreads library.
ps shows all threads in a process.
Core dumps don't contain the stack and machine register information for all threads.
getpid() returns a different result for each thread.
A thread cannot wait for a thread created by another thread.
Threads have parent-child, not peer, relationships.
Threads don't share user and group IDs.


Joe