[clue-tech] Swapped memory with 1GB of RAM?
Adam bultman
adamb at glaven.org
Wed Feb 9 14:58:39 MST 2005
Alex Young wrote:
>I've got a memory problem, and I'm stumped.
>
>I have a webserver (RH 9.0) running 2.4.20-19.9smp, and after a few
>days of uptime it gets ungodly slow at serving pages due to swapping.
> The machine has a gig of ram and is not running x or anything like
>that.
>
>The system in question is serving a bunch of low-volume non-profit
>websites, so rebooting is an issue, but can be done late at night.
>
>I've looked at migrating to debian, but I'd like to try to get this
>machine working as the swap is potentially going to cause a lot of
>issues.
>
>Here is some output from vmstat:
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 0 0 1 139464 16224 1580 35096 2 3 1 3 3 1 1 1 4
> 0 3 4 139464 16304 1600 34904 0 0 234 168 199 392 1 2 97
> 0 0 1 139464 16308 1536 34996 0 0 61 43 135 244 0 1 99
> 0 0 1 139464 16120 1684 35144 0 0 575 198 279 660 2 3 95
>
>Now while a page is loading:
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 0 1 1 139464 13108 1632 34228 2 3 1 3 3 1 1 1 4
> 0 1 2 139464 10116 1740 35064 886 0 1494 243 290 525 13 3 84
> 0 3 1 139464 9040 1632 35436 430 0 874 174 280 562 7 4 89
> 1 0 1 139464 8604 1660 35036 1306 92 1353 217 265 409 13 3 84
> 0 0 1 139388 8728 1724 35224 338 180 496 244 239 497 15 1 84
>
>So, it's swapping hard. So I look at free:
>
> total used free shared buffers cached
>Mem: 1030304 1020468 9836 0 1444 35336
>-/+ buffers/cache: 983688 46616
>Swap: 2072376 139388 1932988
>
>That "-/+ buffers/cache:" line looks bad to me, but I'm not sure. I
>think the buffer line is high for this setup, it's filling most of the
>RAM. Also, there is a lot of swapped data.
>
>So I looked at ps aux:
>
>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>root 1 0.0 0.0 1384 440 ? S 2004 1:22 init
>root 2 0.0 0.0 0 0 ? SW 2004 0:00 [migration/0]
>root 3 0.0 0.0 0 0 ? SW 2004 0:00 [migration/1]
>root 4 0.0 0.0 0 0 ? SW 2004 13:21 [keventd]
>root 5 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU0]
>root 6 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU1]
>root 11 0.0 0.0 0 0 ? SW 2004 0:25 [bdflush]
>root 7 0.5 0.0 0 0 ? DW 2004 411:25 [kswapd]
>root 8 0.0 0.0 0 0 ? SW 2004 0:16 [kscand/DMA]
>root 9 0.0 0.0 0 0 ? SW 2004 57:26 [kscand/Normal]
>root 10 0.1 0.0 0 0 ? SW 2004 73:05 [kscand/HighMem]
>root 12 0.0 0.0 0 0 ? SW 2004 2:02 [kupdated]
>root 13 0.0 0.0 0 0 ? SW 2004 0:00 [mdrecoveryd]
>root 19 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_0]
>root 20 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_1]
>root 24 0.0 0.0 0 0 ? SW 2004 6:49 [kjournald]
>root 82 0.0 0.0 0 0 ? SW 2004 0:00 [khubd]
>root 188 0.0 0.0 0 0 ? SW 2004 0:40 [kjournald]
>root 189 0.0 0.0 0 0 ? SW 2004 0:00 [kjournald]
>root 190 0.0 0.0 0 0 ? SW 2004 0:29 [kjournald]
>root 191 0.0 0.0 0 0 ? SW 2004 12:37 [kjournald]
>root 192 0.0 0.0 0 0 ? SW 2004 7:16 [kjournald]
>root 692 0.0 0.0 1456 512 ? S 2004 1:01 syslogd -m 0
>root 696 0.0 0.0 1376 388 ? S 2004 0:00 klogd -x
>rpc 714 0.0 0.0 1560 440 ? S 2004 0:00 portmap
>rpcuser 733 0.0 0.0 1580 568 ? S 2004 0:00 rpc.statd -p 4000
>root 844 0.0 0.0 3720 484 ? S 2004 1:25 /usr/sbin/sshd
>root 860 0.0 0.0 2072 540 ? S 2004 0:00 xinetd -stayalive
>ntp 879 0.0 0.2 2412 2408 ? SL 2004 4:27 ntpd -U ntp
>lp 895 0.0 0.0 9136 160 ? S 2004 0:19 lpd Waiting
>root 906 0.0 0.0 1424 4 ? S 2004 0:00 gpm -t imps2 -m /
>bin 927 0.0 0.0 2064 40 ? S 2004 0:01 /usr/sbin/cannase
>root 938 0.0 0.0 1432 136 ? S 2004 0:38 crond
>xfs 971 0.0 0.0 5664 28 ? S 2004 0:00 xfs -droppriv -da
>daemon 1010 0.0 0.0 1420 112 ? S 2004 0:04 /usr/sbin/atd
>root 1020 0.0 0.0 3416 388 ? S 2004 0:00 rhnsd --interval
>root 1027 0.0 0.0 1360 0 tty2 SW 2004 0:00 /sbin/mingetty tt
>root 1028 0.0 0.0 1360 0 tty3 SW 2004 0:00 /sbin/mingetty tt
>root 1029 0.0 0.0 1360 0 tty4 SW 2004 0:00 /sbin/mingetty tt
>root 1030 0.0 0.0 1360 0 tty5 SW 2004 0:00 /sbin/mingetty tt
>root 1031 0.0 0.0 1360 0 tty6 SW 2004 0:00 /sbin/mingetty tt
>root 1032 0.0 0.0 2120 0 ? SW 2004 0:00 /bin/sh /command/
>root 1034 0.0 0.0 1400 88 ? S 2004 3:48 svscan /service
>root 1035 0.0 0.0 1352 0 ? SW 2004 0:00 readproctitle ser
>root 1036 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
>root 1037 0.0 0.0 1352 24 ? S 2004 0:00 supervise log
>root 1038 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
>root 1039 0.0 0.0 1352 12 ? S 2004 0:00 supervise log
>qmaild 1040 0.0 0.0 1428 4 ? S 2004 0:05 /usr/local/bin/tc
>qmaill 1041 0.0 0.0 1376 0 ? SW 2004 0:04 /usr/local/bin/mu
>qmails 1042 0.0 0.0 1580 216 ? S 2004 63:05 qmail-send
>qmaill 1043 0.0 0.0 1376 64 ? S 2004 26:29 /usr/local/bin/mu
>root 1046 0.0 0.0 1372 104 ? S 2004 55:27 qmail-lspawn |dot
>qmailr 1047 0.0 0.0 1368 0 ? SW 2004 0:05 qmail-rspawn
>qmailq 1048 0.0 0.0 1364 52 ? S 2004 1:39 qmail-clean
>root 6398 0.0 0.0 1364 4 tty1 S 2004 0:00 /sbin/mingetty tt
>root 13626 0.0 0.0 0 0 ? SW 2004 0:17 [kjournald]
>root 22547 0.0 0.0 0 0 ? SW Jan02 0:00 [nfsd]
>root 22549 0.0 0.0 0 0 ? SW Jan02 0:00 [lockd]
>root 22550 0.0 0.0 0 0 ? SW Jan02 0:18 [rpciod]
>rpcuser 22552 0.0 0.0 1580 4 ? S Jan02 0:00 /sbin/rpc.statd
>root 22554 0.0 0.0 1508 8 ? S Jan02 0:00 /usr/sbin/rpc.mou
>root 22556 0.0 0.0 3628 4 ? S Jan02 0:00 /usr/sbin/rpc.rqu
>rpcuser 30664 0.0 0.0 1580 28 ? S Jan02 0:00 /sbin/rpc.statd
>root 30666 0.0 0.0 1508 0 ? SW Jan02 0:00 /usr/sbin/rpc.mou
>root 30668 0.0 0.0 3636 0 ? SW Jan02 0:00 /usr/sbin/rpc.rqu
>rpcuser 15749 0.0 0.0 1576 0 ? SW Jan05 0:00 /sbin/rpc.statd
>root 15751 0.0 0.0 1508 0 ? SW Jan05 0:00 /usr/sbin/rpc.mou
>root 15753 0.0 0.0 3636 0 ? SW Jan05 0:00 /usr/sbin/rpc.rqu
>root 19011 0.0 0.0 18652 84 ? S Feb04 0:17 /usr/sbin/httpd -
>www-data 27260 0.0 0.4 26720 4760 ? S Feb06 2:21 /usr/sbin/httpd -
>www-data 27261 0.0 0.8 26472 8316 ? S Feb06 2:13 /usr/sbin/httpd -
>www-data 27267 0.0 0.5 26640 5168 ? S Feb06 2:24 /usr/sbin/httpd -
>www-data 27271 0.0 0.5 26724 5640 ? S Feb06 2:23 /usr/sbin/httpd -
>www-data 27272 0.0 0.1 27032 1344 ? S Feb06 2:26 /usr/sbin/httpd -
>www-data 29164 0.0 0.3 26296 3332 ? S Feb06 2:39 /usr/sbin/httpd -
>www-data 19753 0.0 0.0 26300 952 ? S Feb06 2:14 /usr/sbin/httpd -
>www-data 30495 0.0 0.2 26500 2504 ? S Feb06 2:21 /usr/sbin/httpd -
>www-data 30496 0.0 0.9 26544 9468 ? S Feb06 2:19 /usr/sbin/httpd -
>www-data 30497 0.0 0.5 26796 5504 ? S Feb06 2:22 /usr/sbin/httpd -
>www-data 30526 0.0 0.0 26904 612 ? S Feb06 2:20 /usr/sbin/httpd -
>www-data 8409 0.0 0.2 26520 2540 ? S Feb06 2:16 /usr/sbin/httpd -
>www-data 8555 0.0 0.6 26744 7204 ? S Feb06 2:26 /usr/sbin/httpd -
>www-data 26335 0.0 0.8 26544 8624 ? S Feb06 2:21 /usr/sbin/httpd -
>www-data 26256 0.0 0.7 26564 7716 ? S Feb07 1:37 /usr/sbin/httpd -
>root 18493 0.0 0.0 6388 0 ? SW 08:24 0:00 sshd: ayoung [pri
>ayoung 18510 0.0 0.0 6432 304 ? S 08:24 0:02 sshd: ayoung at pts/
>ayoung 18511 0.0 0.0 4604 0 pts/0 SW 08:24 0:00 -bash
>root 18965 0.0 0.0 4312 0 pts/0 SW 08:26 0:00 su
>root 18966 0.0 0.0 4624 672 pts/0 S 08:26 0:01 bash
>named 25776 0.0 0.1 39060 1544 ? S 08:59 0:00 /usr/sbin/named -
>root 13602 0.0 0.0 4200 108 pts/0 S 10:30 0:00 /bin/sh /usr/bin/
>mysql 13657 0.4 0.2 15544 2144 pts/0 S 10:30 0:08 /usr/libexec/mysq
>ayoung 20722 0.0 0.2 4044 2168 ? S 11:05 0:00 /usr/libexec/gcon
>root 20988 0.0 0.0 2700 760 pts/0 R 11:06 0:00 ps aux
>
>Nothing looks huge in the memory area. It hardly looks like the
>memory is being touched at all (7.2%).
>
>I don't get it. Where did my RAM go? What else can I do to troubleshoot?
>
>Thanks in advance,
>
>-Alex
>
>
My first suggestion: install the sar utility and have sar get info, as
well as get process accounting going. Whenever I have problems of this
sort, I use those two utilities.
Second: Run 'top', and sort by memory usage. There's a way to get it
piped out to a file if you want, and you could run that throughout the
night and capture what's gobbling up RAM.
Third: Check 'dmesg' for terminating processes as a result of exhausted RAM.
That's what I'd do first, if I were you. If you are wanting to switch to
Debian becuase you think that RAM problems are inherent to Red Hat,
you're gonna find that the same stuff happens on Debian - there's
something on the system that's eating up your RAM, and switching distros
won't solve it.
Let us know what top says when it's slowing down - you should find
what's eating your RAM (unless there's a process that's hidden and
eating up your RAM - in which case, you've got bigger problems)
Adam
More information about the clue-tech
mailing list