[clue-tech] Swapped memory with 1GB of RAM?
Alex Young
alexander.young at gmail.com
Wed Feb 9 15:41:16 MST 2005
On Wed, 09 Feb 2005 12:58:39 -0900, Adam bultman <adamb at glaven.org> wrote:
> Alex Young wrote:
>
> >I've got a memory problem, and I'm stumped.
> >
> >I have a webserver (RH 9.0) running 2.4.20-19.9smp, and after a few
> >days of uptime it gets ungodly slow at serving pages due to swapping.
> > The machine has a gig of ram and is not running x or anything like
> >that.
> >
> >The system in question is serving a bunch of low-volume non-profit
> >websites, so rebooting is an issue, but can be done late at night.
> >
> >I've looked at migrating to debian, but I'd like to try to get this
> >machine working as the swap is potentially going to cause a lot of
> >issues.
> >
> >Here is some output from vmstat:
> >
> > procs memory swap io system cpu
> > r b w swpd free buff cache si so bi bo in cs us sy id
> > 0 0 1 139464 16224 1580 35096 2 3 1 3 3 1 1 1 4
> > 0 3 4 139464 16304 1600 34904 0 0 234 168 199 392 1 2 97
> > 0 0 1 139464 16308 1536 34996 0 0 61 43 135 244 0 1 99
> > 0 0 1 139464 16120 1684 35144 0 0 575 198 279 660 2 3 95
> >
> >Now while a page is loading:
> >
> > procs memory swap io system cpu
> > r b w swpd free buff cache si so bi bo in cs us sy id
> > 0 1 1 139464 13108 1632 34228 2 3 1 3 3 1 1 1 4
> > 0 1 2 139464 10116 1740 35064 886 0 1494 243 290 525 13 3 84
> > 0 3 1 139464 9040 1632 35436 430 0 874 174 280 562 7 4 89
> > 1 0 1 139464 8604 1660 35036 1306 92 1353 217 265 409 13 3 84
> > 0 0 1 139388 8728 1724 35224 338 180 496 244 239 497 15 1 84
> >
> >So, it's swapping hard. So I look at free:
> >
> > total used free shared buffers cached
> >Mem: 1030304 1020468 9836 0 1444 35336
> >-/+ buffers/cache: 983688 46616
> >Swap: 2072376 139388 1932988
> >
> >That "-/+ buffers/cache:" line looks bad to me, but I'm not sure. I
> >think the buffer line is high for this setup, it's filling most of the
> >RAM. Also, there is a lot of swapped data.
> >
> >So I looked at ps aux:
> >
> >USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> >root 1 0.0 0.0 1384 440 ? S 2004 1:22 init
> >root 2 0.0 0.0 0 0 ? SW 2004 0:00 [migration/0]
> >root 3 0.0 0.0 0 0 ? SW 2004 0:00 [migration/1]
> >root 4 0.0 0.0 0 0 ? SW 2004 13:21 [keventd]
> >root 5 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU0]
> >root 6 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU1]
> >root 11 0.0 0.0 0 0 ? SW 2004 0:25 [bdflush]
> >root 7 0.5 0.0 0 0 ? DW 2004 411:25 [kswapd]
> >root 8 0.0 0.0 0 0 ? SW 2004 0:16 [kscand/DMA]
> >root 9 0.0 0.0 0 0 ? SW 2004 57:26 [kscand/Normal]
> >root 10 0.1 0.0 0 0 ? SW 2004 73:05 [kscand/HighMem]
> >root 12 0.0 0.0 0 0 ? SW 2004 2:02 [kupdated]
> >root 13 0.0 0.0 0 0 ? SW 2004 0:00 [mdrecoveryd]
> >root 19 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_0]
> >root 20 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_1]
> >root 24 0.0 0.0 0 0 ? SW 2004 6:49 [kjournald]
> >root 82 0.0 0.0 0 0 ? SW 2004 0:00 [khubd]
> >root 188 0.0 0.0 0 0 ? SW 2004 0:40 [kjournald]
> >root 189 0.0 0.0 0 0 ? SW 2004 0:00 [kjournald]
> >root 190 0.0 0.0 0 0 ? SW 2004 0:29 [kjournald]
> >root 191 0.0 0.0 0 0 ? SW 2004 12:37 [kjournald]
> >root 192 0.0 0.0 0 0 ? SW 2004 7:16 [kjournald]
> >root 692 0.0 0.0 1456 512 ? S 2004 1:01 syslogd -m 0
> >root 696 0.0 0.0 1376 388 ? S 2004 0:00 klogd -x
> >rpc 714 0.0 0.0 1560 440 ? S 2004 0:00 portmap
> >rpcuser 733 0.0 0.0 1580 568 ? S 2004 0:00 rpc.statd -p 4000
> >root 844 0.0 0.0 3720 484 ? S 2004 1:25 /usr/sbin/sshd
> >root 860 0.0 0.0 2072 540 ? S 2004 0:00 xinetd -stayalive
> >ntp 879 0.0 0.2 2412 2408 ? SL 2004 4:27 ntpd -U ntp
> >lp 895 0.0 0.0 9136 160 ? S 2004 0:19 lpd Waiting
> >root 906 0.0 0.0 1424 4 ? S 2004 0:00 gpm -t imps2 -m /
> >bin 927 0.0 0.0 2064 40 ? S 2004 0:01 /usr/sbin/cannase
> >root 938 0.0 0.0 1432 136 ? S 2004 0:38 crond
> >xfs 971 0.0 0.0 5664 28 ? S 2004 0:00 xfs -droppriv -da
> >daemon 1010 0.0 0.0 1420 112 ? S 2004 0:04 /usr/sbin/atd
> >root 1020 0.0 0.0 3416 388 ? S 2004 0:00 rhnsd --interval
> >root 1027 0.0 0.0 1360 0 tty2 SW 2004 0:00 /sbin/mingetty tt
> >root 1028 0.0 0.0 1360 0 tty3 SW 2004 0:00 /sbin/mingetty tt
> >root 1029 0.0 0.0 1360 0 tty4 SW 2004 0:00 /sbin/mingetty tt
> >root 1030 0.0 0.0 1360 0 tty5 SW 2004 0:00 /sbin/mingetty tt
> >root 1031 0.0 0.0 1360 0 tty6 SW 2004 0:00 /sbin/mingetty tt
> >root 1032 0.0 0.0 2120 0 ? SW 2004 0:00 /bin/sh /command/
> >root 1034 0.0 0.0 1400 88 ? S 2004 3:48 svscan /service
> >root 1035 0.0 0.0 1352 0 ? SW 2004 0:00 readproctitle ser
> >root 1036 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
> >root 1037 0.0 0.0 1352 24 ? S 2004 0:00 supervise log
> >root 1038 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
> >root 1039 0.0 0.0 1352 12 ? S 2004 0:00 supervise log
> >qmaild 1040 0.0 0.0 1428 4 ? S 2004 0:05 /usr/local/bin/tc
> >qmaill 1041 0.0 0.0 1376 0 ? SW 2004 0:04 /usr/local/bin/mu
> >qmails 1042 0.0 0.0 1580 216 ? S 2004 63:05 qmail-send
> >qmaill 1043 0.0 0.0 1376 64 ? S 2004 26:29 /usr/local/bin/mu
> >root 1046 0.0 0.0 1372 104 ? S 2004 55:27 qmail-lspawn |dot
> >qmailr 1047 0.0 0.0 1368 0 ? SW 2004 0:05 qmail-rspawn
> >qmailq 1048 0.0 0.0 1364 52 ? S 2004 1:39 qmail-clean
> >root 6398 0.0 0.0 1364 4 tty1 S 2004 0:00 /sbin/mingetty tt
> >root 13626 0.0 0.0 0 0 ? SW 2004 0:17 [kjournald]
> >root 22547 0.0 0.0 0 0 ? SW Jan02 0:00 [nfsd]
> >root 22549 0.0 0.0 0 0 ? SW Jan02 0:00 [lockd]
> >root 22550 0.0 0.0 0 0 ? SW Jan02 0:18 [rpciod]
> >rpcuser 22552 0.0 0.0 1580 4 ? S Jan02 0:00 /sbin/rpc.statd
> >root 22554 0.0 0.0 1508 8 ? S Jan02 0:00 /usr/sbin/rpc.mou
> >root 22556 0.0 0.0 3628 4 ? S Jan02 0:00 /usr/sbin/rpc.rqu
> >rpcuser 30664 0.0 0.0 1580 28 ? S Jan02 0:00 /sbin/rpc.statd
> >root 30666 0.0 0.0 1508 0 ? SW Jan02 0:00 /usr/sbin/rpc.mou
> >root 30668 0.0 0.0 3636 0 ? SW Jan02 0:00 /usr/sbin/rpc.rqu
> >rpcuser 15749 0.0 0.0 1576 0 ? SW Jan05 0:00 /sbin/rpc.statd
> >root 15751 0.0 0.0 1508 0 ? SW Jan05 0:00 /usr/sbin/rpc.mou
> >root 15753 0.0 0.0 3636 0 ? SW Jan05 0:00 /usr/sbin/rpc.rqu
> >root 19011 0.0 0.0 18652 84 ? S Feb04 0:17 /usr/sbin/httpd -
> >www-data 27260 0.0 0.4 26720 4760 ? S Feb06 2:21 /usr/sbin/httpd -
> >www-data 27261 0.0 0.8 26472 8316 ? S Feb06 2:13 /usr/sbin/httpd -
> >www-data 27267 0.0 0.5 26640 5168 ? S Feb06 2:24 /usr/sbin/httpd -
> >www-data 27271 0.0 0.5 26724 5640 ? S Feb06 2:23 /usr/sbin/httpd -
> >www-data 27272 0.0 0.1 27032 1344 ? S Feb06 2:26 /usr/sbin/httpd -
> >www-data 29164 0.0 0.3 26296 3332 ? S Feb06 2:39 /usr/sbin/httpd -
> >www-data 19753 0.0 0.0 26300 952 ? S Feb06 2:14 /usr/sbin/httpd -
> >www-data 30495 0.0 0.2 26500 2504 ? S Feb06 2:21 /usr/sbin/httpd -
> >www-data 30496 0.0 0.9 26544 9468 ? S Feb06 2:19 /usr/sbin/httpd -
> >www-data 30497 0.0 0.5 26796 5504 ? S Feb06 2:22 /usr/sbin/httpd -
> >www-data 30526 0.0 0.0 26904 612 ? S Feb06 2:20 /usr/sbin/httpd -
> >www-data 8409 0.0 0.2 26520 2540 ? S Feb06 2:16 /usr/sbin/httpd -
> >www-data 8555 0.0 0.6 26744 7204 ? S Feb06 2:26 /usr/sbin/httpd -
> >www-data 26335 0.0 0.8 26544 8624 ? S Feb06 2:21 /usr/sbin/httpd -
> >www-data 26256 0.0 0.7 26564 7716 ? S Feb07 1:37 /usr/sbin/httpd -
> >root 18493 0.0 0.0 6388 0 ? SW 08:24 0:00 sshd: ayoung [pri
> >ayoung 18510 0.0 0.0 6432 304 ? S 08:24 0:02 sshd: ayoung at pts/
> >ayoung 18511 0.0 0.0 4604 0 pts/0 SW 08:24 0:00 -bash
> >root 18965 0.0 0.0 4312 0 pts/0 SW 08:26 0:00 su
> >root 18966 0.0 0.0 4624 672 pts/0 S 08:26 0:01 bash
> >named 25776 0.0 0.1 39060 1544 ? S 08:59 0:00 /usr/sbin/named -
> >root 13602 0.0 0.0 4200 108 pts/0 S 10:30 0:00 /bin/sh /usr/bin/
> >mysql 13657 0.4 0.2 15544 2144 pts/0 S 10:30 0:08 /usr/libexec/mysq
> >ayoung 20722 0.0 0.2 4044 2168 ? S 11:05 0:00 /usr/libexec/gcon
> >root 20988 0.0 0.0 2700 760 pts/0 R 11:06 0:00 ps aux
> >
> >Nothing looks huge in the memory area. It hardly looks like the
> >memory is being touched at all (7.2%).
> >
> >I don't get it. Where did my RAM go? What else can I do to troubleshoot?
> >
> >Thanks in advance,
> >
> >-Alex
> >
> >
> My first suggestion: install the sar utility and have sar get info, as
> well as get process accounting going. Whenever I have problems of this
> sort, I use those two utilities.
I think I now have that working. Sar produces a lot of info, so
I'm going to have to look at it a bit to tell what I'm looking at.
Any pointers?
>
> Second: Run 'top', and sort by memory usage. There's a way to get it
> piped out to a file if you want, and you could run that throughout the
> night and capture what's gobbling up RAM.
>
I think this output will look a lot like "ps aux" won't it? When I
run top the numbers are the same as ps.
> Third: Check 'dmesg' for terminating processes as a result of exhausted RAM.
dmesg is clean since boot.
>
> That's what I'd do first, if I were you. If you are wanting to switch to
> Debian becuase you think that RAM problems are inherent to Red Hat,
> you're gonna find that the same stuff happens on Debian - there's
> something on the system that's eating up your RAM, and switching distros
> won't solve it.
I just run some other Debian boxes and I'm better versed with it.
I'm used to kernel hacking (not the best, I've just had good results)
in Debian, so that's why I will eventually switch. Not to ignore
the forced migration due to EOL for RH 9.0.
>
> Let us know what top says when it's slowing down - you should find
> what's eating your RAM (unless there's a process that's hidden and
> eating up your RAM - in which case, you've got bigger problems)
>
> Adam
Thanks for the advice!
More information about the clue-tech
mailing list