[clue-tech] Swapped memory with 1GB of RAM?

Alex Young alexander.young at gmail.com
Wed Feb 9 15:41:16 MST 2005


On Wed, 09 Feb 2005 12:58:39 -0900, Adam bultman <adamb at glaven.org> wrote:
> Alex Young wrote:
> 
> >I've got a memory problem, and I'm stumped.
> >
> >I have a webserver (RH 9.0) running 2.4.20-19.9smp, and after a few
> >days of uptime it gets ungodly slow at serving pages due to swapping.
> >   The machine has a gig of ram and is not running x or anything like
> >that.
> >
> >The system in question is serving a bunch of low-volume non-profit
> >websites, so rebooting is an issue, but can be done late at night.
> >
> >I've looked at migrating to debian, but I'd like to try to get this
> >machine working as the swap is potentially going to cause a lot of
> >issues.
> >
> >Here is some output from vmstat:
> >
> >   procs                      memory      swap          io     system      cpu
> > r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
> > 0  0  1 139464  16224   1580  35096    2    3     1     3    3     1  1  1  4
> > 0  3  4 139464  16304   1600  34904    0    0   234   168  199   392  1  2 97
> > 0  0  1 139464  16308   1536  34996    0    0    61    43  135   244  0  1 99
> > 0  0  1 139464  16120   1684  35144    0    0   575   198  279   660  2  3 95
> >
> >Now while a page is loading:
> >
> >   procs                      memory      swap          io     system      cpu
> > r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
> > 0  1  1 139464  13108   1632  34228    2    3     1     3    3     1  1  1  4
> > 0  1  2 139464  10116   1740  35064  886    0  1494   243  290   525 13  3 84
> > 0  3  1 139464   9040   1632  35436  430    0   874   174  280   562  7  4 89
> > 1  0  1 139464   8604   1660  35036 1306   92  1353   217  265   409 13  3 84
> > 0  0  1 139388   8728   1724  35224  338  180   496   244  239   497 15  1 84
> >
> >So, it's swapping hard.    So I look at free:
> >
> >             total       used       free     shared    buffers     cached
> >Mem:       1030304    1020468       9836          0       1444      35336
> >-/+ buffers/cache:     983688      46616
> >Swap:      2072376     139388    1932988
> >
> >That "-/+ buffers/cache:" line looks bad to me, but I'm not sure.    I
> >think the buffer line is high for this setup, it's filling most of the
> >RAM.    Also, there is a lot of swapped data.
> >
> >So I looked at ps aux:
> >
> >USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> >root         1  0.0  0.0  1384  440 ?        S     2004   1:22 init
> >root         2  0.0  0.0     0    0 ?        SW    2004   0:00 [migration/0]
> >root         3  0.0  0.0     0    0 ?        SW    2004   0:00 [migration/1]
> >root         4  0.0  0.0     0    0 ?        SW    2004  13:21 [keventd]
> >root         5  0.0  0.0     0    0 ?        SWN   2004   0:00 [ksoftirqd_CPU0]
> >root         6  0.0  0.0     0    0 ?        SWN   2004   0:00 [ksoftirqd_CPU1]
> >root        11  0.0  0.0     0    0 ?        SW    2004   0:25 [bdflush]
> >root         7  0.5  0.0     0    0 ?        DW    2004 411:25 [kswapd]
> >root         8  0.0  0.0     0    0 ?        SW    2004   0:16 [kscand/DMA]
> >root         9  0.0  0.0     0    0 ?        SW    2004  57:26 [kscand/Normal]
> >root        10  0.1  0.0     0    0 ?        SW    2004  73:05 [kscand/HighMem]
> >root        12  0.0  0.0     0    0 ?        SW    2004   2:02 [kupdated]
> >root        13  0.0  0.0     0    0 ?        SW    2004   0:00 [mdrecoveryd]
> >root        19  0.0  0.0     0    0 ?        SW    2004   0:00 [scsi_eh_0]
> >root        20  0.0  0.0     0    0 ?        SW    2004   0:00 [scsi_eh_1]
> >root        24  0.0  0.0     0    0 ?        SW    2004   6:49 [kjournald]
> >root        82  0.0  0.0     0    0 ?        SW    2004   0:00 [khubd]
> >root       188  0.0  0.0     0    0 ?        SW    2004   0:40 [kjournald]
> >root       189  0.0  0.0     0    0 ?        SW    2004   0:00 [kjournald]
> >root       190  0.0  0.0     0    0 ?        SW    2004   0:29 [kjournald]
> >root       191  0.0  0.0     0    0 ?        SW    2004  12:37 [kjournald]
> >root       192  0.0  0.0     0    0 ?        SW    2004   7:16 [kjournald]
> >root       692  0.0  0.0  1456  512 ?        S     2004   1:01 syslogd -m 0
> >root       696  0.0  0.0  1376  388 ?        S     2004   0:00 klogd -x
> >rpc        714  0.0  0.0  1560  440 ?        S     2004   0:00 portmap
> >rpcuser    733  0.0  0.0  1580  568 ?        S     2004   0:00 rpc.statd -p 4000
> >root       844  0.0  0.0  3720  484 ?        S     2004   1:25 /usr/sbin/sshd
> >root       860  0.0  0.0  2072  540 ?        S     2004   0:00 xinetd -stayalive
> >ntp        879  0.0  0.2  2412 2408 ?        SL    2004   4:27 ntpd -U ntp
> >lp         895  0.0  0.0  9136  160 ?        S     2004   0:19 lpd Waiting
> >root       906  0.0  0.0  1424    4 ?        S     2004   0:00 gpm -t imps2 -m /
> >bin        927  0.0  0.0  2064   40 ?        S     2004   0:01 /usr/sbin/cannase
> >root       938  0.0  0.0  1432  136 ?        S     2004   0:38 crond
> >xfs        971  0.0  0.0  5664   28 ?        S     2004   0:00 xfs -droppriv -da
> >daemon    1010  0.0  0.0  1420  112 ?        S     2004   0:04 /usr/sbin/atd
> >root      1020  0.0  0.0  3416  388 ?        S     2004   0:00 rhnsd --interval
> >root      1027  0.0  0.0  1360    0 tty2     SW    2004   0:00 /sbin/mingetty tt
> >root      1028  0.0  0.0  1360    0 tty3     SW    2004   0:00 /sbin/mingetty tt
> >root      1029  0.0  0.0  1360    0 tty4     SW    2004   0:00 /sbin/mingetty tt
> >root      1030  0.0  0.0  1360    0 tty5     SW    2004   0:00 /sbin/mingetty tt
> >root      1031  0.0  0.0  1360    0 tty6     SW    2004   0:00 /sbin/mingetty tt
> >root      1032  0.0  0.0  2120    0 ?        SW    2004   0:00 /bin/sh /command/
> >root      1034  0.0  0.0  1400   88 ?        S     2004   3:48 svscan /service
> >root      1035  0.0  0.0  1352    0 ?        SW    2004   0:00 readproctitle ser
> >root      1036  0.0  0.0  1352   12 ?        S     2004   0:00 supervise qmail-s
> >root      1037  0.0  0.0  1352   24 ?        S     2004   0:00 supervise log
> >root      1038  0.0  0.0  1352   12 ?        S     2004   0:00 supervise qmail-s
> >root      1039  0.0  0.0  1352   12 ?        S     2004   0:00 supervise log
> >qmaild    1040  0.0  0.0  1428    4 ?        S     2004   0:05 /usr/local/bin/tc
> >qmaill    1041  0.0  0.0  1376    0 ?        SW    2004   0:04 /usr/local/bin/mu
> >qmails    1042  0.0  0.0  1580  216 ?        S     2004  63:05 qmail-send
> >qmaill    1043  0.0  0.0  1376   64 ?        S     2004  26:29 /usr/local/bin/mu
> >root      1046  0.0  0.0  1372  104 ?        S     2004  55:27 qmail-lspawn |dot
> >qmailr    1047  0.0  0.0  1368    0 ?        SW    2004   0:05 qmail-rspawn
> >qmailq    1048  0.0  0.0  1364   52 ?        S     2004   1:39 qmail-clean
> >root      6398  0.0  0.0  1364    4 tty1     S     2004   0:00 /sbin/mingetty tt
> >root     13626  0.0  0.0     0    0 ?        SW    2004   0:17 [kjournald]
> >root     22547  0.0  0.0     0    0 ?        SW   Jan02   0:00 [nfsd]
> >root     22549  0.0  0.0     0    0 ?        SW   Jan02   0:00 [lockd]
> >root     22550  0.0  0.0     0    0 ?        SW   Jan02   0:18 [rpciod]
> >rpcuser  22552  0.0  0.0  1580    4 ?        S    Jan02   0:00 /sbin/rpc.statd
> >root     22554  0.0  0.0  1508    8 ?        S    Jan02   0:00 /usr/sbin/rpc.mou
> >root     22556  0.0  0.0  3628    4 ?        S    Jan02   0:00 /usr/sbin/rpc.rqu
> >rpcuser  30664  0.0  0.0  1580   28 ?        S    Jan02   0:00 /sbin/rpc.statd
> >root     30666  0.0  0.0  1508    0 ?        SW   Jan02   0:00 /usr/sbin/rpc.mou
> >root     30668  0.0  0.0  3636    0 ?        SW   Jan02   0:00 /usr/sbin/rpc.rqu
> >rpcuser  15749  0.0  0.0  1576    0 ?        SW   Jan05   0:00 /sbin/rpc.statd
> >root     15751  0.0  0.0  1508    0 ?        SW   Jan05   0:00 /usr/sbin/rpc.mou
> >root     15753  0.0  0.0  3636    0 ?        SW   Jan05   0:00 /usr/sbin/rpc.rqu
> >root     19011  0.0  0.0 18652   84 ?        S    Feb04   0:17 /usr/sbin/httpd -
> >www-data 27260  0.0  0.4 26720 4760 ?        S    Feb06   2:21 /usr/sbin/httpd -
> >www-data 27261  0.0  0.8 26472 8316 ?        S    Feb06   2:13 /usr/sbin/httpd -
> >www-data 27267  0.0  0.5 26640 5168 ?        S    Feb06   2:24 /usr/sbin/httpd -
> >www-data 27271  0.0  0.5 26724 5640 ?        S    Feb06   2:23 /usr/sbin/httpd -
> >www-data 27272  0.0  0.1 27032 1344 ?        S    Feb06   2:26 /usr/sbin/httpd -
> >www-data 29164  0.0  0.3 26296 3332 ?        S    Feb06   2:39 /usr/sbin/httpd -
> >www-data 19753  0.0  0.0 26300  952 ?        S    Feb06   2:14 /usr/sbin/httpd -
> >www-data 30495  0.0  0.2 26500 2504 ?        S    Feb06   2:21 /usr/sbin/httpd -
> >www-data 30496  0.0  0.9 26544 9468 ?        S    Feb06   2:19 /usr/sbin/httpd -
> >www-data 30497  0.0  0.5 26796 5504 ?        S    Feb06   2:22 /usr/sbin/httpd -
> >www-data 30526  0.0  0.0 26904  612 ?        S    Feb06   2:20 /usr/sbin/httpd -
> >www-data  8409  0.0  0.2 26520 2540 ?        S    Feb06   2:16 /usr/sbin/httpd -
> >www-data  8555  0.0  0.6 26744 7204 ?        S    Feb06   2:26 /usr/sbin/httpd -
> >www-data 26335  0.0  0.8 26544 8624 ?        S    Feb06   2:21 /usr/sbin/httpd -
> >www-data 26256  0.0  0.7 26564 7716 ?        S    Feb07   1:37 /usr/sbin/httpd -
> >root     18493  0.0  0.0  6388    0 ?        SW   08:24   0:00 sshd: ayoung [pri
> >ayoung   18510  0.0  0.0  6432  304 ?        S    08:24   0:02 sshd: ayoung at pts/
> >ayoung   18511  0.0  0.0  4604    0 pts/0    SW   08:24   0:00 -bash
> >root     18965  0.0  0.0  4312    0 pts/0    SW   08:26   0:00 su
> >root     18966  0.0  0.0  4624  672 pts/0    S    08:26   0:01 bash
> >named    25776  0.0  0.1 39060 1544 ?        S    08:59   0:00 /usr/sbin/named -
> >root     13602  0.0  0.0  4200  108 pts/0    S    10:30   0:00 /bin/sh /usr/bin/
> >mysql    13657  0.4  0.2 15544 2144 pts/0    S    10:30   0:08 /usr/libexec/mysq
> >ayoung   20722  0.0  0.2  4044 2168 ?        S    11:05   0:00 /usr/libexec/gcon
> >root     20988  0.0  0.0  2700  760 pts/0    R    11:06   0:00 ps aux
> >
> >Nothing looks huge in the memory area.   It hardly looks like the
> >memory is being touched at all (7.2%).
> >
> >I don't get it.     Where did my RAM go?    What else can I do to troubleshoot?
> >
> >Thanks in advance,
> >
> >-Alex
> >
> >
> My first suggestion: install the sar utility and have sar get info, as
> well as get process accounting going.  Whenever I have problems of this
> sort, I use those two utilities.

I think I now have that working.     Sar produces a lot of info, so
I'm going to have to look at it a bit to tell what I'm looking at.    
 Any pointers?

> 
> Second:  Run 'top', and sort by memory usage.  There's a way to get it
> piped out to a file if you want, and you could run that throughout the
> night and capture what's gobbling up RAM.
> 

I think this output will look a lot like "ps aux" won't it?   When I
run top the numbers are the same as ps.

> Third: Check 'dmesg' for terminating processes as a result of exhausted RAM.

dmesg is clean since boot.

> 
> That's what I'd do first, if I were you. If you are wanting to switch to
> Debian becuase you think that RAM problems are inherent to Red Hat,
> you're gonna find that the same stuff happens on Debian - there's
> something on the system that's eating up your RAM, and switching distros
> won't solve it.

I just run some other Debian boxes and I'm better versed with it.    
I'm used to kernel hacking (not the best, I've just had good results)
in Debian, so that's why I will eventually switch.     Not to ignore
the forced migration due to EOL for RH 9.0.

> 
> Let us know what top says when it's slowing down - you should find
> what's eating your RAM (unless there's a process that's hidden and
> eating up your RAM - in which case, you've got bigger problems)
> 
> Adam

Thanks for the advice!



More information about the clue-tech mailing list