Re: Memory loss after long uptime

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

On 04/27/2012 02:10 PM, Konstantin Svist wrote:
Hi all,

I have a strange recurring problem on some of my servers, maybe someone
can help me figure out what might be causing it.

After running a MySQL machine with fairly high load for a while
(~month), RAM usage in top stops making sense. Normally, RES column
accounts for everything currently present in RAM (not swap) and
corresponds pretty well with mem_used-buffers-cache. This is always the
case after a fresh boot and seems to be the case on most other servers.
But that's not the case here:

59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G)
supposedly used by all processes
sum(RES column from top) ~= 35127296k (33.5G)

So where's the other 21G??

I'm pretty sure I'm not nitpicking here, this is >30% of total system
RAM unaccounted for. I've tried stopping all non-OS specific processes
(and restarting some services that seemed to eat more RAM that they
should have (irqbalance)) -- and memory is not reclaimed.

Over the few years that I've seen this problem, I've already replaced
all the hardware and upgraded Fedora from 8 to 14 (currently running
2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of
improvement.

Interestingly, another machine with same hardware & OS runs MySQL in
slave mode to replicate the DB -- that machine has uptime of 134 days
and does not exhibit the same symptoms. In fact, here is its mem footprint:

63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
sum(RES column from top) ~= 45.5G (which makes sense since a few
RAM-hungry processes share memory)

We've seen this before in some of our MySQL platforms. It appears that
some of the older MySQL servers hemorrhage shared memory segments which
will not be reclaimed when the process is terminated.

You might try having a look at the output of ipcs after stopping MySQL
and see if your missing memory is in one or more of the shm segments.
If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be
surprised at how many programs don't release IPC resources--especially
if they are rudely terminated (e.g. SIGSEGV or SIGKILL).

Also have a serious look at updating your MySQL platform if possible.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks@xxxxxxxxxxxxxx -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-     Try to look unimportant.  The bad guys may be low on ammo.     -
----------------------------------------------------------------------
--
users mailing list
users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org



Photo 4 Less

[Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [Fedora News]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [SSH]     [Find Someone Special]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora QA]     [Fedora Triage]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Yosemite Photos]     [Linux Apps]     [Maemo Users]     [Gnome Users]     [KDE Users]     [Fedora Tools]     [Fedora Art]     [Fedora Docs]     [Maemo Users]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Libvirt Users]     [Fedora ARM]

Add to Google Powered by Linux