Server memory issue, it's eating it all! :(

[Rawh]

Greetings,

I've been busy trying to get data which support my idea of the FF server eating (to) much memory, even when in idle mode. As gathering data for only one process is a bit difficult, per machine I decided to use munin.

Munin is a program which builds graphs (via the linux RDDtool) by gathering data each X minutes and process it. This is a lot more data then simply cpu or memory usage. Plus there are quite a lot of custom scripts to get more information.
Yet I'm only going to address the memory usage here.

We had an announced power outage last weekend which shows up nicely in the graph (a few hours of no-data) and then it's memory usage when the server starts up again. Due to past experience with server outages and myself not being around to start up the server in that time I made a few bash scripts for the servers I manage. Those bash scripts have been added to the crontab to simply run when the machine has been rebooted resulting in my game servers being started up immediately.

You can sadly not really notice when people are connecting and playing on the server by simply looking at the graphs created by memory usage. The stats website I run and the graphs being build there, with a sum of people around during that time, give out a better view of how many people are around.

It seems that my servers get populated between 14:00 till something around 01:00 the next day. During that time you can see the memory go up quite a lot, totalling up to 500MiB each day. Note: This is memory that goes up for the green part, being application memory use!



The total amount when such a day ended remains the same, even though there are no more people around on the server to play. The next day however, when again people join to play on the server, at the end of the day it raises itself another 500MiB (more or less). It will continue doing this till the amount of available memory has been filled with just that of the game servers.

By the time the total memory of the server is in use you will start to notice a few things ingame. The server has a few moments of hanging, nothing more then half a second, if not less, but it is noticeable! A few users on my forum have noted this issue to me and while I cannot really feel and/or see the issue myself, I do notice some other ingame players complaining about it during the time I’m there.

I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.

Anyone care to add anything to this issue? Maybe even give a fix on something that might work better? I’m not really waiting for comments as “Upgrade your server” or “Put in more hardware” as this is clearly not the issue! Next to that I’m not quite sure if this is something related to FF or just the dedicated software tools I’m using from steam/valve, it’s just something I noticed while hosting FF servers myself.

-- Rawh