PDA

View Full Version : Server memory issue, it's eating it all! :(


Rawh
07-13-2009, 01:50 PM
Greetings,

I've been busy trying to get data which support my idea of the FF server eating (to) much memory, even when in idle mode. As gathering data for only one process is a bit difficult, per machine I decided to use munin.

Munin is a program which builds graphs (via the linux RDDtool) by gathering data each X minutes and process it. This is a lot more data then simply cpu or memory usage. Plus there are quite a lot of custom scripts to get more information.
Yet I'm only going to address the memory usage here.

We had an announced power outage last weekend which shows up nicely in the graph (a few hours of no-data) and then it's memory usage when the server starts up again. Due to past experience with server outages and myself not being around to start up the server in that time I made a few bash scripts for the servers I manage. Those bash scripts have been added to the crontab to simply run when the machine has been rebooted resulting in my game servers being started up immediately.

You can sadly not really notice when people are connecting and playing on the server by simply looking at the graphs created by memory usage. The stats website I run and the graphs being build there, with a sum of people around during that time, give out a better view of how many people are around.

It seems that my servers get populated between 14:00 till something around 01:00 the next day. During that time you can see the memory go up quite a lot, totalling up to 500MiB each day. Note: This is memory that goes up for the green part, being application memory use!

http://gs2.snt.utwente.nl/stats/ff1_memusage_week.png

The total amount when such a day ended remains the same, even though there are no more people around on the server to play. The next day however, when again people join to play on the server, at the end of the day it raises itself another 500MiB (more or less). It will continue doing this till the amount of available memory has been filled with just that of the game servers.

By the time the total memory of the server is in use you will start to notice a few things ingame. The server has a few moments of hanging, nothing more then half a second, if not less, but it is noticeable! A few users on my forum have noted this issue to me and while I cannot really feel and/or see the issue myself, I do notice some other ingame players complaining about it during the time I’m there.

I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.

Anyone care to add anything to this issue? Maybe even give a fix on something that might work better? I’m not really waiting for comments as “Upgrade your server” or “Put in more hardware” as this is clearly not the issue! Next to that I’m not quite sure if this is something related to FF or just the dedicated software tools I’m using from steam/valve, it’s just something I noticed while hosting FF servers myself.

-- Rawh

lordpain
07-13-2009, 02:11 PM
I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.


The weird thing is that map cycling causes this; if you don't map cycle the memory 'eating' stops / goes to dead slow.

On the other hand: if you map cycle the caches should clear / refresh and clear on memory allocated.

It seems like a memory leak indeed, restarting the game service seems a not beautiful but working way to clear memory.
But I can recall the same sort of issues with the old HalfLife 1 server engine (TFC) where a scheduled restart was needed too :(

[AE] 82694
07-14-2009, 01:07 AM
So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.

lordpain
07-14-2009, 01:53 PM
82694;443476']So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.

I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...

Rawh
07-14-2009, 02:13 PM
82694;443476']So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.Oh I totally forgot to mention that I did that after the power upgrade.

You can see in the graph that the server increased in memory size from week 7 till 10. After the powerupgrade at our serverrack (week 11) and the downtime after it, I renamed the metamod.vdf file, resulting in that not being loaded.

As you can see however the memory still increased so I'm afraid the conclusion there is that the meta/sourcemod things aren't the cause of it.

I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...Dirty indead but at least it seems to help.

It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans :) Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return :)

lordpain
07-15-2009, 08:07 AM
It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans :) Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return :)

If you run a linux machine you could insert a 'dead time' restart via a cron script in the early morning, like 6 am or something.
Make a small script that:
1. gives the server a shutdown
2. waits period x (say 5 min for a proper shutdown of the halflife server)
3. greps for the server proces
4. kills remaining processes
5. starts the server again

The time you should fire of this script is of course depending on the server load. As it is never a good time to perform the stop start.
The effective downtime shouldn't be more then 10 ~ 15 min.

Rawh
07-15-2009, 08:26 AM
I'm quite known with the term cronjobs and killing the servers.
Atm I already have bash scripts in place to start the server in an event of server crash and a fresh 'boot'.

Current script I have in place is: "killswitch.sh (http://rawh.ampaste.net/f2a7f45d9)"
I already had server starting scripts, so it was easier to make a call to those when the killswitch killed them first.

My servers all run withing a screen (which you can attach/detach if you need the command line and keep running in the event of your terminal freezing up). Used a bit of awk to put up a field seperator and just get the pid of the screen. After which is was an easy task just forcing a kill (kill -9). Not one of the most friendly commands, but useable.

Currently the restart has been running at 05:40 am (GMT +1). I suppose there shouldn't be anyone playing on the servers during that time :)
So far it seems the mem usage graph has been going down, which is good!
Hope this helps the people play on the servers better without delay, lag or anything else unpleasant!!

lordpain
07-15-2009, 08:31 AM
Quick & dirty, as long as it works it's ok :D

Rawh
07-20-2009, 08:48 AM
Well it's been running for a few days now and as you can see the memory usage has been better. Though I have to admit the work around is quite dirty :D

http://gs2.snt.utwente.nl/stats/ff1_memusage2_week.png

hlstriker
07-20-2009, 04:21 PM
Is there a way you can graph your most popular FF server without graphing your other processes? I'm interested in seeing the results.

Rawh
07-20-2009, 05:37 PM
There is a possible way of including srcds_ in munin. I've tried so in the past but did not succeed. There's only a handful of manuals about this on the internet and they aren't that fully written :/