Server memory issue, it's eating it all! :(

Rawh · 07-13-2009, 02:50 PM

Greetings,

I've been busy trying to get data which support my idea of the FF server eating (to) much memory, even when in idle mode. As gathering data for only one process is a bit difficult, per machine I decided to use munin.

Munin is a program which builds graphs (via the linux RDDtool) by gathering data each X minutes and process it. This is a lot more data then simply cpu or memory usage. Plus there are quite a lot of custom scripts to get more information.
Yet I'm only going to address the memory usage here.

We had an announced power outage last weekend which shows up nicely in the graph (a few hours of no-data) and then it's memory usage when the server starts up again. Due to past experience with server outages and myself not being around to start up the server in that time I made a few bash scripts for the servers I manage. Those bash scripts have been added to the crontab to simply run when the machine has been rebooted resulting in my game servers being started up immediately.

You can sadly not really notice when people are connecting and playing on the server by simply looking at the graphs created by memory usage. The stats website I run and the graphs being build there, with a sum of people around during that time, give out a better view of how many people are around.

It seems that my servers get populated between 14:00 till something around 01:00 the next day. During that time you can see the memory go up quite a lot, totalling up to 500MiB each day. Note: This is memory that goes up for the green part, being application memory use!

The total amount when such a day ended remains the same, even though there are no more people around on the server to play. The next day however, when again people join to play on the server, at the end of the day it raises itself another 500MiB (more or less). It will continue doing this till the amount of available memory has been filled with just that of the game servers.

By the time the total memory of the server is in use you will start to notice a few things ingame. The server has a few moments of hanging, nothing more then half a second, if not less, but it is noticeable! A few users on my forum have noted this issue to me and while I cannot really feel and/or see the issue myself, I do notice some other ingame players complaining about it during the time I’m there.

I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.

Anyone care to add anything to this issue? Maybe even give a fix on something that might work better? I’m not really waiting for comments as “Upgrade your server” or “Put in more hardware” as this is clearly not the issue! Next to that I’m not quite sure if this is something related to FF or just the dedicated software tools I’m using from steam/valve, it’s just something I noticed while hosting FF servers myself.

-- Rawh

lordpain · 07-13-2009, 03:11 PM

Quote:

Originally Posted by Rawh

I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.

The weird thing is that map cycling causes this; if you don't map cycle the memory 'eating' stops / goes to dead slow.

On the other hand: if you map cycle the caches should clear / refresh and clear on memory allocated.

It seems like a memory leak indeed, restarting the game service seems a not beautiful but working way to clear memory.
But I can recall the same sort of issues with the old HalfLife 1 server engine (TFC) where a scheduled restart was needed too

[AE] 82694 · 07-14-2009, 02:07 AM

So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.

lordpain · 07-14-2009, 02:53 PM

Quote:

Originally Posted by [AE] 82694

So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.

I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...

Rawh · 07-14-2009, 03:13 PM

Quote:

Originally Posted by [AE] 82694

So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.

Oh I totally forgot to mention that I did that after the power upgrade.

You can see in the graph that the server increased in memory size from week 7 till 10. After the powerupgrade at our serverrack (week 11) and the downtime after it, I renamed the metamod.vdf file, resulting in that not being loaded.

As you can see however the memory still increased so I'm afraid the conclusion there is that the meta/sourcemod things aren't the cause of it.

Quote:

Originally Posted by lordpain

I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...

Dirty indead but at least it seems to help.

It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans

Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return

lordpain · 07-15-2009, 09:07 AM

Quote:

Originally Posted by Rawh

It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans

Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return

If you run a linux machine you could insert a 'dead time' restart via a cron script in the early morning, like 6 am or something.
Make a small script that:
1. gives the server a shutdown
2. waits period x (say 5 min for a proper shutdown of the halflife server)
3. greps for the server proces
4. kills remaining processes
5. starts the server again

The time you should fire of this script is of course depending on the server load. As it is never a good time to perform the stop start.
The effective downtime shouldn't be more then 10 ~ 15 min.

Rawh · 07-15-2009, 09:26 AM

I'm quite known with the term cronjobs and killing the servers.
Atm I already have bash scripts in place to start the server in an event of server crash and a fresh 'boot'.

Current script I have in place is: "killswitch.sh"
I already had server starting scripts, so it was easier to make a call to those when the killswitch killed them first.

My servers all run withing a screen (which you can attach/detach if you need the command line and keep running in the event of your terminal freezing up). Used a bit of awk to put up a field seperator and just get the pid of the screen. After which is was an easy task just forcing a kill (kill -9). Not one of the most friendly commands, but useable.

Currently the restart has been running at 05:40 am (GMT +1). I suppose there shouldn't be anyone playing on the servers during that time

So far it seems the mem usage graph has been going down, which is good!
Hope this helps the people play on the servers better without delay, lag or anything else unpleasant!!

lordpain · 07-15-2009, 09:31 AM

Quick & dirty, as long as it works it's ok

Rawh · 07-20-2009, 09:48 AM

Well it's been running for a few days now and as you can see the memory usage has been better. Though I have to admit the work around is quite dirty

hlstriker · 07-20-2009, 05:21 PM

Is there a way you can graph your most popular FF server without graphing your other processes? I'm interested in seeing the results.

Rawh · 07-20-2009, 06:37 PM

There is a possible way of including srcds_ in munin. I've tried so in the past but did not succeed. There's only a handful of manuals about this on the internet and they aren't that fully written :/

07-13-2009, 02:50 PM	#1
Rawh FF Whiner Server Owner Beta Tester Join Date: Sep 2007 Location: Chair.. sometimes a couch Class/Position: D Engy, D Soldier Gametype: Capture the Flag Posts Rated Helpful 1 Times	Server memory issue, it's eating it all! :( Greetings, I've been busy trying to get data which support my idea of the FF server eating (to) much memory, even when in idle mode. As gathering data for only one process is a bit difficult, per machine I decided to use munin. Munin is a program which builds graphs (via the linux RDDtool) by gathering data each X minutes and process it. This is a lot more data then simply cpu or memory usage. Plus there are quite a lot of custom scripts to get more information. Yet I'm only going to address the memory usage here. We had an announced power outage last weekend which shows up nicely in the graph (a few hours of no-data) and then it's memory usage when the server starts up again. Due to past experience with server outages and myself not being around to start up the server in that time I made a few bash scripts for the servers I manage. Those bash scripts have been added to the crontab to simply run when the machine has been rebooted resulting in my game servers being started up immediately. You can sadly not really notice when people are connecting and playing on the server by simply looking at the graphs created by memory usage. The stats website I run and the graphs being build there, with a sum of people around during that time, give out a better view of how many people are around. It seems that my servers get populated between 14:00 till something around 01:00 the next day. During that time you can see the memory go up quite a lot, totalling up to 500MiB each day. Note: This is memory that goes up for the green part, being application memory use! The total amount when such a day ended remains the same, even though there are no more people around on the server to play. The next day however, when again people join to play on the server, at the end of the day it raises itself another 500MiB (more or less). It will continue doing this till the amount of available memory has been filled with just that of the game servers. By the time the total memory of the server is in use you will start to notice a few things ingame. The server has a few moments of hanging, nothing more then half a second, if not less, but it is noticeable! A few users on my forum have noted this issue to me and while I cannot really feel and/or see the issue myself, I do notice some other ingame players complaining about it during the time I’m there. I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though. Anyone care to add anything to this issue? Maybe even give a fix on something that might work better? I’m not really waiting for comments as “Upgrade your server” or “Put in more hardware” as this is clearly not the issue! Next to that I’m not quite sure if this is something related to FF or just the dedicated software tools I’m using from steam/valve, it’s just something I noticed while hosting FF servers myself. -- Rawh

07-14-2009, 02:07 AM	#3
[AE] 82694 Retired FF Staff Join Date: Mar 2007 Posts Rated Helpful 0 Times	So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod. __________________ I Love GenghisTron . I miss you sooooo Much. LOL.

07-15-2009, 09:31 AM	#8
lordpain Join Date: Jul 2009 Location: NL - Almelo Class/Position: Engi D Gametype: Capture the Flag Affiliations: [BDFC] Posts Rated Helpful 0 Times	Quick & dirty, as long as it works it's ok __________________ BEEP BEEP Chew on my sentrygun

07-15-2009, 09:26 AM	#7
Rawh FF Whiner Server Owner Beta Tester Join Date: Sep 2007 Location: Chair.. sometimes a couch Class/Position: D Engy, D Soldier Gametype: Capture the Flag Posts Rated Helpful 1 Times	I'm quite known with the term cronjobs and killing the servers. Atm I already have bash scripts in place to start the server in an event of server crash and a fresh 'boot'. Current script I have in place is: "killswitch.sh" I already had server starting scripts, so it was easier to make a call to those when the killswitch killed them first. My servers all run withing a screen (which you can attach/detach if you need the command line and keep running in the event of your terminal freezing up). Used a bit of awk to put up a field seperator and just get the pid of the screen. After which is was an easy task just forcing a kill (kill -9). Not one of the most friendly commands, but useable. Currently the restart has been running at 05:40 am (GMT +1). I suppose there shouldn't be anyone playing on the servers during that time So far it seems the mem usage graph has been going down, which is good! Hope this helps the people play on the servers better without delay, lag or anything else unpleasant!!

07-20-2009, 09:48 AM	#9
Rawh FF Whiner Server Owner Beta Tester Join Date: Sep 2007 Location: Chair.. sometimes a couch Class/Position: D Engy, D Soldier Gametype: Capture the Flag Posts Rated Helpful 1 Times	Well it's been running for a few days now and as you can see the memory usage has been better. Though I have to admit the work around is quite dirty

07-20-2009, 05:21 PM	#10
hlstriker QUAD ROCKET Server Owner Fortress Forever Staff Join Date: Jul 2007 Class/Position: Soldier Gametype: Rocket Jumping Affiliations: -g1 ]qS[ -eC- :e0: [ESAD] Posts Rated Helpful 11 Times	Is there a way you can graph your most popular FF server without graphing your other processes? I'm interested in seeing the results.

07-20-2009, 06:37 PM	#11
Rawh FF Whiner Server Owner Beta Tester Join Date: Sep 2007 Location: Chair.. sometimes a couch Class/Position: D Engy, D Soldier Gametype: Capture the Flag Posts Rated Helpful 1 Times	There is a possible way of including srcds_ in munin. I've tried so in the past but did not succeed. There's only a handful of manuals about this on the internet and they aren't that fully written :/

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)