Fortress Forever

Go Back   Fortress Forever > Help & Technical > Server Administration

Reply
 
Thread Tools Display Modes
Old 07-13-2009, 01:50 PM   #1
Rawh
FF Whiner
Server Owner
Beta Tester
 
Rawh's Avatar
 
Join Date: Sep 2007
Location: Chair.. sometimes a couch
Class/Position: D Engy, D Soldier
Gametype: Capture the Flag
Posts Rated Helpful 1 Times
Server memory issue, it's eating it all! :(

Greetings,

I've been busy trying to get data which support my idea of the FF server eating (to) much memory, even when in idle mode. As gathering data for only one process is a bit difficult, per machine I decided to use munin.

Munin is a program which builds graphs (via the linux RDDtool) by gathering data each X minutes and process it. This is a lot more data then simply cpu or memory usage. Plus there are quite a lot of custom scripts to get more information.
Yet I'm only going to address the memory usage here.

We had an announced power outage last weekend which shows up nicely in the graph (a few hours of no-data) and then it's memory usage when the server starts up again. Due to past experience with server outages and myself not being around to start up the server in that time I made a few bash scripts for the servers I manage. Those bash scripts have been added to the crontab to simply run when the machine has been rebooted resulting in my game servers being started up immediately.

You can sadly not really notice when people are connecting and playing on the server by simply looking at the graphs created by memory usage. The stats website I run and the graphs being build there, with a sum of people around during that time, give out a better view of how many people are around.

It seems that my servers get populated between 14:00 till something around 01:00 the next day. During that time you can see the memory go up quite a lot, totalling up to 500MiB each day. Note: This is memory that goes up for the green part, being application memory use!



The total amount when such a day ended remains the same, even though there are no more people around on the server to play. The next day however, when again people join to play on the server, at the end of the day it raises itself another 500MiB (more or less). It will continue doing this till the amount of available memory has been filled with just that of the game servers.

By the time the total memory of the server is in use you will start to notice a few things ingame. The server has a few moments of hanging, nothing more then half a second, if not less, but it is noticeable! A few users on my forum have noted this issue to me and while I cannot really feel and/or see the issue myself, I do notice some other ingame players complaining about it during the time I’m there.

I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.

Anyone care to add anything to this issue? Maybe even give a fix on something that might work better? I’m not really waiting for comments as “Upgrade your server” or “Put in more hardware” as this is clearly not the issue! Next to that I’m not quite sure if this is something related to FF or just the dedicated software tools I’m using from steam/valve, it’s just something I noticed while hosting FF servers myself.

-- Rawh
Rawh is offline   Reply With Quote


Old 07-13-2009, 02:11 PM   #2
lordpain
 
lordpain's Avatar
 
Join Date: Jul 2009
Location: NL - Almelo
Class/Position: Engi D
Gametype: Capture the Flag
Affiliations: [BDFC]
Posts Rated Helpful 0 Times
Quote:
Originally Posted by Rawh View Post
I’ve tried rebooting the game server (not the machine itself) and it seems this does “fix” the memory usage. The memory usage drops down and fills up a bit again when people begin to play at the server. I’m planning to implement a bash script that kills and restart my server each night to address to this memory (leak) issue. It does not in any way address the problem at hand though.
The weird thing is that map cycling causes this; if you don't map cycle the memory 'eating' stops / goes to dead slow.

On the other hand: if you map cycle the caches should clear / refresh and clear on memory allocated.

It seems like a memory leak indeed, restarting the game service seems a not beautiful but working way to clear memory.
But I can recall the same sort of issues with the old HalfLife 1 server engine (TFC) where a scheduled restart was needed too
__________________
*BEEP BEEP* Chew on my sentrygun
lordpain is offline   Reply With Quote


Old 07-14-2009, 01:07 AM   #3
[AE] 82694
Retired FF Staff
 
[AE] 82694's Avatar
 
Join Date: Mar 2007
Posts Rated Helpful 0 Times
So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.
__________________
I Love GenghisTron . I miss you sooooo Much. LOL.
[AE] 82694 is offline   Reply With Quote


Old 07-14-2009, 01:53 PM   #4
lordpain
 
lordpain's Avatar
 
Join Date: Jul 2009
Location: NL - Almelo
Class/Position: Engi D
Gametype: Capture the Flag
Affiliations: [BDFC]
Posts Rated Helpful 0 Times
Quote:
Originally Posted by [AE] 82694 View Post
So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.
I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...
__________________
*BEEP BEEP* Chew on my sentrygun
lordpain is offline   Reply With Quote


Old 07-14-2009, 02:13 PM   #5
Rawh
FF Whiner
Server Owner
Beta Tester
 
Rawh's Avatar
 
Join Date: Sep 2007
Location: Chair.. sometimes a couch
Class/Position: D Engy, D Soldier
Gametype: Capture the Flag
Posts Rated Helpful 1 Times
Quote:
Originally Posted by [AE] 82694 View Post
So uh you think it's FF related? Maybe you need to cut down on the plug ins/admin or disable it. Then you'll know if it's ff related or source/metamod.
Oh I totally forgot to mention that I did that after the power upgrade.

You can see in the graph that the server increased in memory size from week 7 till 10. After the powerupgrade at our serverrack (week 11) and the downtime after it, I renamed the metamod.vdf file, resulting in that not being loaded.

As you can see however the memory still increased so I'm afraid the conclusion there is that the meta/sourcemod things aren't the cause of it.

Quote:
Originally Posted by lordpain View Post
I don't think it's FF related...
Try running a 'plain' HL2 DM server, it'll clog up the mem as well.
I'll stick to old faithful: restart the server instance now and then.
Quick & dirty...
Dirty indead but at least it seems to help.

It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return

Last edited by Rawh; 07-14-2009 at 02:15 PM. Reason: putting in another quote.
Rawh is offline   Reply With Quote


Old 07-15-2009, 08:07 AM   #6
lordpain
 
lordpain's Avatar
 
Join Date: Jul 2009
Location: NL - Almelo
Class/Position: Engi D
Gametype: Capture the Flag
Affiliations: [BDFC]
Posts Rated Helpful 0 Times
Quote:
Originally Posted by Rawh View Post
It's a bit hard to exactly plan when to do it however. The amount of players on our server are a bit out of sync with their normal plans Last night it rebooted in the middle of a small match on AvD. Ofcourse after the forced reboot of the server, the players didn't return
If you run a linux machine you could insert a 'dead time' restart via a cron script in the early morning, like 6 am or something.
Make a small script that:
1. gives the server a shutdown
2. waits period x (say 5 min for a proper shutdown of the halflife server)
3. greps for the server proces
4. kills remaining processes
5. starts the server again

The time you should fire of this script is of course depending on the server load. As it is never a good time to perform the stop start.
The effective downtime shouldn't be more then 10 ~ 15 min.
__________________
*BEEP BEEP* Chew on my sentrygun
lordpain is offline   Reply With Quote


Old 07-15-2009, 08:26 AM   #7
Rawh
FF Whiner
Server Owner
Beta Tester
 
Rawh's Avatar
 
Join Date: Sep 2007
Location: Chair.. sometimes a couch
Class/Position: D Engy, D Soldier
Gametype: Capture the Flag
Posts Rated Helpful 1 Times
I'm quite known with the term cronjobs and killing the servers.
Atm I already have bash scripts in place to start the server in an event of server crash and a fresh 'boot'.

Current script I have in place is: "killswitch.sh"
I already had server starting scripts, so it was easier to make a call to those when the killswitch killed them first.

My servers all run withing a screen (which you can attach/detach if you need the command line and keep running in the event of your terminal freezing up). Used a bit of awk to put up a field seperator and just get the pid of the screen. After which is was an easy task just forcing a kill (kill -9). Not one of the most friendly commands, but useable.

Currently the restart has been running at 05:40 am (GMT +1). I suppose there shouldn't be anyone playing on the servers during that time
So far it seems the mem usage graph has been going down, which is good!
Hope this helps the people play on the servers better without delay, lag or anything else unpleasant!!
Rawh is offline   Reply With Quote


Old 07-15-2009, 08:31 AM   #8
lordpain
 
lordpain's Avatar
 
Join Date: Jul 2009
Location: NL - Almelo
Class/Position: Engi D
Gametype: Capture the Flag
Affiliations: [BDFC]
Posts Rated Helpful 0 Times
Quick & dirty, as long as it works it's ok
__________________
*BEEP BEEP* Chew on my sentrygun
lordpain is offline   Reply With Quote


Old 07-20-2009, 08:48 AM   #9
Rawh
FF Whiner
Server Owner
Beta Tester
 
Rawh's Avatar
 
Join Date: Sep 2007
Location: Chair.. sometimes a couch
Class/Position: D Engy, D Soldier
Gametype: Capture the Flag
Posts Rated Helpful 1 Times
Well it's been running for a few days now and as you can see the memory usage has been better. Though I have to admit the work around is quite dirty

Rawh is offline   Reply With Quote


Old 07-20-2009, 04:21 PM   #10
hlstriker
QUAD ROCKET
Server Owner
Fortress Forever Staff
 
hlstriker's Avatar
 
Join Date: Jul 2007
Class/Position: Soldier
Gametype: Rocket Jumping
Affiliations: -g1 ]qS[ -eC- :e0: [ESAD]
Posts Rated Helpful 11 Times
Is there a way you can graph your most popular FF server without graphing your other processes? I'm interested in seeing the results.
hlstriker is offline   Reply With Quote


Old 07-20-2009, 05:37 PM   #11
Rawh
FF Whiner
Server Owner
Beta Tester
 
Rawh's Avatar
 
Join Date: Sep 2007
Location: Chair.. sometimes a couch
Class/Position: D Engy, D Soldier
Gametype: Capture the Flag
Posts Rated Helpful 1 Times
There is a possible way of including srcds_ in munin. I've tried so in the past but did not succeed. There's only a handful of manuals about this on the internet and they aren't that fully written :/
Rawh is offline   Reply With Quote


Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 01:35 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2024, vBulletin Solutions, Inc.