Closed beta test VS Open beta test

[Grayfox]

I was interested why the dev-team of FF decided to keep it a closed beta. The most popular and successful online FPS ever was even released when it was stil l in alpha stage.

Anyway, let me hear some arguments from both sides. Preferably longer than 3 words and not being "the dev-team only wants to release FF when its completely finished so it wont lose in popularity by giving an unfinished impression and it wouldn't be representing the final product". So what about Counterstrike, released as an early alpha? What makes FF so different from CS that under no conditions FF could be released as an open beta? If you look at CS, it is pretty clear releasing the game in a very early stage doesnt damage the community, quite the opposite.

I've read through the forums, but still have failed to find any decent arguments on both sides that have been thought through and couldn't easily be countered.

[Nezumi]

Defrag's 4-21 2006 dev jernal:

Originally Posted by Defrag

I've been posting in a thread on the FF forums regarding release methods and the different obstacles we've had to overcome when modding the source engine as well as the release model to choose. I've decided this is sufficiently interesting to tidy up and post in a dev journal. I'm not saying I'm correct; this is merely one guy's perspective. You can debate many aspects of the issue. Anyway, now that I've got the formalities out of the way, on to the dev journal itself.

The forum thread was simply about the way in which modders are approaching the release conundrum these days. You can go two ways (or fall somewhere in-between):
Release early, release often (I'm going to call this RERO from this point in)
Release late and try and get it largely right the first time. (RLATAGILRTFT. Forget it)

The thread starter simply asked why we didn't knock up some basic maps and use placeholders to facilitate an early launch. Quite simply, I feel that the vast majority of mods are destined to follow the late, complete release model. Why? Because unless you have an idea that is so gripping and amazing from the start, I believe the mod is destined to fail when following this paradigm.

Quirky and cool mod/game ideas like Garry's mod, Geometry Wars and other stuff still exist in gaming, but it is my opinion that, unless you've got one of those ideas, going the long-winded make-it-awesome route is pretty much the only choice, as that's what everyone else is doing; it's what the gamer now expects. Most of the HL2 mods that released early (and some released often, too) have now fallen by the wayside. Members of Valve have consistently stated why releasing as early as possible is a great idea and that it lets modders have the advantage over games companies. However, for HL2, how many mod success stories have we seen when it comes to early starters? I can't really think of any unless the idea was a fairly simple one. Why? People expect cool stuff as technology moves on. If you make a mod that has some cool stuff but looks really plain or like it's using 4/10th of the engine's power, people won't even look twice. Bear in mind that we're roughly 18 months into HL2's tenure and (Garry's mod aside, well done to Garry) there really aren't any really popular mods out there. Nothing has caught the imagination like Counter-Strike or Day of Defeat.

Take Dystopia: I played the first release for a few hours and thought it had some cool stuff but was really rough around the edges. Have I ever tried it again? Nope. I'm as guilty as everyone else when it comes to expectations and attention span. I'd rather spend my time playing something like BF2 (well, not anymore haha, they ruined it) than wait on a game to improve a lot before I play it, nor am I inclined to continually check up on a mod and chart its progress. Unless dystopia shows some new stuff that just blows me away and proves the mod is now several orders better than it was when I played it, I'm never going to touch it again. I have a feeling the majority of gamers operate on the same principles (regardless of whether those principles are sound). Please note that this is not a slight on dystopia as it's one of the better HL2 mods around, but I'm just explaining my stance on why I think the RERO model is flawed when dealing with today's gamers.

Back when CS et al arrived, modding was in its infancy. Expectations were low to zero and competition was non-existent. Making something look great for HL1 was far easier than it is for HL2. By this I simply mean the gap between poor/average and awesome has grown a lot. As technology increases, modders have to be willing to spend a lot of time and resources to make something look cool. Example: For HL1 I could knock up a good looking map in a few days and then spend a few weeks tweaking & playtesting it. For source, the first two days will go on the basic layout and I'll perhaps spend a month solid on tweaking & polishing. I'd much rather test early with dev textures than test later when the theme is developed, else I'll have to spend a significant amount of time correcting issues. The days of putting a few texture lights into a room and making some nice brush based architecture are over. Do this for newer engines and people will just grimace when they see it. Don't believe me? Go check out some mod news posts on the community news sites. People don't like ugly mods. Many won't even play an ugly mod unless there's nothing else out there. It takes many, many times more time and effort to make a source map and it can involve multiple people. So far, the only person I know who's successfully made an enormous and gorgeous custom map by themselves is hessi and it took him over a year!

Now take the mapping issues and extend this to every aspect of mods; for example:

-Character models grow from 800 triangles to 3-4k and some require normal and spec maps to complete the effects. With high budgets and better animation, many errors that would be passable (such as proportional issues or rubbish UVing) are now far more glaring.

-Each vertex can be influenced by multiple bones, so enveloping is more complicated etc.

-Prop models are required because brushes just don't cut it any more. You then need someone to model, uv, skin and compile the props.

-Textures increase from 128x128 up to 1024x1024. With greater image resolution comes greater responsibility and expectation. You need skill to make a high resolution texture justify its size. This goes for player and prop skins plus map textures.

-All of the asset creation pipelines have increased in complexity. I remember learning to compile a prop; gone are the days of just dropping a sodding mesh into a directory and being done. You've got smds, qcs and lots of hair tearing to contend with, not to mention a lot of the tools are command-line based. A lot of artists aren't technically minded and take a while to truly get to grips with this stuff, particularly when animation is involved. Relying on 3rd party tools that get broken by steam updates is also a joy.

-Oh, and the SDK is a fairly bewildering thing until you get a handle on it, and even then it's still fairly complicated at times.

Everything has gone up in complexity just as expectations are rising. I wonder how long the mod community can sustain this before it implodes due to a skills vacuum and rising program and tool complexity. Unless processes are streamlined AND the tools required are affordable, things don't look good for the future. In my opinion, the mod community is slowly wilting under the demands of utilising the power of modern technology.

Valve's SDK documentation is lacking in many areas, too. Back in the day things were a lot simpler. This rise in complexity is inevitable, but I do think valve and other companies need to rethink their support for modders. Shipping a bunch of command line compilers and scant or vague (or app-specific) tutorial resources makes it extremely tough for the average person to get into modding. The valve wiki may look expansive and all-encompassing, but go search for something particular and you'll probably find it doesn't answer your question, or the wiki only provides an overview as opposed to the nitty-gritty. For HL1, there was a lot less you could do, so it was far easier to figure things out for yourself.

Again, it's a natural problem, but it's something that has to receive some attention in future if modding is to survive as a true fun pastime that anyone can jump into. I'm not pointing the finger at Valve or anyone in particular. Valve have supported the mod community better than the majority of developers; they also realise how important a thriving mod community is when it comes to longevity of their products. One might surmise that valve's interest in mods is not altogether altruistic. It's good business sense to lend a hand to anything that can aid your product's shelf life, or indeed generate even more revenue (particularly with Steam's rapid rise). Valve is a forward thinking company with a smart business acumen. I just wonder whether they can do more? The lack of quality HL2 mods out there would suggest to me that they definitely can.

This is a time when professional tools like ZBrush are evolving to help developers keep pace with the strain of expectation. Just because an engine can push several million polygons these days doesn't mean things will look the part -- you need the skills to fully exploit these new limits. This is only going to get worse as complexity increases and the tools and provided information do not keep pace. If your mod looks like it is under-utilising the engine, it's hard to get people to spend the time downloading & installing it when other games and mods look a lot more appealing. Saying that, people are still fairly eager to try HL2 mods regardless of their quality, but I personally think this is because there's a huge mod vacuum out there at present. There is no counter-strikes or Day of Defeats occupying centre stage.

Back to the topic: To my mind, the RERO model is now a contentious issue. It definitely applies to some mods (mods with new and innovative and/or world-beating ideas), but I feel it is unfair to vaunt it above others. If your mod concept is something familiar but with a new spin (say a WW2 game with some new ideas) then people will immediately expect gritty combat, shells bursting everywhere, heavy machine-gun fire and authentic art and design. Now, I pose a few hypothetical questions: What happens if you release the mod six months in with a few rough player models, a bland map or two and a tonne of bugs? Would you honestly expect people to download and play it, then stick by it as the mod progresses, or would they go play Call of Duty 2 or Day of Defeat instead?

The Red Orchestra leader came out and basically said the opposite: Keep going and making it better until you're happy with it, then release it. People won't look twice if you don't get it (largely) right first time. I'm inclined to agree with him for most mod types. There's an enormous amount of games and mods out there waiting to be played, so unless you've got a kickass central idea that is world-beating, why would you expect anyone to play something that is half-finished and comparatively ugly? It's even more pronounced for mods based on a non-original concept. People have predefined notions and expectations for traditional mods and it's pretty darn hard to be truly original these days. Just as games companies are throwing potentially hundreds of employees and tens of millions in capital at next-gen games, modding is falling into the same trap. Fortress Forever is just as guilty as the majority of 'big' unreleased mods (Insurgency, Nuclear Dawn, Black Mesa Source etc.) when we go the one-big-release way; it potentially forces other mods to follow suit and it also raises expectations. This can only go on for so long because I don't think the modding community can sustain it.

Then we've got the PR situation and mods doing media releases and so forth. People ooh and ahh at stuff they can't play and won't be able to play for ages. Is this the way we should be doing things? Again, it raises expectations. People see what they could be playing and ultimately may not be satisfied with what they can play right now. In our defence, Fortress Forever happily went on in secrecy for a long time and was mainly revealed to stop other Source TF teams forming. This isn't to say that other people couldn't have done a better job or that we were being greedy, it just made sense for us to try and unite a passionate community and pool the skills to do the best job we can. Having several rival factions would've meant slower progress and potentially a split community, plus FF already had a huge head-start on other teams due to the groundwork we laid previous to HL2's release. Once we'd revealed the mod, we quickly had a lot of interest and people requesting updates (the "bitch about updates" thread on our public forum where people can complain about slow progress is now something like 90 pages long), so we felt obliged to show people. Also, it was very hard to recruit people without showing our work. We might collapse under our own hype like other mods, but I can't really see a different path to choose. It sucks when you realise you're participating in something which may ultimately harm the mod community. We're just a brick in the wall, but it's a bloody huge wall. It's dawned upon me that we're part of a mod community which operates on hype, fancy screenshots and promising the moon on a stick (does anyone ever actually want the moon on a stick? I'd much rather have a couple of quid or maybe Kelly Brook to come around and clean my windows). It does strike me as odd that a lot of mods have actual dedicated PR managers etc. What ever happened to "if you build it, they will come"? Modding is slowly becoming more corporate -- everyone is learning the art of bullshit. With the advent of people attempting to sell mods it's only going to get worse. Bullshit is a lot more prevalent when there's money involved.

Once again I'd like to reiterate that the RERO method does definitely have its uses. If your central mod idea is something that ideally needs rapid prototyping to evaluate whether you've actually got something to run with, then an early alpha release with rapid iterations afterwards would definitely serve as a useful tactic.

One big release also has risks associated with it. Imagine spending two years on making some mod and then finding out everyone hates it? It could yet happen to us, but we're willing to take that chance. We're more mindful of the fact that if we release a really rough version it's possible that a lot of the people who initially download the mod and can't see past the rough parts won't ever give us a second look.

Lastly, I'd again like to say this was all written as a stream of consciousness and written using my own observations and experiences. Am I qualified to talk about this stuff? Do I sound preachy? Take this as you will; it's just my brain leaking onto my keyboard. I don't have any answers, either (or at least I haven't thought about it enough, yet). Even though this was a fairly monstrous but weary sounding dev journal, I'd to cap this off by saying that progress is going well and that we're doing a huge round of bug squashing at the moment. Once this is taken care of, we'll be much better positioned to work towards a release. The training system is underway and we're working on new ways to tie all of the help available to players into one tight package. We must support the new blood or the mod will never be successful.

[Afty]

definately more than 3 words there

I have to say i am for closed beta (with me in it - go on you know you wanna) im really excitied about this mod and i dont think it would feel the same after release if you saw it with all the bugs and holes in.

Also with a small beta team you get a better overview of how the bugs are working and you can direct people to specific all test certain areas, with an open beta you would (i expect) get flooded with queries about the sniper tracer being imba etc etc

[GhOsT]

I love how people throw terms around like alpha and beta and seem to have no idea what they really mean. There also seems to be a trend going lately where people expect every game to have an open beta and be allowed to play in it.

There's a difference between a beta and the final release. The whole point of a beta release is to let a wider audience review your work and find bugs with it. Just with anything else, if you've worked on something for a while you may become short sighted on issues that come may come up.

I would say the vast majority of people wanting to get into the FF beta really just want to get to playing the game. They have no intention at all on trying to find bugs and giving real feedback on the found bugs.

Edit: Took out part referring to CSS since I have no idea where I got that from in the original post.

[mervaka]

Originally Posted by GhOsT

I would say the vast majority of people wanting to get into the FF beta really just want to get to playing the game. They have no intention at all on trying to find bugs and giving real feedback on the found bugs.

that pretty much sums it up. people throw out their toys when they dont get in on the beta, but if it were open, they'd snuff it at the first hurdle and move on.

and in regard to the first CS betas, the mod scene was in a relatively infant stage back then. you can't compare it to the HL2 mod scene.

i must say i'm guilty as far as supreme commander goes. i got a copy of the beta, and didnt contribute to the testing because i knew there were a million other joes out there who could do that for me. a smaller team are a lot more dedicated to the job at hand. don't call me a cheapskate though, because i went out and bought this game after. it's truly awesome.

[Torn]

I think peoples expectations are overly high, partly due to the quality of previous tf mods. Releasing something will just make loads of people come the the forums and nit pick about absolute bs thru 100 threads.

From what i've seen so far this mod looks excellent and I cant say i've seen a better / more professional mod in development atm.

yer when cs game out i think the only popular mod at the time would have been quakeworld: teamfortress (which i was playing at the time =P )... cant really compare it these days when the mod world is flooded with heaps of new mods, in comparison to days when some of us were playing Platoon - ouch if you even remember that (way old mod!)

ghost: yeah I didnt really notice the change in the qwtf community in Aus when tfc came out, but cs definitly took alot of players away...

[GhOsT]

I'm pretty sure when CS came out the most popular mod was TFC. If not TFC than definitely TF.

[Agret]

Originally Posted by Defrag

Take Dystopia: I played the first release for a few hours and thought it had some cool stuff but was really rough around the edges. Have I ever tried it again? Nope.

Considering they only did one "demo" release and have only recently done another one (The major V1) I don't know why he bothers to mention it. They only had one freaking map!

[GhOsT]

That's exactly his point though. They released something for the masses to play really early in the game's development. That was enough for people to try it and pass judgement on it then, even though a later release will add a lot more to the game.

[IvaQuaL]

the dev's goal is to have a high quality first release, with the least possible quirks. hence, a closed beta makes sense.

[Fail]

Open betas are 99.9% of the time just to get the product out to the users. Game developers know you aren't going to give any meaningful feedback or bug reports nor do they expect you to. These 'open betas' are more like 'it's a demo but people don't like the word demo so we used open beta to also make you feel like you're contributing'.

Also, Agret, there have been 4 demo versions of Dystopia. Update 1, Update 2, Update 3, and Update 4/5. v1 is a final product. Updates from here on in will be adding content and fixing bugs.

[Jiggles]

Yeah, part of the challenge for us is that we already get more feedback/bug reports than we can implement/fix in a timely fashion. We're especially thin on the coding front, and unfortunately, this area requires the most work before a "release." Mirv was a coding god, but, sadly, the more talented our dev members, the greater the chances that they get a “real” full-time job in the industry and thus have little time for FF.

So, don’t think we’re waiting to release a supremely polished version of FF – we’re deciding many things are “good enough” left and right, but there are certain things that just can’t be put off. For example: making sure the LUA mapping stuff is setup properly so we won’t go changing it heavily later, thus breaking every custom map.

Anyway, we’re working as fast as possible, and if you know anyone that has some serious coding talent (and almost more importantly, is dedicated), we’re still “hiring”!

[s0undch4s3r]

I don't want to play it until the Devs are happy with the final result.

The End.

[Anshinritsumai]

Originally Posted by s0undch4s3r

I don't want to play it until the Devs are happy with the final result.

The End.

Indeed I must concur.

[Lost]

Ohmgee, Eff Eff will nevar be dun!!11

[Everything]

Y0u aar teh poast0r

[Lost]

/me poasts in Kenny's face.

[Circuitous]

Sumtimes I think Lost poasts just 2 poast.

[Johnny Mullet]

I agree.

[TheodorusRex]

I'm mixed on the idea. As a mod team that has released so much media, has a dedicated following and with such a huge 800 pound gorilla in the room (TF2) I thought they would have released an Open Beta. Within a few months, a lot can happen, but I thought they would take a chance at some point in time. Even if it isn't done, a playable game is better than a forum about a non-playable one (at least, for the majority of players).

I doubt many share my opinion on this, but I feel more confident in a mod team if they feel confident enough in their mod to just go ahead and release it. It's going to be bugged, but hell, the release'll be bugged too. Unless it's in a terrible shape, I'm all for open beta. Soon.