Another gaming related blog...

[ThunderPeel2001]

There's another gaming-related post on my blog, and since the last one went down so well, I thought I'd post a link to it (hope no-one minds). It's pretty "light-weight" compared to the last one, but comments are still appreciated if anyone's get them...

http://thunderpeel2001.blogspot.com/2007/06/gaming-review-scores-off.html

[Wrestlevania]

The problem I have with the whole "review scores" in main stream media is the same problem pretty much every publisher has; the slathering masses demand it.

Let's not beat around the bush here. It's been stated over and over again that an embarassingly high number of "readers" don't actually bother reading reviews at all. The first thing they'll do is flip to the last page and check the score. If it's favourable to them, maybe they'll read the last paragraph -- which, if the reviewer is evenly remotely competant, should be a pithy summing up of the whole review anyway. As a percentage, very few people read reviews from beginning to end. Which is why review scores were invented and why they inevitably appear in big garish boxes using (comparatively) huge fonts.

I also don't like the way so many people criticise a very commonly used form of scoring, just to substitute it for another. You've suggested 1.5 alternatives, but they equate, basically, to the same thing as I've mentioned above; perceived bracketing of scores plus a simple summation of what the reviewer thinks of the game. This isn't a personal attack, but your suggestion is arguably irrelevant in the wider context.

Take, for example, your citation of Halliwell's film rating scheme:

A score of No Stars meant, "Equally unremarkable or missable". (That's right, no stars.)

1 Star meant, "Contains things of merit, but overall, flawed".

2 Stars meant, "Good".

3 Stars meant, "Excellent".

4 Stars meant, "A milestone in film history." (Something that's difficult to assess immediately, really, but that doesn't put off gaming mags trying.)

What exactly is the distinction between 2 and 3 stars here? Descision-making, in the majority of occurances, has to come down to one of two things, either:

1. it's worth doing something, or;
   
2. it's not worth doing something

Having two ratings for "it's worth doing something" (i.e. 2 stars or 3 stars) is making grey when things should be black and white.

What it comes down to, at a very fundamental level, is readership. If you think your readers would be happy with a 0-4 star(s) rating system, go with that. If percentages get you more readers, rate things that way instead. I mean, fuck it; I can vaguely recall one reasonably successful publication using a different rating system every issue -- one of which was based entirely on fruit.

[elmuerte]

Never cared much about the points given in reviews. It's the text that is important. A number (or any grading system) doesn't say anything. And when the grading system because very fine grained it's just retarded. Just like you said "Is a game worthy of 81% or 82%? Who can actually argue such a fine difference?"

The score of a review is useless when you don't know the reviewer. If a reviewer doesn't like game X it wouldn't mean you don't like it. (or vice versa).

A 5 level system would be good enough, but you should mainly focus on the highest, middle and lowest score. Use the 2nd and 4th only when absolutely needed.

5: awesome game, play it

4: just not awesome, but it's absolutely worth buying

3: meh .... good enough to spend money on this, it's enjoyable but it's not a great game.

2: worthy buying if you find it in a bargain bin

1: waste of time and money.

This is my _personal_ grading system. Others can use it too, but it's sort of useless to me unless you know the reviewer good enough. In that case I could convert their score to mine.

For example, I would put Half Life 2 at 3, Deus Ex at 4 (it might have been a 5 if I played it back in 2000), Farcry at 2, Prey at 3, Prey Demo at 5, Tron 2.0 at 5, ...

Usage of "milestones" or "masterpiece" in a grading system is just bad. Because it adds an additional label to the grading.

[Thrik]

Anyone who seriously thinks the 100% rating system is a good idea needs their head checking. I mean, does any industry other than the one we love still use it? I'm fairly sure essentially every other medium one can imagine uses a star system or a basic thumbs up/down system, realising that anything more specific than that is superfluous bullshit. The film industry clearly recognises how subjective a review of such a medium is and are almost always graded with stars unless it's a statistic created by a lot of people voting on it (thus being an average). Games shouldn't be much different in this respect.

I personally think anything that helps sum up the reviewer's opinion quickly is a good idea, but the way the games industry goes about it is lunacy. The thumbs up/down system is my personal favourite as it answers the question of "Does the reviewer like this game?" explicitly. The star system is okay, but as Wrestlevania says the grey lines are shit.

At the end of a day, a reviewer should recommend a game or they shouldn't. Yes, this should be elaborated upon greatly in the text of the review, but ultimately it should come down to a "yes" or a "no". That's how any actual gamer works; I don't think I've come across anyone who can't specifically say whether or not they like a game after completing it.

I don't hold much hope for things changing any time soon though so I'll continue separating it into a thumbs up/down system myself; if it's below 75% (or the 10.x system equivalent) I'll take that as a "don't bother", and if it's above 75% I'll be prepared to give it a go. I'm willing to bet this is how the majority of people respond to review scores.

[ThunderPeel2001]

Thanks for comments!

What exactly is the distinction between 2 and 3 stars here? Having two ratings for "it's worth doing something" (i.e. 2 stars or 3 stars) is making grey when things should be black and white.

Whether you like it or not, there are shades of grey. A simple black and white/yes or no system is not really sufficient with games, nor very practical, in my opinion. Even the most famous reviewers to consistently use such a basic system, Ebert and Roeper, actually pepper their grades by having two reviewers give their opinion, adding a shade of grey.

Both reviewers use a system of seven grades for their personal reviews.

To use your binary system in a practical example; Prey scored averagely in reviews, whereas Guitar Hero II scored overwhelmingly positively... but under your proposed solution, they both would be graded the same: BUY IT. What if you could only buy one? Which should you buy first? What about Deus Ex 1 & 2, KOTOR 1 & 2, Guitar Hero 1 & 2? Such information is completely lost in your system.

Realistically, if you're going have grades at all, they do need to include shades of grey.

Two is too little, a hundred, too much... hence my post.

Since it's extremely unlikely that game reviews will ever lose the scoring system (I really can't imagine the gaming equivalent of Premiere ever coming to light), I was simply trying to propose an alternative that gets across the same information with less fuss.

Maybe I should have made that clearer?

Thanks for the feedback!

[Thrik]

I think having grey tends to veer dangerously into subjective territory. What one reviewer might think is a ridiculously good game, another might think is not so good but just OK. By working with thumbs up/down it encourages the person to make the decision with their own experience and judgement ("Do I like this genre? Does it look good? Do I like the games it's compared to?"). You never know, they might even read the review. Might be a bit ambitious there, though.

I'd personally approach it with a thumbs up/down approach, and then have a little "Get this game if..." title below it with three bullet points. They could be anything: "If you liked Command & Conquer: Generals", "If you loved the classic adventure games", "If you enjoy driving nails through your face", etc.

I personally like slightly offbeat scoring things like that, which you frequently see in magazines and stuff (pros/cons, etc). I find them much more useful and interesting than sterile numbers; unfortunately they're too non-standard/random to ever really be ubiquitous.

Bleh, I don't really care that much anyway. My decisions tend to come from past experience and what other gamers are saying on their blogs, in forums, and on IRC. It's just a shame you end up with great games ignored because the reviewer couldn't think of anything to justify an extra 10%, elevating it to the level of milestone classics.

[Wrestlevania]

Thanks for comments!

Whether you like it or not, there are shades of grey. A simple black and white/yes or no system is not really sufficient with games, nor very practical, in my opinion. Even the most famous reviewers to consistently use such a basic system, Ebert and Roeper, actually pepper their grades by having two reviewers give their opinion, adding a shade of grey.

Right, that's something that makes much more sense than just relying on scores of some sort. It's perhaps more demanding of the reader, but, with a decent spread of "tastes" amongst the editorial team, a publication can embellish the Do/Don't decision-making with alternative viewpoints.

This is somewhat counter-point to my first reply, because it requires those lazy readers to work harder in making their minds up. They can't just glance at one arbitrary score and decide; they'll need to understand what makes one reviewer rate a game differently to another reviewer.

What this doesn't do, however, is debase the simplicity of a Do/Don't rating system. As long as the reader makes the effort to read why.

- - - -

Since writing my initial response, I've been studying how I rate stuff on a regular basis to meet my own needs. Typically this is organising my music in iTunes, and I've noticed I actually only use a 3-tiered system -- instead of iTunes' implied 1-5 system. It essentially equates as follows:

• ***-- Passable / Very Niche
  
• ****- Great
  
• ***** Classic

What this equates to in the real world is that anything under three stars never gets listened to again after I've heard it once. (I don't delete them for historical reasons, however.) Four star tracks I listen to all day long. Five star tracks I save for when I really want something very particular or special to listen to.

This sytem has stood up over time, in that allows for a modicum of grey without being at all complex, or - at the other end of the spectrum - completely meaningless. It's simple enough to allow for quick decision-making, to find something in particular.

A ratings system along these lines for games would suit my own personal needs. But then I'm the one doing the scoring, so it's completely biased and therefore still not that accessible, without first understanding my particular tastes. Which is where the "read the damn review!" bit comes in. Still, I suppose a rough translation of my iTunes example applied to gaming would be:

• --- Avoid
  
• *-- Playable (very niche; dedicated genre/series fans only)
  
• **- Great (most people should enjoy it)
  
• *** Outstanding (a genuine classic)

I agree that this is a complicated subject, but it remains hugely subjective. Given the benefit of hindsight, I also agree that offering 100 degrees of seperation when scoring anything is fucking stupid, especially as it's never apparent what 100% would tangibly equate to. But I also concede that you do need some grey.

My point about Do/Don't decision making was that, taking the "alternate viewpoint" review process into account, you only need only two things in order to make a decision:

1. Empathise with one of reviewer's tastes
   
2. A yes/no response from this reviewer

I can't see it needing to be any more complicated than that. If everyone reviewing the game is howling "buy it!" it's pretty obvious it's a good game.

[ThunderPeel2001]

I think having grey tends to veer dangerously into subjective territory. What one reviewer might think is a ridiculously good game, another might think is not so good but just OK. By working with thumbs up/down it encourages the person to make the decision with their own experience and judgement ("Do I like this genre? Does it look good? Do I like the games it's compared to?"). You never know, they might even read the review.

I think a review without opinions is, well, probably just a press release. Of course you need objectivity, but you have to trust that the reviewer you're reading is attempting to be objective as possible. It's really them who should be saying if something is good or not. It's up to us to decide if we agree with them.

Wrestlevania: Sorry, I'm not sure I follow you. Apart from wanting review scores to be scrapped, you seem to have come to the some of the same conclusions as me, to the point where you've put forward the same grading as me (minus the hyperbolic "milestone" grade).

I think a magazine where every staff writer offers their opinions is, again, impractical and unworkable, though. Firstly, you should be hiring reviewers who have enough skills to be as objective as possible, so you shouldn't, in theory, need everyone else's opinion (why not just have a website that collects public votes and scrap magazines altogether?).

Secondly, a review needs a writer to spend a fair amount of time formulating their opinions to write a comprehensive article, so everyone couldn't do it. (If they did all find the time, you might get six different articles saying the exact same thing... Five unprintable reviews, six fees to pay, one article as a result.)

Thirdly, in magazines where they have done something similar, allowing staff writers to offer alternate opinions in side boxes, it's often that they don't actually have anything to say (much more often than not).

Meaning that really, all the opinions fall onto that one original reviewer again.

[DanJW]

I think the percentage scores are the result of the industry/art form still being so young. Giving things a percentage score is the kind of thing that appeals to male children - it's decisive and slightly technical and falls into the "collecting and counting" phase of male behavioural development. It fitted in well with all those early 90's gaming magazines that had robots and machine-type stuff plastered all over them. So thats where it came from, and it has just never gone away.

As pointed out earlier, no other arts world reviews things this way. The best movie/music/theatre reviews etc that I read tend to be summed up by pull-out quotes from the article. It's these that I scan through to see if the move/whatever is worth reading more about, so seem to fill the same function as a score. 5/star rankings are a good meothod for the masses as well, and are more common in publications which may have a lot of reviews and listings in one issue.

Although the stars are still a scoring system as such it is much closer to a qualitative approach - the only way to assess any form of entertainment. Purporting to have a quantative system of grading entertainment or art is just plain retarded.

[elusivespoon]

I think what gaming really needs is a system like movielens. Basically it works by collecting ratings for various movies from its users, and then finds users that correlate together well. From this information it then predicts a rating for each movie a user hasn't rated based on these correlations. So for instance, if you and another user have always agreed on games in the past, and they like the latest game, it predicts that you will like it. Even if you always disagree with a user, it can use this information to assume you will disagree with them in the future.

So basically it automates the process of considering the opinions of various reviewers taking into account what you thought of their past opinions, and on a much larger scale.

I also like their suggested rating scale:

5=Must See

4=Will Enjoy

3=It's OK

2=Fairly Bad

1=Awful

And I think the meaning behind the stars suggests that some shades of gray are needed. 5 stars is reserved for the best of all time. 4 for those that are good, but aren't the best of the best. 3 for those you're indifferent to, and so on. Without the choice between 4 or 5 stars you can't differentiate between the Grim Fandangos, and that fun racing game. (Though movielens can still work if you only use 1 through 3 stars, or even if you think 1 star is good and 5 is awful, as long as you're consistent).

[toblix]

Shouldn't the system be non-linear, though? Who cares if it's horrible or just really bad? It should be more like:

5=Best. Movie. Ever.

4=Totally Awesome

3=Great

2=Okay

1=Shit

[elmuerte]

Well.. some magazines use detailed scoring and a formula

for example:

Sound (1x): 7

Graphics (1x) 8

Gameplay (2x) 6

which will have the result: 6.75

that would explain some of the odd scores

[Thrik]

Don't take my statement too literally Thunderpeel; I wasn't literally suggesting that subjectivity in reviews is a bad thing. What I meant to put across is that said subjectivity shouldn't be incorporated into a numerical scoring system that's often used alongside hundreds of other games and largely assigned by multiple reviewers who all have different ideas of what makes X deserve X.

The current percentage system in the industry almost seems like a satire of just how bad a scoring system can be; even boiling it down to ten points would be too much, nevermind the ridiculousness we have now.

Perhaps if every single game was reviewed by one guy a numerical system would be fine. Or at the very least, all scores were handed out by one guy who reviews each review and uses his judgement. Kind of like how Metacritic looks at the content of a review if the site doesn't happen to give out scores and then guesses the appropriate percentage.

Having reflected on it slightly while being sat in my chair drinking some coffee, I also concede that a slightly grey scoring system would be okay but I absolutely believe it shouldn't go beyond four levels. The more points you give people, the more ammo you give them to just automatically reject games of the lower points; it's human nature to organise things like this, and nothing makes it easier than assigning two separate levels of scoring to "not worthwhile" which nobody except someone who hasn't read the review will actually bother buying. In fact, they probably won't even read the review. Why bother differentiating them?

By making the system that bit more ambiguous without sacrificing the ability to, as said earlier, differentiate between Grim Fandangos and fun racing games, it encourages people to use their minds a bit more and make the decision properly.

Classic game, great game, worth a rental, not worth anything. Works for me.

[DanJW]

How's this for a hypothetical scoring system; a badge or medal that describes the type of fun the game gives you. There could be a badge for deep, captivating drama, stupid testosterone fun, innocent feel-like-a-kid fun, tetris-like-puzzle-addiction and so on.

And then a big dogturd badge for "do not buy".

Ok maybe not.

[ThunderPeel2001]

Well.. some magazines use detailed scoring and a formula

for example:

Sound (1x): 7

Graphics (1x) 8

Gameplay (2x) 6

Yes! Thank goodness we've moved away from those insanely arbitrary scoring systems, they were the absolute pits! Of course, MobyGames still uses them

I think the percentage scores are the result of the industry/art form still being so young. Giving things a percentage score is the kind of thing that appeals to male children - it's decisive and slightly technical and falls into the "collecting and counting" phase of male behavioural development.

Great point! I think you're probably right about this.