Sign in to follow this  
toblix

Console Certification

Recommended Posts

Jonathan Blow about console certification processes (Microsoft's in particular,) and how they need to change but likely won't: http://the-witness.net/news/2012/07/thoughts-on-consoles-and-certification-processes/

The rules he mention are just ridiculous, like everyone having to implement their own screen calibration, and not having a reliable system-wide save system. I'm pretty sure he's correct at the end about what's going to happen with the next generation of consoles. A lot of tiny changes that gives us eight more years of optical media in plastic boxes, terrible user interfaces, more NHFBL Football Live Madden and less video games.

Share this post


Link to post
Share on other sites

Started reading this and had to stop, for starters some of what he says is wrong (the message at the beginning of the game that he refers to is not required nor is the screen calibration).

Further to that I cannot stand people WHO FEEL THE NEED TO CAPITALISE some of their sentence to PUT EMPHASIS on some points. I find it to be the sign of a bad writer.

There is a lot more wrong with it but I am not at liberty to say. I wrote an article that touched on some of this a while back in 2011 (pre-update that made content searches pretty terrible) when Ron Carmel also went on a rant about this:

http://www.arcadianr...rmel-of-2d-boy/

Share this post


Link to post
Share on other sites

Started reading this and had to stop, for starters some of what he says is wrong (the message at the beginning of the game that he refers to is not required nor is the screen calibration).

Further to that I cannot stand people WHO FEEL THE NEED TO CAPITALISE some of their sentence to PUT EMPHASIS on some points. I find it to be the sign of a bad writer.

There is a lot more wrong with it but I am not at liberty to say. I wrote an article that touched on some of this a while back in 2011 (pre-update that made content searches pretty terrible) when Ron Carmel also went on a rant about this:

http://www.arcadianr...rmel-of-2d-boy/

For what it's worth, Blow never said that screen calibration is mandatory. I don't play any console games, so I don't know about the message at the beginning of the game, but from your article it seems like it is a requirement unless you negotiate some sort of special deal with Microsoft, perhaps through your publisher, which isn't exactly something your average indie game developer is going to be able to do, right?

Share this post


Link to post
Share on other sites

It seems like in both of his examples, what he's really complaining about isn't the certification process, it's that a convenient feature isn't built into the Xbox. Sure, it would be great if the console did the work for you, but given that it doesn't, these don't seem like unreasonable requirements to me (especially if they're not even actually required). Title-safe sucks, but I'd rather have the developer spend a little extra time dealing with it than end up with a game where I can't see important information because it ends up displayed off the edge of my screen.

Also:

There is almost no certification process for iOS, so by the Microsoft/Sony/Nintendo theory, the apps should be crashing all the time, everyone should think of iOS as sucky, etc. But in fact this is not what is happening.

Has he used iOS software lately? Like, not the few good apps that people actually talk about, but the other 95% of stuff on the app store? Because there's plenty of sucky, crashy stuff on there. I'm sure it's a pain having to get your stuff certified as a developer, but as a consumer, I actually do appreciate that that process is there so I can be reasonably confident that the software I buy is going to work the way I expect it to.

Share this post


Link to post
Share on other sites
Further to that I cannot stand people WHO FEEL THE NEED TO CAPITALISE some of their sentence to PUT EMPHASIS on some points. I find it to be the sign of a bad writer.

No, he's an amazing writer. He filled Braid with vague walls of text that lead to no full story in particular.

I really hate this stuff though. I think the most glaring issue in with the certification process with Microsoft and Sony is that it costs developers money to "publish" their patch because they need to employ testers each time to make sure the game is not messed up. However if that happened why can't this be handled through the developer's own support, skipping the QA, and releasing another patch if needed. Every platform should be like Steam, where developers have control over their product, not the publishers with the pocket book nor the fee holding developers back. At my job, where we make games live on the internet, I can easily fix art if I catch it, all on my own, making patches and bug fixes very smooth here among a company that is otherwise broken in process.

Brutal Legend on PS3 is broken at about 80% where you can't finish the game unless you get to the point where the game will not reload and play until the end and never turn the console off or exiting the game. The was apparently caused by a patch that they prompt you to install when you start the game. Double Fine made yet another patch for this but EA refused to pay to publish a second one.

Deathspank Thongs of Virtue suffered the same fate, also published by EA, where the Xbox and PS3 versions had a game stopping bug near the end where an item you need may not ever get dropped. Unfortunately EA would not publish the patch, leaving Hothead with some very angry customers. I remember they had a programmer assigned for weeks to fix PS3 save files that were sent to them. 360 owners were shit out of luck because you can't back up your saves on USB.

Share this post


Link to post
Share on other sites

Putting emphasis on certain parts of your text isn't the sign of a bad writer, IMO. You can do it using italics, underlining, bold, or capital letters. The latter is just the ugliest.

It's a common trait in graphics novels to help the reader understand a character's tone, for instance.

Anyways, I digress.

Share this post


Link to post
Share on other sites

Not this years E3, but last years E3, the Giant Bomb crew had a couple guys from Microsoft on who talked about the certification process for a while. I believe with Jonathan Blow and David Jaffee. It was enlightening, and kind of depressing.

Share this post


Link to post
Share on other sites

For what it's worth, Blow never said that screen calibration is mandatory. I don't play any console games, so I don't know about the message at the beginning of the game, but from your article it seems like it is a requirement unless you negotiate some sort of special deal with Microsoft, perhaps through your publisher, which isn't exactly something your average indie game developer is going to be able to do, right?

No it is not a requirement, communicating a save action is. However, if like what Blow describes occurs in his game then he could file for an exception. All developers published by Microsoft are assigned a release manager and producer. The producer is the one that the developer should have taken advantage of to get the exception (as I mentioned in the article), problem is that too many of the indies see these guys as a pain in the arse rather than an asset to be used.

I listened to that Bombcast and it was a group of cert guys not wanting to go into the ins-and-outs, because they can't, being attacked by a developer who didn't really understand what he was talking about.

The title safe area requirement is not a universal requirement for the 3 platforms.Sony have it and Microsoft used to.

As for the whole 'the platform holders should just incorporate it' it really isn't that simple. If you make all save handling, for example, handled by the system software then you stifle potentially creative uses of it. It would be hard to predict people using things like the Mass Effect and Lost Planet 2 system (both of which read saves from other games to unlock stuff) and incorporating it into the system at inception. Instead the requirements are set up to enforce minimum expectancy but with leeway for negociation.

Personally, I think that the Team Meat criticisms are far more important (Microsoft making promises in return for TM meeting certain deadlines and then reneging on all the support) than the 'requirements are annoying' stuff coming from Blow.

Anyway, I can't say more than that or I am just going to get myself in trouble.

Share this post


Link to post
Share on other sites

I don't know much about computers, but I'm pretty sure they could implement a robust saving API that all but guaranteed games were saved reliably, even when the user hit shut down in the middle, while still providing raw access for those special occasions. Really, unless there's some sort of hardware/power failure, there's no excuse for letting save games become corrupted.

Share this post


Link to post
Share on other sites

In regards to the general writing of the article, it drives me crazy when people make arguments like "3 days multiplied by [big number] equals OVER A DECADE OF LOST TIME." It's so vague, abstract, and removed from tangible reality. He doesn't present a very convincing case that those hypothetical 3 lost days have negatively impacted the quality of any particular game, not even his own.

Share this post


Link to post
Share on other sites

As for the whole 'the platform holders should just incorporate it' it really isn't that simple. If you make all save handling, for example, handled by the system software then you stifle potentially creative uses of it.

Here is iOS, "solving" the save problem:

- (BOOL)writeToFile:(NSString *)path atomically:(BOOL)flag

Set atomically to true/YES, and you're done. More detail:

If flag is YES, the dictionary is written to an auxiliary file, and then the auxiliary file is renamed to path. If flag is NO, the dictionary is written directly topath. The YES option guarantees that path, if it exists at all, won’t be corrupted even if the system should crash during writing.

I say "solved" with a hint of sarcasm, because the idea of atomic writes (or a transactional model) has been around forever. It's not a matter of Sony/Microsoft being incapable of doing this, it's just that they won't, for whatever reason.

---

I'm actually cool with platform holders doing QA... remember that leaving QA in the hands of the developer can produce buggy results (I'm looking at you, every East European PC game). But why not just on the initial release? Or maybe a simplified/cheaper QA for patches? Again, the problem isn't that there aren't solutions, it's that Sony/Microsoft aren't willing to look into them.

---

The examples Blow uses are very smartly chosen: these are issues that aren't just an annoyance from a developer perspective, but from a user perspective too. I *hate* having to skip passed the stupid "This game uses an autosave system..." for every game, every time. Similar with configuring the title safe region*.

(* I can understand why it's not a requirement, but for a 2D game it pretty much is. For both the 2D games I developed for XBIG, title safe was a big deal.)

It's telling that these days, when I boot up the Xbox, I have my iPad sitting right next to me, to keep me occupied while I wait to start playing the game. Laugh or whatever, but the competition (iPad, smartphones, etc) boot up in seconds. I've had to wait upto 20 minutes to play a new Xbox game. And on average, I'd say 2-5 minutes before I'm actually in the game.

Usability and user experience *really* matter. I worry that Blow is right in his conclusion:

The edge that both Apple and Valve have going into the future is that they both genuinely care about the end-user experience and want to make it as good as possible. (Their end-user experience is already way better than any of the consoles and they are always working to improve it).

It would not surprise me to see an app-based Apple TV (if that ever happens) take significant market share from the successors to the PS3/Xbox. It sounds crazy, but so did "the iPhone/iPod touch will take market share from the DS". People did not believe that would happen (myself included!) but it kinda crept up on the gaming industry.

I'm worried because the industry keeps ignoring user experience (or at least misunderstands it), but the consumer electronics market is making it clear that it is the most important part of the package.

Share this post


Link to post
Share on other sites

I don't know much about computers, but I'm pretty sure they could implement a robust saving API that all but guaranteed games were saved reliably, even when the user hit shut down in the middle, while still providing raw access for those special occasions. Really, unless there's some sort of hardware/power failure, there's no excuse for letting save games become corrupted.

What about removing a storage device during a write? What if the write takes more than a couple of seconds (Games that save upwards of 40MB in one go can cause writes of 10 seconds or more)?

Share this post


Link to post
Share on other sites

Here is iOS, "solving" the save problem:

- (BOOL)writeToFile:(NSString *)path atomically:(BOOL)flag

Set atomically to true/YES, and you're done. More detail:

I say "solved" with a hint of sarcasm, because the idea of atomic writes (or a transactional model) has been around forever. It's not a matter of Sony/Microsoft being incapable of doing this, it's just that they won't, for whatever reason.

This doesn't take into account removable storage devices and the larger save file sizes on a console or the fact that the console might not have a storage device to write to at all or limited space to write to in the first place.

The examples Blow uses are very smartly chosen: these are issues that aren't just an annoyance from a developer perspective, but from a user perspective too. I *hate* having to skip passed the stupid "This game uses an autosave system..." for every game, every time. Similar with configuring the title safe region*.

(* I can understand why it's not a requirement, but for a 2D game it pretty much is. For both the 2D games I developed for XBIG, title safe was a big deal.)

It's telling that these days, when I boot up the Xbox, I have my iPad sitting right next to me, to keep me occupied while I wait to start playing the game. Laugh or whatever, but the competition (iPad, smartphones, etc) boot up in seconds. I've had to wait upto 20 minutes to play a new Xbox game. And on average, I'd say 2-5 minutes before I'm actually in the game.

Usability and user experience *really* matter. I worry that Blow is right in his conclusion:

The XBLIG scene is a very different one as it is peer review, a lot of the time the people reviewing this don't take into account the severity of each of the requirements (the Title safe area requirement does not exist for XBLA or disc games and hasn't for almost 5 years) some of the things that developers on that platform, who should know better, have failed other people's games is ridiculous (the bullying of the zombie avatar massager guys comes to mind).

If by smartly chosen you mean 'not representative of the requirements as a whole so as to mislead the average user into thinking that the platform holders are insane' then I can agree with that. Your example of the boot up sequence: there is a requirement that says that it must take no longer than 20 seconds to reach the Press START screen, this is exactly in line with your complaint. However, developers and publishers have pushed back on this requirement saying that it is unfeasible. In order to placate them and get the games out this requirement has been relaxed over the years with exceptions for legally required screens (developer and publisher logos), the auto save message was another compromise (note it is the developer's decision to put it in the boot up sequence rather than waiting until after a storage device has been chosen) when developers wanted to make the saving messages more artistically in line with their game and use icons instead.

Blow doesn't mention requirements like making sure that the game handles Ethernet cable disconnection, gamer profile sign-outs, minimum bandwidth requirements, and not displaying debug throughout your game as well as countless others that every single developer I have ever encountered fail and actually do real damage to a user's experience.

Several of the requirements have even evolved due to customer complaints (Title Updates and Save transfers for example), the violent reaction to the Guitar Hero patch is one example as well as people complaining about modified save files.

Share this post


Link to post
Share on other sites

What about removing a storage device during a write? What if the write takes more than a couple of seconds (Games that save upwards of 40MB in one go can cause writes of 10 seconds or more)?

First thing off the top of my head: Always save to internal non-removable storage first, then to removable storage, and if the latter is removed before writing has been committed, prompt the user to reinsert it.

Share this post


Link to post
Share on other sites
What about removing a storage device during a write? What if the write takes more than a couple of seconds (Games that save upwards of 40MB in one go can cause writes of 10 seconds or more)?
Wouldn't an atomic write work? It doesn't matter how long the write takes, since the only stage that can cause a file corruption is the Rename Switcharoo (technical term). If the storage device is removed during the temp file write, you have a corrupt temp file, the last save is still in place, and the OS can clean up the unused temp file at a later point.
This doesn't take into account removable storage devices and the larger save file sizes on a console or the fact that the console might not have a storage device to write to at all or limited space to write to in the first place.

I think the above covers this stuff too... except limited space, that's a good point! Still, could the OS not reserve, say, 50 MB for cache or something? Or even just 10 MB, so games with save files smaller than 10 MB (most games, I think) can omit the "autosaving" message.

The XBLIG scene is a very different one as it is peer review, a lot of the time the people reviewing this don't take into account the severity of each of the requirements (the Title safe area requirement does not exist for XBLA or disc games and hasn't for almost 5 years) some of the things that developers on that platform, who should know better, have failed other people's games is ridiculous (the bullying of the zombie avatar massager guys comes to mind).

Oh man, don't even get me started on XBIG. Totally agree with you, the "Evil Checklist" is total a dick move by members of the community who (as you said) should know better. Stuff like "oh guess what your saves don't work on corrupt memory cards, fail lol!" infuriated me more than even the lack of discoverability.

(in my defence: the first game I did was a launch title for XBIG. I had no intention of doing much else, but then I had a game done in XNA that was just better on console (4 player) so just submitted for the hell of it)

But I think you misunderstood my point. I'm saying that even if it's not a requirement, if you're designing a 2D game being conscious of the title safe region is important. See: every Pixeljunk game, Braid, etc. Microsoft and Sony know this, and 2D games are quite popular, so why not build it into the OS? The first time you launch a game that requires an accurate title-safe region, bring up this config screen, and remember it for every game after. Easier for developers, easier for users.

Your example of the boot up sequence: there is a requirement that says that it must take no longer than 20 seconds to reach the Press START screen, this is exactly in line with your complaint. However, developers and publishers have pushed back on this requirement saying that it is unfeasible. In order to placate them and get the games out this requirement has been relaxed over the years with exceptions for legally required screens (developer and publisher logos)

Loading takes that long, I suppose? That's optical media for you.

But even with XBLA games, where loading should be fast, skipping a half-dozen logos every time is tedious. If the compliance requirements were really geared toward user experience, they'd make it a requirement to be able to skip over these after watching them once.

the auto save message was another compromise (note it is the developer's decision to put it in the boot up sequence rather than waiting until after a storage device has been chosen) when developers wanted to make the saving messages more artistically in line with their game and use icons instead.

Before, or after, it's still annoying. I understand the importance of the autosave message, but an atomic save would mitigate most of the risk. The only real risk would be yanking the coord, and that's a really tiny and silly edge case (even turning off the console via the power button is a software operation, so it could wait for file ops to finish)

Regarding all of the above: I realise this stuff isn't trivial, but it's not impossible either. There are always edge cases, but I don't think that's an excuse to have everyone suffer through a worse user experience. Nail the 99% case, do an okay job with the 1% case, not the other way around.

Blow doesn't mention requirements like making sure that the game handles Ethernet cable disconnection, gamer profile sign-outs, minimum bandwidth requirements, and not displaying debug throughout your game as well as countless others that every single developer I have ever encountered fail and actually do real damage to a user's experience.

Blow probably didn't mention them because he didn't take issue with those requirements.

Totally agree here too, btw! Read the QA bit of my last post. I think there's value in Platform Holder QA, but I also think there's lots of room to streamline some aspects of it.

Share this post


Link to post
Share on other sites

At work right now so can't reply in any great length but some really good points in the last couple of posts. Will reply later if I am not too tired after work.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this