Any playtest experiences here?

[Rxanadu]

I wanted to know if anyone has gone through a playtest of any games in the past or recently.  I recently playtested a game (which I can't go into much detail due to a non-disclosure), and I was taken off the list for subsequent tests after expressing my honest opinion (i.e. without any of the Internet rhetoric you see on most forums) to the lead designer of the game.  Has anyone who playtested a game been through an experience of that sort, where you were told your services were no longer necessary?

[Joewintergreen]

I'm more on the other side, having people playtest my stuff, but I never took anyone off a list because their feedback annoyed me. Negative feedback is often the most useful kind.

Sounds like that guy might just be a bit of a jerk.

[Gormongous]

I've been in a couple of large-scale beta-testing environments where particularly analytical posters have been asked to post less in order to "prevent bias of the playtest results." It really depends on what the developer and/or publisher is looking for in a beta test or playtest. If they just want to generate word-of-mouth or release a demo, someone who crunches the numbers and finds serious flaws in the game is going to be unwelcome.

[Reyturner]

I wanted to know if anyone has gone through a playtest of any games in the past or recently. I recently playtested a game (which I can't go into much detail due to a non-disclosure), and I was taken off the list for subsequent tests after expressing my honest opinion (i.e. without any of the Internet rhetoric you see on most forums) to the lead designer of the game. Has anyone who playtested a game been through an experience of that sort, where you were told your services were no longer necessary?

Was your honest feedback "this sucks and instead of giving you specifics, let me tell you in exhaustive detail how to make your game"?

[Sean]

I playtested a game at Supergiant and it was the first game I've playtested since being in the industry and it was incredibly difficult and really mentally taxing.

 

I'm not going to say a word about which game it was (Bastion or their new one, Transistor) and every verb/button/comment I make is purely made-up in terms of specifics but an honest recollection of tone. Nevertheless, I will say when it came to giving honest feedback, it was really really challenging. I kept wanting to say "I would do X differently, maybe, well, if it were my game, but you probably did X because of Y, which makes sense, but still, if Y is the problem then maybe you should think about it in terms of Z, no?" until I finally realized that wasn't the reason I was there. I had to think about the experience as if I were them and what I'd want to hear and it got a little easier.

 

My rule of thumb became "give these guys problems to solve instead of solutions to my user experience problems." Like, I, swear to God, said something akin to "Couldn't there be a run button?" when what I should've said "I was frustrated I couldn't get to where I wanted to be fast enough." Maybe that means they tune the move speed (remember, all hypothetical scenarios here) or maybe they do implement a run button. Their job, ultimately, is to turn my rambling player-response bullshit into problems to solve BUT I felt better about contributing my thoughts when I didn't ask for things in the game and simply shared my thoughts about the experience. 

 

Anyway, just kind of rambling here, but being on the other side of the table in a playtest fundamentally changed how I listen to them as a designer and also turned me into a neurotic mess. 

[Rxanadu]

The major thing about the game is that it's in pre-alpha, so I would think any criticism on the game so far would be welcome.  However, I did say something along the lines of "I wouldn't buy this game no matter how polished it was."  I explained my reasoning in the first of 3 large paragraphs to the lead designer by basically stating the game was going to end up as a "me-too" game rather than something that was unique.  This was a major issue I noticed when the CEO (who was in the room) was discussing how to implement collectables in their game.

 

I didn't tell this to the lead designer in person; I emailed these concerns to him because I strongly felt that I didn't completely say what I felt about the game in its current state during the post-test meeting.  The overall message I was trying to convey was that if the game evolved along the path it was on when I saw it, it may not sell as well at they thought it would.  Thinking back now, all I really wanted to ask was who the game was targeted for.  For some reason, though, I didn't feel comfortable asking about that, as if it may take it as some sort of insult.

 

Ultimately, I want all games to succeed, and "succeed" in most industries means to sell millions of product to make up for costs.  So, I went for what I thought was the main issue and told them they may not sell for reasons I won't go into in fear of giving too much information of the game away.

[elmuerte]

I try to approach it in the same way as I approach testing: I'm the idiot who broke something (here's how I did it). I'm the idiot who doesn't understand what you're doing here. I'm the idiot... not the creator. A good developer will think about the remarks, but his/her ego is not attacked. A bad developer will simply think you're an idiot and discard your questions.

Talking amongst peers I usually take a different role and try to push people into extremes so that they will fight back so that everything will be exposed/explored. Of course this won't work if the peers don't defend their ideas.

 

 

So it's like Sean said, give them problems to think about rather than solutions.

[syntheticgerbil]

I guess I can give my playtest experiences even though I have never been employed as a playtester. Since I've done art and animation on video games, it usually leads to me playing for a day or two at some point mostly looking for bugs on the art side. However, in every situation I have been encouraged to give my feedback on the game design and how it plays.

 

  However, I did say something along the lines of "I wouldn't buy this game no matter how polished it was."

I don't know exactly what you said and how, but this quote is probably a factor in you being let go, sorry to say. I'm sure there's a part of unfairness and people not having a thick skin or wanting to hear negative criticism out of pride, but this is not exactly a productive comment unless you were specifically asked if you would buy the game. Also as others said, generally I don't think designers want to hear all of the possible solutions to their gameplay issues. I actually do find that a bit backwards though, since QA and playtesting are stepping stones to design.

 

But here goes my experiences:

 

My last job was in the edutainment market consisting of research motivated focus group games actual fun educational games. The focus group games were garbage and pretty much no one in the studio was in charge of design and it was just an acceptance of, "this is crap I know, so lets just make sure the game runs with very little bugs." The edutainment games I generally enjoyed and could see kids having fun with them, so I didn't really need to get too hard on the games. It was especially encouraging when they would be brought to a kiosk in the city's children's museum and kids were genuinely loving them, so they tended to be the true playtesters.

 

My second game industry job which is my current is just a mess of office politics, overbearing free to play garbage, and people with heads up their asses. I've written up a page or two of the four games I've worked on so far on them being unplayable, weak design, and horrible polish. It often doesn't matter how good I make animations because they'll most likely be implemented wrong and no one will give a shit on changing them to work as intended even if it is acknowledged. I've actually been forced to start constantly playtesting our newest game for about 4 weeks now because I've had very little tasks, sometimes a complete week without anything to do. I should be laid off for being dead weight they pay for, but I guess when I get animation tasks, I produce better quality quicker than the Chinese and local outsourcers they pay. It also costs them less hourly because my pay is well below median standard.

 

When it comes to playtesting, no one wants to hear my feedback though as I tend to make it a point to be brutally honest and don't really have to worry about my job or how I explain the feedback since it's not my actual position, but a cheap way to get around hiring actual QA. This tends to have an inverse affect on anyone in the office (who are also not QA) because they want to feel good about the games they wasted months on and offer very little in the way of criticism. So the design leads in every game tend to have excuses why something is bad or just want to think the game they thought up is amazing since it's their baby. I understand to a degree because it's upsetting to hear extreme negative criticism when you do a piece of art work or animation you put a lot of care into. But in the end, every time I have gotten a good degree of specific negative criticism, the draining emotional effect leads to me really paying attention and fixing my current piece to the best of my ability and keeping my grave mistakes in mind for future projects.

 

But maybe my situation tends to be similar to yours as most of my criticisms when it doesn't come to polish are shrugged off as, "Well you aren't the target audience for this game" or, "You don't normally play this genre." Besides that I believe most people serious in games can appreciate multiple genres and how the play and offer feedback based on what they know of as a smooth gameplay experience, it kind of makes no sense why I'm tasked with giving my feedback of the fun in the game when I'm not considered a contender to actually give it.

 

[Dewar]

If you're saying that you wouldn't buy the game, no matter the polish, why would your critisim be useful? I imagine a lot of the overall style of the game has been set by whomever is funding the project and is likely not going to change. Taking the approach of "how can I make what we have better" is helping, but "the design of this game is poor and we need to start again from the ground up" is less so.

[Nachimir]

It's hard to know without more specifics.

However, I did say something along the lines of "I wouldn't buy this game no matter how polished it was."

I suppose they could interpret that in a lot of ways, and quite a few of them would indicate that maybe you wouldn't be useful in future. If the team has inertia toward making a particular game, then probably not much you could say would sway that either (None of that necessarily makes you a bad tester in general).

To an extent, people can just be way oversensitive though. I was once a famously harsh tester in one mapping community, because I was a perfectionist myself and spent way to long making and tweaking projects. Most people valued my feedback; one, who asked me to test after he shat out an ugly, boxy, huge, messy, badly designed map responded with "LOL, none of that really matters" and immediately released it.

[Rxanadu]

When it comes to playtesting, no one wants to hear my feedback though as I tend to make it a point to be brutally honest and don't really have to worry about my job or how I explain the feedback since it's not my actual position, but a cheap way to get around hiring actual QA. This tends to have an inverse affect on anyone in the office (who are also not QA) because they want to feel good about the games they wasted months on and offer very little in the way of criticism. So the design leads in every game tend to have excuses why something is bad or just want to think the game they thought up is amazing since it's their baby. 

 

For the longest time, I've agreed with the notion that changing a game significantly can be hard thing for game developers to do since, "it's their baby," but I now see this a complete malarkey.  The logic behind this quote alone makes no sense: you don't put your babies on the market; you train them to be better than you when they're out in the world; it's their job to create awareness for themselves on their own, whether through their own initiative or with the help of those they know.  What you do with a game you wish to sell, however, is you "whore" them out to the market, creating awareness to those you wish to sell to, and make a profit based on whom you sell to.  

 

Everyone knows what happens to a sold product if it doesn't sell well: it's mourned, not for its distinctive traits or misunderstood potential, but for how it didn't benefit the developers with enough income.  It's shunned by the very people who created it as some sort of leper they wished they had no part on creating.  It's left in the market to fend for itself as the people who wished it well in the past pray for its effect on their social standing to pass like the common cold.  A product someone (or a party of people) wish to sell does not get the "Prodigal Son" treatment: it doesn't get to come back home after a run-in with the cops and one too many contacts with the wrong crowd; it doesn't get a second chance.

 

There is no reason to have a personal attachment to a sold product, as you know from Day 1 how you will treat it depending on its performance in its specific market: either you shower it with praise for bringing you millions; or you will despise (even hate) the product for not doing its only job, which is to make money. 

[syntheticgerbil]

Okay then, opinion deleted because my logic is complete malarkey.

 

You're assuming "everyone knows" what you know and have the prescience to forsee what happens to a creation in all cases in which it is sold. Your analogy becomes very long winded and starts to not even apply to the situation, so I'm not even going to bother with you, Mr. Rude.

[Erkki]

I've playtested some board games and have had my own board game playtested. It's a difficult thing for sure. I find it more difficult when I'm the tester -- it's hard to come up with feedback that isn't "do this" or "do that" (that seems what I want to offer by default), but giving the designers problems to solve sounds like a better idea.

 

With my own game I found some feedback form on a BG forum and modified it a bit, then asked the testers specific questions from that and wrote it down. Got some good suggestions, but even then it turned out that the game just wasn't for some players, and I can totally see how I could decide that "that person doesn't like games like this, I don't want his feedback any more". But actually what I took away from it that maybe the game wasn't as good as I thought and decided to redesign it somewhat (have been postponing that so maybe I just should have finalized the first version as more or less what I originally came up with and tried to get that published somehow).

[Frenetic Pony]

I playtested a game at Supergiant and it was the first game I've playtested since being in the industry and it was incredibly difficult and really mentally taxing.

 

I'm not going to say a word about which game it was (Bastion or their new one, Transistor) and every verb/button/comment I make is purely made-up in terms of specifics but an honest recollection of tone. Nevertheless, I will say when it came to giving honest feedback, it was really really challenging. I kept wanting to say "I would do X differently, maybe, well, if it were my game, but you probably did X because of Y, which makes sense, but still, if Y is the problem then maybe you should think about it in terms of Z, no?" until I finally realized that wasn't the reason I was there. I had to think about the experience as if I were them and what I'd want to hear and it got a little easier.

 

My rule of thumb became "give these guys problems to solve instead of solutions to my user experience problems." Like, I, swear to God, said something akin to "Couldn't there be a run button?" when what I should've said "I was frustrated I couldn't get to where I wanted to be fast enough." Maybe that means they tune the move speed (remember, all hypothetical scenarios here) or maybe they do implement a run button. Their job, ultimately, is to turn my rambling player-response bullshit into problems to solve BUT I felt better about contributing my thoughts when I didn't ask for things in the game and simply shared my thoughts about the experience. 

 

Anyway, just kind of rambling here, but being on the other side of the table in a playtest fundamentally changed how I listen to them as a designer and also turned me into a neurotic mess. 

 

I playtested your guys first game! Ok, I don't know if you were actually at Telltale at the time, but the first Sam and Max episode after Jake put out (on a different website) that playtesters were needed. Considering it was an incredibly short game near the end of production, the playtesting only consisted of a single playthrough. It wasn't bad at all  Obviously all such playtesting needs to be so.

 

Other than that all I can think of is a few betas, none of the them particularly memorable. I once showed up to a multiplayer playthrough of Spec Ops (over a year before it was released) for which I got paid to do nothing because they already had enough people. I guess I keep getting lucky in terms of what I've been asked to playtest so far. Nothing bad at all has happened.

[shammack]

There is no reason to have a personal attachment to a sold product, as you know from Day 1 how you will treat it depending on its performance in its specific market: either you shower it with praise for bringing you millions; or you will despise (even hate) the product for not doing its only job, which is to make money. 

 

Wow, that's pretty cynical. Obviously developers hope their games sell well, but I have to believe that you don't spend years of your life working on anything without developing some sort of personal attachment to it, and some specific vision of what you think it should be.

 

I don't have a lot of playtest experience, but I used to do video editing and get notes from clients and such, so I have plenty of experience with the feeling of having made certain choices for carefully thought-out reasons, ending up with something that you're happy with, and then having someone on the outside come in and tell you what's wrong with it, not knowing the limitations you're working against or all the things you've already tried and why they didn't work. It's easy to get frustrated in that situation. Ultimately your job is to listen to their complaints, use that information to figure out where the real problem is, and decide how/whether it can be addressed in a way that makes sense for what you want the end result to be. But if the feedback is just "I wouldn't buy this unless it were a completely different game," there's nothing useful that can be extracted from that.

 

Even if you are making something whose only purpose is to make money, you're still making creative decisions and judgment calls, and it sucks to have someone shit all over them. Maybe they were being overly sensitive, or maybe they just decided that if that's really how you felt, they weren't going to get anything useful out of you and they'd be better off getting opinions from someone who had some amount of interest in the game in the first place.

[Nachimir]

Remember kids, games are just a product! Any talk of culture and art is just malarkey.

[Rxanadu]

Okay then, opinion deleted because my logic is complete malarkey.

 

You're assuming "everyone knows" what you know and have the prescience to forsee what happens to a creation in all cases in which it is sold. Your analogy becomes very long winded and starts to not even apply to the situation, so I'm not even going to bother with you, Mr. Rude.

 

Sorry if I insulted you in any way.  The overall message of my response was to merely show that, despite the attachment you may have with the game, at the end of the day, if it's made for profit you need to leave your emotions for it behind.  Otherwise, you may be too emotionally hurt for you to continue making games if the game doesn't do well.  

 

Whenever I make a project for school, I tend to get far too emotional whenever I send it in for evaluation.  I work from beginning to end trying to polish it off for the grader.  If I get it back and I get a bad grade, I feel as if the grader spit on me in disgust; it's as if I shouldn't have ever tried.  With experiences like that, I learned to not get too attached to my work.  As a result, I'm able to have a more objective stance on the quality of the work and see what works and what doesn't.  Even if all of that was for naught -and I still get a bad grade-, I shrug it off, learn from my mistakes, and work on the next project so it becomes better than the last.

 

I have the fear that most developers and publishers don't see their emotional attachments (whether it be their product not getting the attention they feel it deserves or the fear of a salary decrease) as a potential negative.  Rather, they should be as detached from the product as they can be to see in an objective standpoint whether it is of the quality they want it to be.

[Rxanadu]

Wow, that's pretty cynical. Obviously developers hope their games sell well, but I have to believe that you don't spend years of your life working on anything without developing some sort of personal attachment to it, and some specific vision of what you think it should be.

 

I don't have a lot of playtest experience, but I used to do video editing and get notes from clients and such, so I have plenty of experience with the feeling of having made certain choices for carefully thought-out reasons, ending up with something that you're happy with, and then having someone on the outside come in and tell you what's wrong with it, not knowing the limitations you're working against or all the things you've already tried and why they didn't work. It's easy to get frustrated in that situation. Ultimately your job is to listen to their complaints, use that information to figure out where the real problem is, and decide how/whether it can be addressed in a way that makes sense for what you want the end result to be. But if the feedback is just "I wouldn't buy this unless it were a completely different game," there's nothing useful that can be extracted from that.

 

Even if you are making something whose only purpose is to make money, you're still making creative decisions and judgment calls, and it sucks to have someone shit all over them. Maybe they were being overly sensitive, or maybe they just decided that if that's really how you felt, they weren't going to get anything useful out of you and they'd be better off getting opinions from someone who had some amount of interest in the game in the first place.

 

That's the worst part of it: I do have an interest in the game.  I just wish there wasn't so much dishonesty with the developers with themselves.  Even talking about the main character's motivations showed a baffling amount of ludonarrative dissonance with the character and what he/she does throughout the game mechanics.  To be honest, it didn't seem as if they knew what they wanted to do with the character, so they gave him/her such an interesting backstory.  Honestly, I wish I was playing the game that character belonged to, instead.  

 

Again, I wish I could go more into it, but the non-disclosure prevents me from doing so.

[Merus]

I can think of a ton of games that were sequels to games that didn't sell well but the creators were too attached to let go. It's nonsense to assume that a game that doesn't sell well for whatever reason is kicked to the curb.

[Frenetic Pony]

I can think of a ton of games that were sequels to games that didn't sell well but the creators were too attached to let go. It's nonsense to assume that a game that doesn't sell well for whatever reason is kicked to the curb.

 

Yep, pretty sure Kojima for one doesn't give a flying CRAP about how well his games sell if he can't do what he wants. This is the guy that will fuck your progression unless you read a tiny note in the manual, or will pull a fast one over on every single customer about what the main character of game even is just because he can. That Metal Gear continues to sell well is certainly because other people enjoy his craziness as much as he does, and not because of any direct commercial aspirations.

[Lu]

I can kind of see why they fired you now, Rxanadu. 

 

 

[Rxanadu]

I can kind of see why they fired you now, Rxanadu.

 

 

 

What?

[Lu]

Okay, a second reading of what I said that comes across with way more dickishness than I intended (I added a smiley!). So, apologies, I didn't mean it like that. I'm way nicer in person, honest. I was referring the following (if I heard you say that out loud and you playtested for me, you could probably pack your shit and get out):
 

For the longest time, I've agreed with the notion that changing a game significantly can be hard thing for game developers to do since, "it's their baby," but I now see this a complete malarkey.  The logic behind this quote alone makes no sense: you don't put your babies on the market; you train them to be better than you when they're out in the world; it's their job to create awareness for themselves on their own, whether through their own initiative or with the help of those they know.  What you do with a game you wish to sell, however, is you "whore" them out to the market, creating awareness to those you wish to sell to, and make a profit based on whom you sell to.

 

There is no reason to have a personal attachment to a sold product, as you know from Day 1 how you will treat it depending on its performance in its specific market: either you shower it with praise for bringing you millions; or you will despise (even hate) the product for not doing its only job, which is to make money.

 

I have the fear that most developers and publishers don't see their emotional attachments (whether it be their product not getting the attention they feel it deserves or the fear of a salary decrease) as a potential negative.  Rather, they should be as detached from the product as they can be to see in an objective standpoint whether it is of the quality they want it to be.

 
It really, really shows that you've never had a full-time gig actually making games. What you are talking about is simply not possible. I think you're confusing emotional attachment with the ability to be able to externalize criticism, as opposed to taking it personally. It's a skill you learn and get better at. The average developer is killing metaphorical babies on a daily basis and, if you really believed in an idea, that can be way harder to deal with than any criticism a playtester might level at your game. And as far as playtesting goes, I think all criticism and suggestions are useful (even if you don't end up doing anything with it), constructive criticism is the best, but armchair game designing is not. It's annoying, more than anything.

 

As a side note, I'd be instantly weary of a developer who referred to his game as "product".

[Ben X]

The overall message I was trying to convey was that if the game evolved along the path it was on when I saw it, it may not sell as well at they thought it would.  Thinking back now, all I really wanted to ask was who the game was targeted for. 

 

This stood out to me as the lead suspect. They probably wanted the playtesters to tell them their personal experience of the game, not project sales figures. "I felt like I'd seen it all before" should be fine. "This game is going to fail" is not. 

[Rxanadu]

Okay, a second reading of what I said that comes across with way more dickishness than I intended (I added a smiley!). So, apologies, I didn't mean it like that. I'm way nicer in person, honest. I was referring the following (if I heard you say that out loud and you playtested for me, you could probably pack your shit and get out):

 

 

 

 

It really, really shows that you've never had a full-time gig actually making games. What you are talking about is simply not possible. I think you're confusing emotional attachment with the ability to be able to externalize criticism, as opposed to taking it personally. It's a skill you learn and get better at. The average developer is killing metaphorical babies on a daily basis and, if you really believed in an idea, that can be way harder to deal with than any criticism a playtester might level at your game. And as far as playtesting goes, I think all criticism and suggestions are useful (even if you don't end up doing anything with it), constructive criticism is the best, but armchair game designing is not. It's annoying, more than anything.

 

As a side note, I'd be instantly weary of a developer who referred to his game as "product".

 

So, what are you suggesting I do, at least in this situation?  Everyone here (myself especially) has already come to the conclusion that I don't know what I'm talking about and I shouldn't be as critical about games during playtests.  Should I talk to the designer again to get my "job" back?  Should I forget about it and continue with making games on my own time (which I've been having trouble doing recently with Unity)?  

 

I don't want this discussion to just be bashing me for not being well-equipped for a playtest and me coming out of it feeling worse off.  I want to learn something about how to do this.  "Armchair development" is a stigma I don't want to be associated with, and my actions (which were based on my assumption that I didn't help the developers out as much during the actual playtest discussion) resulted in negative feelings all around.  

 

I don't talk that much (even in forums), as I'm terrible at expressing myself.  However, every time I try to have a constructive discussion with others about anything, one of two initial results occur:

• My discussion doesn't get picked up and is forgotten
• The discussion receives joke responses
• Others see me as a jerk and respond accordingly

In terms of the latter result, I either end up running away from it all or defending myself with flimsy arguments just to make sure I don't feel bad the rest of the day.  You could tell me to just shrug it off, saying things like, "Welcome to the Internet" all you like, but I've been here for over a decade; I know the Internet is better than that.  I know I'm better than that.  I just don't know how to express it, is all.  

 

But I'm not giving up until I do.