The latest episode of Snoutcast is about location scouting for games. But my brain got ahold of one little twist in the conversation, and then drifted off to a little incident and then... Like, when they talk about the things that GC likes might not be what the players like, and how players are all different. So then I think, since we're all different, how can we even tell what we... Ahem, sorry, I'm rambling.
OK, here's questions for you experienced GC folks to pontificate on in some upcoming LJ post/Podcast/screed/Soapbox oratory: How do you observe playtesters? How do you second-gues interpret their feedback? What have you learned to look for?
Wow, that question (or, rather, set of questions) is pretty wide open. Sorry about that.
It's kind of a touchy subject to talk about. You don't want to say anything bad about playtesters—they are awesome for volunteering to playtest. They're trusting you to watch them when they're, uhm, exposing their fallibility: there's this puzzle that nobody sure if it's possible to solve, and they're taking the chance that you'll be watching them fail to solve it. It makes it kind of hard to talk about interpreting playtester feeback. "We decided to ignore that remark because that guy was basically a loser." Ha ha! I exaggerate. I've been lucky with playtesters so far: they've been smart; they've delivered criticism without getting whiny or stab-by. I don't want to say mean things about them because they don't deserve to have mean things said about them. They've been great! I guess I can fall back on anonymity. And now that I think about stories, I don't think they say mean things about the playtesters. Or about GC. One of these stories is about GC.
(If any of these stories seem to be saying mean things, that means that I'm telling them wrong. I just double-checked; in all of these stories, people were acting reasonably.)
Anyhow, here's some things on my mind which prompted the question...
OK, here's a story where I didn't know what to do with playtester feedback. This was in the 2-Tone playtests. A team had struggled with one puzzle for a long time. The 2-Tone puzzles are of, uhm, varying a-ha-ness. A couple of them flow kinda like you'd expect. A couple of them... uhm, you have to figure out a couple of things before you even get onto the right track. Oh man, sorry about that. So they struggled for a long time. And one of the players seemed, y'know, kind of pissed off about it.
At the end of the day, I asked "So, I gave you more puzzles today than I actually want to use in the final. Which two puzzles should I cut?" I thought for sure that puzzle would be on the chopping block. But it wasn't. Instead it was... one of the favorites?!? The thing was, when that team solved that puzzle, they felt great triumph. Like a mother forgetting the pain of childbirth, this guy had forgotten the agony of the struggle.
So what do you do with feedback like that? Try and smooth out the middle of the puzzle, I guess. But try to leave the ending alone.
What if you can't watch the players? What if they're remote playtesters who are just emailing you? What kind of feedback do you want?
I'm playtesting the Ghost Patrol bang, delivering feedback by email. I'm guessing at what kind of feedback would be useful, going by the kinds of notes that I take when I'm watching playtesters. The various approaches taken, time spent on them. But of course my notes to myself... make no sense to anyone but myself. So I tried a Prestemonic Presentation, by which I mean I copped the style of Prestemon's excellent Solving Really Hard Puzzles Blog.
My notes to myself are terse, and these notes are like that. And most time is spent on false trails, of course. That's how it goes, right? And then I notice that I send off one of my reports and the reply from GC sounds concerned: Oh, we should have warned you, that puzzle's really hard... I think that they thought I was genuinely upset.
You know how people say that email is a bad medium for collaboration? It tends to devolve into flame wars? Because you can't see the other person. So that quirk of word choice seems like they're totally unreasonable and upset. And you have to keep reminding yourself that they're probably nice. And I look back on my emails, and they're pretty easy to interpret as flame-ish. Oh man, this guy is cataloging every possible false step along the way! It's like a litany of things going wrong! From such emails, flame wars can erupt. (It didn't happen in this case. But I wonder how I could have made that email, uhm, better... how to convey "This turned out to be a false lead, but I didn't, you know, resent it or anything.")
I don't know how to convey upsetness-level in an email report. I'm not sure if I (or any playtester) can even self-diagnose upsetness-level. At the time you're solving the puzzle, you're in the frenzy. If you get stuck, then you say "I was angry! That frenzy was the frenzy of anger!" If you solve the puzzle, you say "I was thrilled! That frenzy was the thrill-frenzy! Don't change a thing!"
So... what do folks like to see in an email report?
Something that makes me sad: a puzzle gets some "this oughta change" feedback in playtesting, but then GC doesn't change it. There seems to be some correlation between that and a puzzle's owner not being there to watch the playtest. It's a very compelling experience to watch a team struggle with your puzzle. They're struggling with a puzzle; they probably remind you of those times when your team struggled with a puzzle. When they fail (and they sometimes will fail; that's why you playtest), it breaks your heart.
If your a puzzle owner, I'm guessing that the watch-the-playtesters experience is much more powerful than the get-a-report-later experience. If you get an email report from someone else on GC: Hey they playtesters had a lot of trouble with your puzzle. Do you think we can so something to guide folks through the chainsaw reassembly step? It's so abstract. You didn't watch those playtesters struggle. It's so easy to think Oh I'm sure it wasn't that bad.
Suppose you're one of the GC folks who observed a playtest, and you have to convey some serious feedback to a puzzle owner who wasn't there. How do you convey the... well, the upset-ness levels? This part could change; the players seemed kinda grumpy about it. But this part really has to change.
For the 2-Tone game, for the first few rounds of playtesting, I changed anything that a playtester seemed grumpy about. For the last couple of rounds of playtesting, I changed smaller things, or tried changes thinking "depending on how this goes, I might need to change it back." I didn't want to make big changes after the last playtest: better a known kinda-broken puzzle than a we-don't-know-how-broken-it-is puzzle.
Probably some of those changes weren't necessary; but I don't know which ones. I'd just set up this policy in my head: if somebody says anything about changing some thingy, then that thingy changes. If I were good at second-guessing playtesters, maybe I'd know which pieces of feedback were genuine and which were just that... somebody needed a hug. But I'm not good at it, so I just made changes.
It helps if you know them. For the 2-Tone playtests, I had a much easier time "reading" Team Longshots than I had reading AC Durand. But I'd solved puzzles (and failed to solve puzzles) with two Longshots already. I'd seen them in times of adversity, in times of triumph. I had some idea of what facial expressions to look for. (But then, I was watching them. Suppose that they were remote-solving and sent me email. Would I have known what to do with their feedback?)
Maybe I should talk to somebody who runs marketing focus groups, I dunno.
OK, that was all over the map and in a bewildering set of font styles. If any part of that inspires you to start delivering wisdom, go for it.
