[I re-watched another 2009 GC Summit lecture. In this one, Scott Blomquist of Team Sharkbait talks about measuring puzzle quality. It's kinda a measure of puzzle simplicity--avoiding putting stuff in the puzzle which suggests red herrings. As with my previous set of notes, I paraphrase the speaker except that I have my own comments in [square brackets]]
- Fair warning--this isn't something that lets you plug in a puzzle and get back a difficulty number. It just helps you subjectively judge.
- He really wants our help fleshing out the idea. [Now I feel bad. I thought about this for a few minutes and then stopped.]
- We're talking about information reduction puzzles. [Start with a huge field of dots, reduce it to "GO TO PULGAS WATER TEMPLE"]
- We're not talking about constraint puzzles, though the Theory might cover them. [Theory vs Sledgehammer: Fight!]
- How can we describe these puzzles?
- Information Streams. The starting data is an information stream. But we'll create more of them through...
- Transformation Steps. Transforming information from one form to another. [_ _ . _ _ _ -> "GO"]
- Sample puzzle for analysis: a bunch of automobile mfr logos scattered. E.g., 8 Mercedes Benz logos.
- What's our information stream? Audience starts calling out
- page of car badges
information about mfrsposition of logoscounts of logos
- OK, those things that you people were yelling out. Those are actually transformations. E.g.,
- identify mfr names
- describe locations
- count
- Information streams accumulate; they don't go away. You might hope that teams will forget about the "extra" streams as they tunnel through the next layer of your puzzle, but teams still get distracted by those streams.
- This puzzle was originally on a grid. Teams really, really wanted to use location. They wasted a lot of time. [Argh]
- OK, let's look at counts. And mfr names. And using count to index into mfr name. Also ordering by count. Hey, cool, we solved the puzzle!
- OK, so how do we measure puzzle quality?
- Teams are faced with decisions. Let's look at their decision tree.
- page of badges
- describe locations
- count logos
- id mfr names
- order by count
- index by count
- order by count
- Q: How do you decide that teams won't follow location branch? A: Judgement. [I hope he's right. I thought about constellation, ordering by "northmost" logo.]
- Danger signs of a bad puzzle: Broad tree. Links of tree are weak--non-obvious, error-prone. Lack of confirmation at intermediate steps. Obvious streams not used. Abuse of the "Aha!" transform.
- If you hand teams a CD, that's a broad tree. [Heck yeah.]
- Unfortunately, don't know a way to measure a good puzzle.
- Projects another sample puzzle: So what would be a minimal puzzle withou (brief interruption as someone calls out the answer)
- There's a Puzzle Theory Google Group. There's a Puzzle Theory Wiki.
- Queston time!
- Q: Corey points out: red herring removal ain't automatic. If you give teams a bunch of text strings, sorted alphabetically, a good team will say "oh, it's sorted alphabetically, so this order doesn't matter", but a not-so-good team will still wonder "why did alfa come before benz here?" A: Yeah. Lay out your tree. Then playtest and find out about transforms you didn't anticipate.
- Sean Gugler points out: I call those steps "layers". [Yeah.] If there's a red herring branch, maybe you can tweak it to a mini-solution that redirects to the right way. I've been calling those "signposts".
- Teresa points out: We have this phrase "billions to three". You start out handed a puzzle, there's a billion options. And the team reduces that. Also, This graph seems like a great start for your help system.
- Q: Corey brings the discrete math: Teresa called this a graph and I think she's right--it's a graph [vs always a tree], because teams will try to re-use products of previous "dead ends". A: Yeah, and do teams worry about the dead-ends that they don't use? Like, if there was a signpost that they never hit, do they wonder why that extra data was in there?
- Red says: We think about it this way, in tems of layers. Have talked w/John Owens about how you even talk about this stuff. Might want to think about how you represent the solution. [hey yeah in car logo example, maybe I ordered the mfrs, and then indexed. I still get the answer out, but I took a different path through the tree. Maybe that's what Corey was getting at?]
Has anyone tried mapping out everything that teams could try on a puzzle and turning that into a help system? That seems hard when I think about it, but maybe no harder than coming up with a help system in general.
Labels: link, puzzle scene
Help systems are hard in general. The only way reasonable way I know to make one is to start with everything you can think of that could go wrong, and then playtest the puzzle a lot, at which point you'll discover lots of new things that go wrong. Then you'll iterate on the puzzle in response to the playtest, which will invalidate a bunch of the help system data. And then you'll run the game and hope for the best.
Which is why, I think, most games (Shinteki aside) are converging on live talk-to-GC hinting, despite the subjective unfairness of it all.
Sure, some formalism for noting data streams and possible transformation steps could help you think about how teams could go wrong, but I don't think that's the big problem. I think the big problem is actually figuring out what steps exist in the first place -- i.e. given some presentation, what will teams think to do?
Yeah, you're right. If there was just some way to prevent teams from being so darned creative.