What is the BrainQuake Score?

If you have downloaded the new BrainQuake app (discussed in previous posts on this blog), you may have clicked on the “Score” button and seen your BrainQuake Score. See image below.

After completing 111 puzzles in the BrainQuake app, I clicked on the “Score” button to see my BrainQuake Score. On this run through the game, I was playing as a strong player (hence the username), so was already starting to meet some challenging puzzles. All my 111 puzzles earned three stars, so I had amassed a total of 333 stars. Image by Keith for BrainQuake.

The first thing you will have seen is you are not presented with a single (percentage) number, but three, labeled: Performance, Persistence, and Creativity. (See image below.) The first tells you how good your solutions were. (It’s not a simple question of being right or wrong. Many BrainQuake puzzles don’t have unique right answers, and almost all admit varying degrees of optimization — just like most math problems in the real world!)

Performance is the closest we come to the familiar grade you get from the majority of math assessments that are available today.

This is the BrainQuake Score I had earned in completing 111 puzzles. I was playing as a strong player, solving all puzzles with 3-star solutions. This was reflected in my Performance score of 100%. Many early puzzles do not require much, if any, creativity, but by puzzle 111, I had solved enough puzzles that my Creativity score had climbed to 59%. I solved each puzzle at the first attempt, so had not had a chance to demonstrate Persistence, which measures how a player responds to failure — either not getting an optimal solution or not getting a solution at all. Responding positively and effectively to failure is arguably the most important mathematical skill of all. In mathematics, learning occurs only when we fail. So math learning is all about failing. Image by Keith for BrainQuake.

I should mention that this is the report I got when I asked to see my Score after completing 111 puzzles. The puzzles don’t get difficult for me until further into the game, so my Performance score was 100%. (I’ve also played every puzzle in the game many times over during development and testing.)

The Performance score is related to the number of stars you earn for each puzzle you solve. For instance, if you get 3-star solutions to all puzzles, you will have a high Performance score. But the Performance metric is much more finely grained than the star system, which is primarily a game feature (not a mathematical assessment).

The Creativity measure takes a while to gather enough information to produce a reliable score, and many of the more simple puzzles provide relatively little opportunity to exhibit creativity. (The game has an adaptive engine that selects puzzles at the appropriate level of difficulty for each player, so no one should spend too long wading through puzzles too easy for them, or be presented with puzzles beyond their current ability level, but the adaptive engine too takes time to be able to do that well.) By the time I solved my 111th puzzle, my Creativity Score had climbed to 59%, and continued to go up as I played further. The majority of advanced puzzles require creativity to obtain good solutions.

In contrast, my Persistence Score remained at 0%. More precisely, I was not given a Persistence Score. Persistence measures the degree to which someone keeps trying until they finally get it right. If a player solves every puzzle optimally (i.e. a 3-stars solution) on the first attempt, as I was doing, the game has no way of knowing how they would respond when they fail to solve a puzzle the first time, or the second, etc., or when they manage to solve it but not optimally and have to come back to it and try again to get a better solution.

People who play our games because they simply like challenging puzzles could simply ignore the BrainQuake Score altogether and chase high star counts and bonus items. It would then be very much like many other (non-educational) puzzle games available. That’s actually how I play it when I play for fun, say on a long airplane ride. I know the underlying math, and worked hard with my BrainQuake colleagues to build a game around that math, so I don’t play it to learn that math. But I’m hardly a typical player of this particular game!

The other two components of the BrainQuake Score will be new to most people. They take advantage of, not just how video games are built and operate, but also the design of our particular game-based platform. Many (perhaps the majority) of digital math learning products require the user to solve a math problem and enter the answer. In contrast, to solve a BrainQuake puzzle, you have to carry out each individual step in the app. That makes a big difference.

Sure, you may think about your next move before you act, and you might grab some scratch paper and noodle on the problem for a while. But to actually solve the puzzle, you have to carry out all the key mathematical steps by manipulating the puzzle mechanism. (Take a look at some of the screenshots in previous blog posts, where I indicate the symbolic mathematics equivalents to puzzle mechanics.) This means that the game engine itself can follow every step of your solution. BrainQuake is the ultimate “show all your working” math-learning device.

That design feature makes it possible to measure things that normally require a trained human teacher to assess and evaluate, and in some instances can be evaluated only when the teacher sits alongside the student and observes their progress.

In the current version of our product, we have chosen to measure two additional aspects of learning on top of performance that teachers repeatedly told us they would love to know about: the degree to which a student is persistent, and the degree to which they exhibit mathematical creativity.

Al three of those factors, Performance, Persistence, and Creativity, are critical elements of mathematical problem solving. And that, as I keep repeating in these blog posts, is one of our major learning goals for our technology.

In future posts, I’ll provide more information about the Persistence and Creativity metrics. For now, I’ll leave you with the BrainQuake Score screen I got when I clicked on the Puzzle Summary tab at the top.

The screen I saw when I selected the “Puzzle Summary” tab on my BrainQuake Score page. I selected puzzle 99 to see more details. Image by Keith for BrainQuake.

Of course, we could have combined all three scores into a single number, to look like most other learning products. But as lifelong educators, we believe that a single number is not adequate to measure a person’s mathematical potential. Many educators agree with us on this, and one reason single-number assessments remain so common, despite the expressed desire for richer information, is that measuring anything but performance is difficult even for an experienced teacher, and for many technologies it’s impossible. Moreover, human assessment is time-consuming and expensive to do at scale. Research among teachers indicated that a triple of numbers, like the BrainQuake Score, would be more useful than a single percentage.

There’s actually more to the BrainQuake Score than I have shown you here. For one thing, it’s possible to drill down further. (We will be rolling that feature out in due course.) There are also other factors that could be measured. I’ll keep returning to this whole topic of assessment in future posts.

– Keith