Can a Child’s Creativity and Persistence be Assessed by a Game?

Educators have long known academic standards are only one part of nurturing a well-rounded and successful student. There are a host of other skills like creativity, persistence, critical thinking, collaboration and empathy that help make a student successful in school and in life, but are less quantifiable. Current assessment systems aren't set up to measure these very important but less measurable skills, so policymakers have focused on standardized tests that try to capture what a student knows, not how he or she can apply that information. Game-based learning has entered the assessment field with hopes of measuring both content-specific knowledge and softer skills at the same time, through the mechanics of the game.

“I believe that assessment should be open and transparent and flexible and gently surround and support student learning or groups of students learning together,” said Valerie Shute, a professor of Educational Psychology and Learning Systems at Florida State University in an edWeb webinar. She’s been experimenting with digital games as a way to more firmly link learning and assessment, hoping to show the two can happen inseparably. She’s particularly interested in how to measure improvement in the less tangible areas that are hard to measure, like creativity and persistence.

"One of the big challenges standing in the way of schools embracing some of these new skills is there is a shortage of valid ways of testing them," Shute said. To rethink assessment, Shute, among others, is trying to meld elements of good game design with specific tasks that require students to demonstrate competency to see whether games can be a valid way of formatively assessing students, especially on the hard-to-measure skills.

“Games require a player to apply all sorts of competencies in the process of playing in order to succeed,” Shute said. The best ones work as a system, requiring interactive problem-solving between the player and the game. Good games also have specific goals and rules to keep a student focused and have adaptable challenges to keep the player at a difficulty level tailored to the individual’s outer edge of “do-ability.” Kids enjoy challenging games, as long as they aren’t too frustrated. Games also offer students some control over gameplay, ongoing feedback and sensory stimuli in the form of graphics, sounds or narrative.

Those qualities mirror good learning experiences in any form. The key is to create a game that teaches the concepts students need to learn without messing up the unique engagement that good games provide. That’s where what Shute calls “stealth assessment” comes in -- it’s seamless and ubiquitous, providing important feedback to the student and creating a model of the learner that can help teachers tap the individual needs of each student.

DESIGNING A LEARNING GAME

“It starts by articulating a competency model,” Shute said. “This is what you want to say about the student at the end of the day in terms of knowledge and skills and dispositions.” Shute and her team designed a game to teach physics, creativity and conscientousness using a method called Evidence Based Design. They figured out what evidence would prove kids were learning in those three areas, designed tasks to evoke that evidence and compared the results to other evaluation methods that have already been validated.

For example, the team broke creativity down into a cognitive category and a dispositional one. In the cognitive category there were elements like divergent thinking, flexibility and originality. “Each one of those facets had a number of different indicators which were what a person did in the game that would provide evidence towards those particular facets,” Shute said. One way the game examines creativity is with a challenge for students to design a solvable level on their own for a friend.

For conscientiousness, researchers were looking for evidence of persistence, perfectionism, organization and carefulness. Persistence was measured by the time a student spent on a difficult problem and how many times she restarted the problem to try it again. Shute contends the game provides more accurate information about qualities like creativity and persistence than the common self-reported survey method, where it’s easy for students to say they persist through challenges, but there’s no proof that they actually do.

PHYSICS PLAYGROUND

Shute and her colleagues designed Physics Playground to teach and assess two-dimensional physics simulations dealing Newton’s Laws of Motion, things like gravity, mass, momentum and energy. Embedded in the game are assessments for creativity, conscientousness and physics understanding. Shute has conducted five studies on how well the game works and has gotten positive results each time.

The game has very simple objective: get the green ball to hit the red balloon. The ball and balloon are placed in various locations on the screen, sometimes with obstacles in-between, and students have to draw “agents of motion” like levers, springboards, pendulums or ramps to get the ball to hit the red balloon. To make it more difficult and to receive more points, students can solve the problems using fewer agents to accomplish a task. Other time difficulty is added if a level looks completely different than ones that came before.

“When you start the game the computer doesn’t know whether you are at a high, medium or low level, but once you start accumulating evidence your level starts shifting around,” Shute said. If a student solves a problem with the pendulum and then with a lever, the computer keeps track and builds a model of the learner. It’s tracking how long it takes the student to complete the problem, how many times they attempted it, what physics concept they used and whether they were able to complete it well enough to get a gold or silver trophy.

“There’s no one right answer to these problems,” Shute said, “you can solve it with a number of different agents and do well or poorly and do it in a long time or short time, but all of that is evidence that’s being accumulated.”

 

DO STEALTH ASSESSMENTS WORK?

To measure the effectiveness of Physics Playground, Shute's team compared student metrics on creativity, conscientousness and physics concepts against traditionally accepted and validated measures of those same qualities. For creativity they used self-reported surveys and Wallach and Kogan’s alternate use test (how many ways can you use a shoe, for example) to measure both fluency and originality. For conscientousness, they again used self-reporting, but also looked at the Eisenberger’s anagram task to see how long students would stay on a tough problem. For physics, they used multiple choice tests.

Shute found the correlations between the stealth assessment and the externally validated tests were all significant, consistent and valid. “If we are making a judgment about how well or poorly a person is doing it actually has some basis in reality,” Shute said. Perhaps more significantly, students participating in a test of 166 7th and 8th graders markedly improved between a pre-test and a post-test after only four hours of play and no direct instruction. And, both girls and boys found the game engaging, which is not the case for every video game.

Shute's research has her convinced that there are strong reasons to believe games can make great formative assessment tools in the classroom. She’s hoping to add features to Physics Playground like explicit learning supports connected to certain topics. Kids would still formulate hypotheses, test them out and learn through the trial-and-error process, but afterwards a pop-up would make an explicit connection to the Newtonian law they intuitively grasped already.

Shute and colleagues are also running a study on the affective state of game-players as captured on a webcam while playing. They’d like to see how various facial expressions like frustration, delight and boredom correlate to what’s going on in game play.