The CogTool Project

Tools for Cognitive Performance Modeling for Interactive Devices

 Questions?
Send email
  Overview   People   Publications   Connections   Download  

Cognitive Performance Modeling

Designers want to know how users will interact with their systems, but they often do not have the time or resources to test their designs with actual users. This is especially true for systems intended for trained users. It can be expensive to train users with a prototype, but unskilled users do not interact with a product in the same way as a skilled user. Cognitive performance modeling aims to solve this problem by predicting how a skilled user will interact with a system. Simulation and analysis take the place of expensive training and testing.

Predictive human performance modeling has one of the longest research histories in Human-Computer Interaction. Starting with Card, Moran, and Newell in the 1980s, the prediction of skilled performance time has enjoyed a constant stream of validation and expansion into many areas of user interaction with computers. Over one hundred research papers have been published about GOMS and the Keystroke-Level Model (KLM). Given its validity and predictive value, it is surprising that modeling has not become widespread as a tool for design in the user experience community. Our belief is that the cost of learning and constructing correct models, even ones as simple as the KLM, is perceived to be too high to justify the benefits of estimating skilled performance times.

We don't believe that this has to be true. With the support of tools, the time and training costs of performance modeling can be greatly reduced, and the accuracy of resulting models can be increased. Our project has produced a software application that takes the first steps toward these goals. We have already achieved great reductions in the time to produce new KLM-GOMS models. We have also shown an improvement in the accuracy of predictions compared to previously published examples of KLM.

Tools to Support Modeling

CogTool leverages the concept of a design storyboard to create accurate models of skilled performance behavior. Designers can simply draw onto an existing image of a device or software interface to create a single frame of the storyboard. Individual screens can be linked to form a complete storyboard for one or more tasks. Complex interfaces can be mocked up quickly using these simple techniques.

Editing a Design Storyboard in CogTool 1.0b1

Once the mockup has been created, a designer can demonstrate the steps of a particular task by directly interacting with the frames of the storyboard. As the demonstration proceeds, CogTool builds a model of the task, using heuristics to place "think" operators that approximate cognitive processing time.

Once the model has been generated, it is automatically translated into a KLM-like language called ACT-Simple. This language is executed via the ACT-R cognitive architecture to produce a performance prediction and a detailed trace of modeled behavior. ACT-R is a sophisticated system with a rich theoretical basis and years of use in the cognitive psychology research community and elsewhere. We have modified the perceptual and motor modules of the architecture to interface with CogTool storyboards, so that ACT-R manipulates the storyboard to complete the modeled task. This allows us to leverage the capabilities of ACT-R to produce more accurate predictions.

The main CogTool project window, showing predicted performance times for example tasks

Beyond the Desktop

The simulation window created as part of Dr. Dario Salvucci's ACT-R model of driving performance, including a simulated in-vehicle device panel.

Our tools have already been used to successfully model traditional desktop user interfaces and mobile interfaces for Palm OS devices. In order to further extend the range of uses for CogTool, we have joined with Dr. Dario Salvucci at Drexel University to make it easy to model driver performance with in-vehicle information systems. Models of task performance created with CogTool can now be integrated with his sophisticated ACT-R model of driving performance.

With a few simple changes to the task demonstration, performance of the generated task models can be interleaved with driving performance in a straightforward way. The resulting combined model interacts with a simulated environment to produce quantitative predictions of safety-related measures such as lateral deviation from the center of the road and time to perform the information task.

A poster about our driving models can be found at the Publications tab above. These efforts are part of a broader investigation into the use of in-vehicle devices and their effects on driving performance.

© 2005, 2006 Carnegie Mellon University
All Rights Reserved.

Last Modified: Mon, 16-Apr-2006