Macmillan Learning
Calcpad
Macmillan Learning's assessment tool allows students of varying education levels to interact with their coursework online. This can be problematic for math disciplines because commonly used math glyphs such as π, ∞, etc. aren't easily accessible from a computer keyboard and searching outside of the assessment environment can negatively impact learning. A dynamic "Calcpad" was designed to display altering button configurations tailored to the type of question the student is answering.
TEAM
Jenn Feralli
+ Math Assessments, Product Director
Kaitlin Tasker
+ UX Researcher
Jasmine Winata
+ UX Recruiter
Steve Rutherford
+ Front-end Development
Becca Runyon
+ Learning Science
ROLE
+ UX/UI Design
THE PREVIOUS KEYPAD
BUSINESS PROBLEMS
Customer Experience Specialists were receiving calls from frustrated students who could not complete their assessment because the Calcpad was either covering their answer fields or being covered by their touch device's native keyboard.
PROBLEM VALIDATION
The issue, in Cynefin framework terms, was simple. In order to simplify the Calcpad, we needed to understand how buttons were utilized and what areas still needed improvement.
A/B MODERATED TESTING
Uncovering key shortcuts
A quick, low-fidelity prototype was made to test the usability of the 2-row Calcpad instead of the previous 4-row version. We tested both the previous pad and the new iteration with our participants, Pre-calc and Calculus instructors. Testing with professionals before our real users, students, eliminated any comprehension bias.
THAT AHA MOMENT
While moderating I noticed a few instructors using key shortcuts, like "pi" or "sqrt" to render math characters in the answer input and their time on task was swift. Our tool was leveraging Mathjax, a javascript display engine for math glyphs, and it came with canned commands! At that time our Calcpad repeated many buttons that exist on a standard computer and 3 of our 6 participants mentioned this was unnecessary and confusing.
KEYSTROKE LEVEL MODELING
Validating removing repeating keys
I needed to validate that typing on a keyboard is faster for our student users. More testing could've been conducted, but to save time and resources, I used a form of efffiency tracking named KLM (Keystroke Level Modeling). The basic concept of KLM is observing the micro-actions that go into a task (homing, clicking, pointing, cognitive load) and adding those up by their defined millisecond values.
REVISITING THE INSTRUCTOR SESSIONS
By revisiting the instructor recordings I could tally up each micro-action for each task and then find the sum. Some of the instructors only used the Calcpad while others used a combination of both. By observing these two styles comparatively it was clear that those using their keyboards were more efficient. Although this data isn't statistically significant, it was enough to convince our stakeholders that repeating buttons on our Calcpad was doing more harm than good.
VALIDITY ASSURANCE
When it came to removing repetitive Calcpad and keyboard buttons I realized that almost an entire Calctab could be typed on the keyboard. It wasn't clear why this tab was created in the first iteration and more investigation was needed. Our Learning Science teammembers (who worked on the first iteration) helped our team understand that the Calcpad was not only used for utility but also for a learning aid. For example, trigonemetric functions could be typed on a keyboard but were added to help pre-calculus students learn terminology.
SOLUTION VARIANTS
Touch Devices
Mobile users rely on their native keyboards while using their devices which is contradictory to desktop users reliability on their keyboards. The circumstances are very different; buttons removed for the desktop experience needed re-introduced for mobile. In fact, retro-fitting the original Calcpad to mobile made sense because there were no longer buttons to repeat!
WIZARD OF OZ TEST
In-person moderated study
The Calcpad for desktop was a quick project for our dev partners since most of the logic had already been. To leverage some of this logic, our UX team conducted a Wizard of Oz test where the student used an invision prototype for the tablet Calcpad but their answer field was operated by a wizard behind the scenes. You can learn more about this testing method in my co-written article here.