CATA's Mobile Website Usability Evaluation
Since 1972, the Capitol Area Transportation Authority (CATA) is the regional public transit organization serving the Greater Lansing Area committed to providing safe, reliable, and accessible transportation solutions for the community. They serve over 5 million people in Lansing, East Lansing, Lansing Township, Dl High Township, Meridian Township, Dansville, Leslie, Mason, Onondaga, Stockbridge, Williamston, Weberville, and Delta Township in Michigan.
My Role
UX Researcher
Tools Used
Figma
Miro
Zoom
Type of Work
Heuristic evaluation
Persona creation
Usability testing
Data analysis
Timeline
Jan. - April 2024
The current foundation of their website launched in 2016 and uses Domain Name System (DNS) and DotNetNuke (DNN) as the Content Management System (CMS). However, as the organization has grown, their website is becoming more outdated, lacking clear information hierarchy as new pages are continually added to the site.
They are seeking to conduct a thorough audit of their website, focusing on accessibility and SEO issues. They have sought the help of the students enrolled in the Master of Science in User Experience program at MSU. Their goal is to enhance the overall usability on their Home page, Find Bus by Schedule page, and the Trip Planner page.
Quick Facts
73% of users accessing the Home page are on mobile devices, with 25% of all users visiting the Home page first.
01
CATA has over 24,000 riders on weekdays, with a yearly average of 11 million trips from 2013-2019.
02
Team Heuristic Evaluation
My team and I met via Microsoft Teams to consolidate our individual heuristic evaluations using Figma and FigJam, then voted on the most severe usability issues. We created a written report detailing our findings and recommendations.
We identified 4 high-severity issues:
-
Sign-up form lacks email validation, allowing incomplete emails without error. Recommendation: add validation to ensure valid email entry.
-
Homepage alerts are not prominently displayed, potentially going unnoticed. Recommendation: use a sticky header to notify users of alerts.
-
"Find Bus by Schedule" directions are unclear after selecting start/end times. Recommendation: provide clear next steps and error messages.
-
Error messages across the site do not adequately explain issues or guide users to resolution. Recommendation: improve error messaging to help users recover.
Individual Heuristic Analysis
I was partnered with 3 members of my cohort to conduct a Heuristic Evaluation using Nielson Norman's 10 Usability Heuristics of CATA's mobile website.
I did a simple walkthrough of the mobile website before conducting the evaluation. Since I was not familiar with the website, I wanted to familiarize myself with the interface and tasks. On the second walkthrough, I used the heuristics to look for problems. My task was to navigate the home page as a first-time user of public transit and look for issues with accessibility and interactivity.
Of the 10 heuristics, I found at least one error for 8 of them. The most glaring errors came from the "consistency and standards", "error prevention", and the "help users recognize, diagnose, and recover from errors" heuristics.
Research for User Profile and Personas
I collected data about the demographics of the population who ride the bus. I pulled data from three sources: CATA's website, DataUSA, and the U. S. Census Bureau. During my search of CATA's website, I found 6 articles and the 2023 Community Report which detailed the demographics of their passengers.
User Personas
FigJam Session
After the individual research and creating the user profiles, my team and I came together and grouped our findings on a FigJam board to search for commonalities to build the personas. During this process we discussed our initial findings and placed our data into FigJam on sticky notes. We then moved on to sorting our data to understand what findings more clearly could influence our personas. Our last step involved grouping our data to be able to see what important findings needed to be brought to live within our personas.
User Profile
Constraints
During the creation of the usability test plan, we were made aware of 3 constraints.
-
IRB Exempt Conditions
No targeted recruiting parameters - especially of vulnerable populations such as minors, people of certain income levels (especially low income), people with disabilities, aging, health status, etc. -
Technology Requirements
We will be using Zoom to host the remote usability test, thus all recruited participants must be capable of using Zoom and consent to being video recorded during the session. -
Convenience Sampling
We will recruit participants in 2 ways: through our personal network and MSU SONA participants
We included the following tasks in the usability test plan:
-
Purchasing a one-way pass (up until entering the payment information)
-
Viewing route schedules
-
Viewing alerts to make sure their route won't be impacted and to plan accordingly if it is
These tasks also would align with CATA's goal to enhance the overall usability on their Home page, Find Bus by Schedule page, and the Trip Planner page.
Noting the constraints of the class, we also removed the demographic questions (aside from the 18 years and older question) from the screening questionnaire, focusing more on the transportation habits of the participants and added success/fail criteria.
Table of contents.
Tasks.
Post-task questionnaire.
Usability Test Plan
Small Hurdle
One of our participants who represented a CATA rider was a no-show. Thus, we had to collect data from 3 participants who did not ride CATA and 1 participant who did ride CATA. We originally intended to get a perfect 50-50 split, but c'est la vie. Our data analysis and report reflects such changes.
We conducted our remote usability tests via Zoom. I conducted the test with a participant who is not a CATA rider. The session lasted approximately 35 minutes and the tasks that were evaluated were:
-
Buy a 10-ride adult card
-
Plan a trip to go from MSU Union to Brody Hall
-
Locate a bus stop (scrapped after participant 1 due to time constraints)
While the moderator was conducting the usability test, the observers were taking notes on the participant's notable quotes and if they completed the task successfully. Each member of my group had the chance to moderate a testing session and be an observer for the other three testing sessions.
Conducting the Usability Tests
Notable Quotes
"I’m looking at the options [for the bus passes] and I’m trying to figure out what the difference is.” [This is in references to the “up to 10 miles” and “over 10 miles” passes]
Participant 1, Task 1
“What’s the difference between a regular discount pass and a multi-ride regular pass?”
Participant 3, Task 1
“That last screen had a lot of information on it so I can see it being a bit confusing if I didn’t know exactly which one [pass] I was looking for”
Participant 4, Task 1
“I think I’ve had to be hitting everything [options] a couple times... Oh! It’s happening in the background. Is it?”
Participant 1, Task 2
“It looks like it’s trying to show me a map but I can’t see the map underneath it. That’s a little annoying”
Participant 3, Task 2
“The map is kinda hidden behind here and I’m not sure if it’s just my browser but if other people are viewing it from Firefox, it could be difficult... It makes me want to put it in desktop mode... and it’s still hidden behind here”
Participant 4, Task 2
Overall Findings and Recommendations
Task 1
Observations:
Participants struggled to find the 10-ride adult card and encountered unclear labeling and descriptions for the multiple bus pass options. They also had trouble accessing the cart.
Recommendations:
Provide clearer explanations of the different pass types, the intended user, and their features. Consider following industry best practices for cart functionality to make it easier for users to access, view, and edit.
Task 2
Observations:
Participants were confused about how to access the full map below the trip planner overlay. Participant 4 tried switching to desktop view on their mobile phone, but this did not meet their expectations as the map was still inaccessible.
Recommendations:
Add a clear way to close the trip planner overlay, such as an "X" button. Consider methods to automatically hide the overlay if the user doesn't manually do so. Provide a more guided experience where users actively select their preferences rather than defaulting to the map.
[Additional task 2 participant 4 screenshots]
Data Analysis: System Usability Scale
The System Usability Scale (SUS), developed by John Brooke, is an industry-standard for quickly and reliably measuring usability. It can be used with small sample sizes. The average SUS score is 68, as determined by Jeff Sauro's meta-analysis of 5000 scores from 500 studies. Scores above 68 are considered above average, while scores below 68 are below average. This baseline score of 68 can be used for benchmarking and measuring the effectiveness of usability improvements across iterative studies.
For the post-test questionnaire, participants rated their level of agreement with statements on a scale of 1–5 (Strongly Disagree to Strongly Agree).
The average SUS score for the CATA website is 46.25, which falls 21.75 points below average and [68] the cut-off score considered acceptable. However, this doesn’t necessarily guarantee high acceptability when the website is used in context by users on their own. Individual participant scores are shown in the figure above.
For the post-test questionnaire, participants rated their level of agreement with statements on a scale of 1–5 (Strongly Disagree to Strongly Agree).
The average SUS score for the CATA website is 46.25, which falls 21.75 points below average and [68] the cut-off score considered acceptable. However, this doesn’t necessarily guarantee high acceptability when the website is used in context by users on their own. Individual participant scores are shown in the figure above.
Data Analysis: Affinity Map
During each session, the note takers gathered all the participants' quotes, steps, and other notes into groups. After grouping them, we noticed that each group had a theme (ex: confidence and pain points within the journey) and named them accordingly.
During each session, the note takers gathered all the participants' quotes, steps, and other notes into groups. After grouping them, we noticed that each group had a theme (ex: confidence and pain points within the journey) and named them accordingly.
Final Report and Impact
We delivered and presented our findings to CATA through both a written report and a PowerPoint presentation.
For the PowerPoint presentation, I edited the videos into a compilation of the participants completing task 1 and task 2. During the presentation to CATA, I explained our findings for both tasks alongside the video and provided recommendations for how to mitigate some user frustration and errors.
As much as I would love to say that the students taking this class came in and saved the day by uncovering glaring issues, that just was not the case. However, we made an impact by providing up-to-date usability tests (their last tests were conducted in 2017) which can help guide future design decisions. It also provided a lasting impact on myself, as I learned how to perform a heuristic analysis, plan and conduct a usability test, and analyze and present the findings to stakeholders for a long-lasting impact.
User Persona 1
In our personas, we made noted that the core needs for the college student were affordable and accessible transportation, flexible routes to accommodate their busy schedule, access to reliable information about the bus schedules, and access to appropriate shelter for the elements. We also made this persona in-tune with current technology and trends, as she is a college student who knows how to navigate her way around most websites. That said, she is riding the bus because she has limited access to funds and cannot maintain her old vehicle passed down from her parents.
User Persona 2
In our retiree persona, we noted that mobility and accessibility were prominent issues to touch on. Bus stops being within a reasonable distance from popular sites, stability of the bus schedules, and the ability to obtain information via phone calls were some of the core needs of this persona. He rides the bus because his disability does not allow him to drive anymore. He does not use a smartphone very often, so getting the information he needs to navigate the city should be easy for him.