UX CASE STUDY
PA Human Relations Research & IA
CHARM
CASE STUDY HIGHLIGHTS
-
ORG & SECTOR: PAI NIC (Division of Tyler Technologies), in support of PHRC (PA Human Relations Commission)
MY ROLES: Director of Experience Design, focused on UX Strategy & Research, Workshop Facilitation, Content Creation & Information Architecture.
TEAM: Citizen Advocate (UX), Content Strategist (UX), Project Manager (PAI), Project Coordinator (PAI), Director of Operations (internal project support, PAI), Business Sponsor (PHRC) and me.
-
Due to the incredibly sensitive nature of the individuals seeking help from PHRC, along with their personal situations, we conducted interviews with the advisors of those individuals.
Which, IMO, softened the actual pain, frustration, and desperation felt during the experiences in seeking some kind of resolution and relief from PHRC.
Everything else went very well! Our team worked amazing together, with no need for excessive heroics to carry the project work.
-
With my UX team, and with internal support of a fellow director, we rallied around the best UX tools to use throughout the entire project effort.
The UX tools and exercises employed during discovery included the following: User Personas (including the roles the Business Sponsor juggled as a team of one), Empathy Mapping, Customer Journey Mapping, Experience Blueprinting (short-term — as a rough predecessor to a longer-term Service Blueprint.)
We handled Quantitative User Research from a collected data repository of user feedback.
We utilized Optimal Workshop for various card-sorting and tree-jacking user research activities.
My UX team and I synthesized the OW results, formulating any additional activities, IA, UI, and navigational constructs to consider.
-
We helped our PHRC business sponsor to put into implementation a revised navigation, a reorganization of existing content, and the start of any clarified new content.
Our PHRC business sponsor was thrilled with the level of insights we’d gained and shared through our user research efforts!
-
SalesForce and SharePoint reports and analytics.
ExpressForms feedback form repository.
User Studies through Contextual Interviews of suitable & available user study participants.
Optimal Workshop OptimalSort card-sorting activity.
Optimal Workshop Treejack Study.
UI Element activity tracking & measures for continual usage analytics.
KEY TAKEAWAY
For a shining success, let each team member play to their strengths.
Start with ‘Who are the people who need what you provide?’
One of the best ways to help the entire project team to be open to and aware of the experiences of their fellow humans is to do an Empathy Mapping Exercise.
To prompt a deeper level of empathy and temporarily embody being in the shoes of someone in need of help, I ask the project group questions such as, ‘Why might our fellow PA Residents be coming to this website? What might be going on? If you feel someone violated your rights, and you need X — clarification that it was indeed not right, advice from someone who has seen this before, options to consider, an understanding of the process, how long it might take to resolve, what the inquiries might look like, etc. — describe what you might be going through, what you might think and feel, what you might see, what you might say and do, and things you may hear.
When the project group consists of business stakeholders who have worked with others in these dire situations every day, this exercise helps them connect again to the human need. The team members involved benefit from this exercise because the wheels start turning.
Define the unique characteristics of the pivotal users - customers and employees who provide the services to customers.
Because an organization’s services are not served to customers through means of wizardry or magic, it’s just as essential to fully understand the roles and characteristics of the employees who play a pivotal role.
Such considerations model Service Design thinking and doing.
Here are two personas that needed to be captured for team referral. Ideally, the PA Resident in need persona should be further refined — but this was enough to get started.
Journey-map the entire customer experience for the focused scenarios represented in your potential project effort.
Using a story format — with a narrative arc — set the stage for establishing the trigger, all possible steps, frequent obstacles, the “user goal” of the scenario, and what follows. This journey is usually not all of the possible touchpoints in the life of the customer and their interactions with the organization — just a particular journey to satisfy/complete a specific user goal. This exercise helps to establish a basic sequence of steps and timing of customer needs and actions.
There will always be more digging beyond this point. Sometimes breakouts of even smaller stories or movement to a Service Blueprint.
It’s important to note that at this point we are not yet constraining ourselves with a project scope.
“There is so much fear, reluctance, hesitancy around filing. It’s much more about giving reassurance, staying by their side, making it easy for them, having them feel like somebody’s walking alongside them through the process.”
— Marcia B., Community Advocate, Carlisle, PA
Establish a framework that connects actions to interactions.
After customer journey mapping it is tempting to bridge to a Service Blueprint as a way to represent the various human and computer roles in motion during steps of a scenario. Thus far, I’ve found that a Service Blueprint helps most when there are multiple direct and supporting employee roles, with multiple complex system processes. In this case, we stayed in mapping any differences between the different types of PA Residents in need, supported by different laws (and thus different potential violations of such laws and acts) in Pennsylvania.
Our small project team stayed open to the unknowns. As seen in the second artifact, there were many additional areas we could have continued to explore — given enough time and access. To keep us focused on having a realistic scope to target, we focused on the information we needed to better understand and could gain through user research.
In planning user research, define overall goals, tactics, and timing.
The purpose of our research was to listen to some of the users who have a considerable investment in how PHRC, in collaboration with other local community support groups, helps those in need – all those who feel their human rights have been infringed upon.
Our business sponsor shared that the primary group we should focus on helping first are the PA Residents in Need; however, personal interviews and observations were deemed too sensitive, especially if we were to observe their completion of the highly sensitive complaint form. To remain sensitive to the private affairs of our PA Residents in Need, but to properly support them, we established that key support group members, including those who work as Rights Advocates, who work closest to those PA Residents in Need are where we’d research first.
Working with the project manager, a highly skilled and (thankfully) patient woman, we established a suitable timeframe to handle adequate user research — both qualitative and quantitative. This delight to finally roll up our sleeves and do more robust research was only possible because our company’s Director of Operations truly understood the value of applied user experience.
“I would like to submit a complaint. I have completed the complaint form, but I’m not sure where to send it. Is there a fax number or email that receives complaints?”
— A.M., PA Resident, Emmaus, PA
How have customers been providing feedback to the org?
In most project efforts undertaken, there often exists channels in which customers have already been sharing feedback with an organization — willingly or out of frustration. As my Citizen Advocate would often say, “You can’t beat free feedback!” These channels are often phone logs, help desk tickets, feedback from surveys, and form submissions. Although these shouldn’t be the only sources to consider, they can fill in many of the gaps — and will usually point to “problem areas” — at least enough to know where to start digging further with other research methods.
Lucky for us, we had a mother load within our grasp, courtesy of our company providing services to this state governmental agency in years leading up to the current endeavor. The quantitative research we’d done came from the large repository of information from the Express Forms Feedback form on the PHRC website. We’d gathered a collection of feedback data from January 2019 through mid-May 2021, composed of 2300 submissions. For this research, we took a random sampling of 400 of those 2300 feedback submissions, categorized as seen in the screen capture.
Although we knew we couldn’t give any relief to the 60% of folks who were requesting specialized help, we could still hope to positively affect almost 40% of people with not-so-great experiences who felt they had no other option than submit a form in the hopes of someone reading and replying. And, really, when you consider that only the MOST frustrated/desperate people with a bit of time on their hands reach out through an electronic form, the likelihood of an even higher percentage of headaches we could hope to prevent is pretty dang high.
If you could start with just a few interviewees, who would they be?
To confirm any existing assumptions of problems, we planned contextual interview sessions with scripts and tasks to observe in the current PHRC site and any other websites the users noted as “go-to sites.” All sessions were planned to run real-time through Zoom, with full audio and video, and recorded.
Our original expectation was to user-test with up to five key individuals of different demographics, within the PA Educational Support Groups. Prior to the actual sessions, each participant answered a preliminary questionnaire to establish demographics and identity preference. The three individuals who confirmed participation for our interviews identified within the same demographics, as follows:
Age Range: 51+
Gender, as self-identified: Female
Familiarity w/navigating websites: Fully comfortable
In hindsight, we should have added 2 more questions around “Describe your role to us in one to two sentences” and “Describe how you see your work aligning with the business sponsor’s at PHRC.” This would have helped us screen out any participants that may not be as close to the PA Residents in Need as we’d envisioned or other scenarios that are challenging due to their proximity and connection to digital touchpoints we can reference.
“I was trying to file a complaint with the online form. However, it came as a pdf and I was not able to properly fill it out in that format.”
— T.B., PA Resident, Philadelphia, PA
Consolidate highlights & recommendations to share with others.
Many of the screen captures I’ve included within this case study are taken from the User Research Report I created from our various research methods for this project effort. Each user study is a two or three page layout of organized findings, noteworthy quotes, and website screens the study participant shared with us.
During study #3, I simulated an empathy scenario to get more to the heart of common reactions.
Following the highlights of each study, I included a full page listing of Suggested Experience Improvements, consisting of content recommendations and any additional site/feature functionality that might improve the overall UX.
User test with card-sorting activities to clarify IA & navigation.
The site’s information architecture and navigational constructs changed considerably from before we started our robust user research to after we’d gone through multiple iterations in applying the team’s organizational consensus and in clear results from multiple activities we curated and ran.
Within our UX team’s sandbox, Figma, and shared with our business sponsor and stakeholders, we kept a growing compendium of the entire website. This served to provide the necessary visualizations of not only site pages organized in different ways, but also to communicate our recommendations of missing components, existing features that needed reconfigured, and content that needed reorganized, simplified, and/or clarified.
These visualizations also served to point out any additional areas we should clarify in follow-up questions during user research activities — to ensure we were recommending the best solutions without the overt pressure of our own opinions.
Using Optimal Workshop as our user testing framework, we ran the card sorting activities at the same time as the treejack tasks (seen in the email snippet). This required careful strategy and extensive planning around curating the most insightful feedback from users (without conflicting or duplicate questioning).
The strategy involved also figuring out who would be involved in which activities, with reliance upon users familiar with PHRC, versus users completely unfamiliar with PHRC’s offerings — lending unbiased results as much as possible.
“It’s assuming you can print this. I don’t have a printer… I probably wouldn’t go to Staples. I’ll tell you that!”
— Rachel A., Activist, Unity Coalition of Southern Alleghenies, PA
Use Treejack Studies to identify directness, success and first clicks.
To ensure we would have enough numbers of users to make an impact on decisive actions we’d recommend to PHRC’s Business sponsor, we utilized Optimal Workshop’s ability to recruit additional study participants, at an additional fee. With just a few demographic questions, we were able to establish a little understanding over the “unknowns” of this additional user group.
Among the many insights Optimal Workshop provides, is a Directness Score — the paths that people chose to get to the “right” answers without backtracking. Our directness score across all Treejack Study tasks was 83%.
Other OW insights include success metrics measuring direct success all the way to indirect failure, including any skips. We set up a clear success metric for all tasks except for task #2 — where there was one correct answer. Looking at the second picture, task #2 purposefully doesn’t have any green, which will be clearer in a moment.
Another unexpected but crucial insight is the information provided on the first clicks of the user during each task. First clicks tell a lot about one’s gut instinct and whether we have information grouped correctly into the highest bucket. Seen in the third screen, tasks 1, 3 and 4 resulted in high percentages of first clicks that took the users to the “right place”. Task 2 doesn’t have a correct answer defined but notice the 3 areas that are almost equally balanced in where users would go to start their exploration.
This was the gold we were seeking.
Where are users going? Can they find what they need?
Treejack Study, Task One: Starting from the home page, where would you go to file a complaint?
We defined the correct answer as going into the Filing a Complaint bucket, and then ideally choosing one of the forms from “Complaint Forms”. Because we purposely did not show the actual complaint forms nor the types of complaints (which live in the 4th level) we may have missed sharing the exact context to others to make an even better decision; however, for purposes of this study, we best defined success as making it into the Filing a Complaint bucket.
With that in mind, we had a much higher success rate than the study showed (with a direct success of 31%, and a directness score of 90%).
39/51 = 76% success
Of those who didn’t get it “right,” 75% of them would technically reach others who could help them out for something very specific or to consider Mediation.
Where would users expect to find valuable help?
Treejack Study, Task Two: If you had been subjected to discrimination and needed some basic information or answers to common questions, where would you start?
We purposely did NOT set a correct answer for this, as we wanted to identify patterns of expected areas where people would turn to educate themselves. The results were split almost evenly into 3 main areas: Legal Resources (36%), Education and Outreach (30%), and Filing a Complaint (26%).
What have we learned? Most people start with the Law and an understanding of violations of the law. Then they educate themselves further. Then they look to take action.
Biggest surprise? Very few people are looking to actually talk to a human right away. They’re looking to self-serve.
“My township has a newly established Human Relations commission. They feel they would benefit from training. What options are available for us?”
— S.S., PA Resident and Township Supervisor, Chester Springs, PA
Where would users expect to find types of training available?
Treejack Study, Task Three: If a community organization to which you belonged was interested in receiving educational materials or training by PHRC, where would you go to see what types of training were available?
24/50 were directly successful. 12/50 people were indirectly successful (although the tool defined such ones as “failing”) because they found the correct bigger bucket (of Education and Outreach). 36/50 people found the right biggest bucket. 72% success in finding Education and Outreach.
What have we learned? People tended to want to find helpful information in the Mediation Services area, for perhaps specialized training in how to resolve. Does it somehow look like Mediation Services might be the group that provides the training? It’s possible.
Recommendations I provided: Consider content that clarifies if Mediations Services handles training of its own (or consider it an area to highlight training around “Ways to Mediate” and that many people are looking for this.)
Where would users expect to find contact information?
Treejack Study, Task Four: Where would you go to find contact info for PHRC?
30/51 were directly successful (61%) with another 6/51 choosing the right bigger bucket if About PHRC points to ways to contact them.
36/51 = 71% chose the correct biggest bucket. Five of the remaining 15 people who didn’t choose the correct answer had valid and specific contact information (for specific people and services) = 33% of the remainder still had a good concept of where to find contact information.
41/51 found a logical place for contact information (generic or specific) = 80%
What have we learned? People get it! But they’re also expecting to see contact information at specialized levels as well — with Mediation Services, with Advisory Councils, and with About PHRC itself.
Recommendations I provided: Consider a Contact Us preliminary form that allows the user to specify who they need help from, how they can be reached, and what time works the best for them.