🗞 The error rate of ChatGPT for ‘UX Audits’

😌 Humans of User Research with Colton Perry

Tell us a little about yourself, Colton.
Hello! I’m a senior user researcher with NetEase Games primarily working in AAA studios in North America, Europe, and Japan. I got my research start in academia where I got my PhD in cognitive psychology where I focused on the interactions of cognitive processes and wearable technology. I started industry research back in 2015 with Xbox Research, and since then have split my time evenly between gaming and productivity domains at Xbox, Bungie, Microsoft Mixed Reality, and now NetEase. In my hobbyist life, I’d like to have more time for too many things, including photography, indie game development, beer brewing, tending my Japanese maples, and music (trumpet, piano).

What’s your favorite thing about your current job?
My favorite thing about my team in NetEase is that as part of the foundational team, my colleagues and I get to build a new world-class research program based on our perspectives and priorities. As an academic theory nerd, it was wonderful to find myself surrounded by other theory nerds that are interested in building process through ideation, experimentation, and application of foundational learnings we’ve developed across our careers. This also extends to our studio partners, who are often experienced professionals building new studios and IPs in their own vision–we’re all on the ground floor together.

What do you remember most about your first UXR job?
This is my space to gush about former colleagues and hopefully inspire others out there to help junior folks coming in have a good first impression of the industry. I started at Xbox Research in 2015 fresh out of grad school with basically no industry experience. They were were immediately welcoming and willing to help me learn, from the lab coordinators to my manager to my new hire cohort. And while my training involved being thrown straight into a usability study on a AAA franchise, I was lucky enough to have an incredible researcher to train me. I credit a lot of my career to the people that were generous and welcoming at the start, and my message to the field is to support and nurture your junior folks–it’s the only way the field will keep progressing.

How can people find you?
I am most easily found on LinkedIn, and while I can’t guarantee immediate responsiveness (job, family, aforementioned hobbies, etc.), I do try my best to get back to everyone that contacts me, and I’m happy to answer questions if I can be helpful. Some of my writing can also be found on Medium (Evolving Human-Centered Design). I hope to do more in that venue, time willing.

Thank you, Colton!

Spotlight > Articles > Video > Audio

📚 Articles of the Week.

Testing ChatGPT-4 for ‘UX Audits’ shows an 80% error rate & 14–26% discoverability rate
Testing ChatGPT-4 for 'UX Audits' reveals concerning results: it has an 80% false-positive error rate, a 20% accuracy rate in UX suggestions, and only discovered 26% of UX issues in webpage screenshots and 14% on live webpages, emphasizing the limitations of AI for comprehensive UX auditing when compared to human professionals. Christian Holst 10/18

How to start a UXR practice from scratch in 90 days
Setting up a UX research practice from scratch in 90 days is achievable with these key steps: understand your team's context and how UXR fits in, make the right connections early, systemize and automate processes, focus on short-term research with quick wins, and iterate to a flexible strategy, ensuring you can make an impact within the first two months of joining your team. Caitlin Sullivan 10/19

Getting started with AI for UX
Use generative-AI tools to support and enhance your UX skills — not to replace them. Start with small UX tasks, and watch out for hallucinations and bad advice. Jakob Nielsen 10/18

Re-evaluating usability: delivering value via research practice
Turning an ethnographic lens onto the friction between usability assessment and broader, macro research can shift our thinking about what research is tactical, and what is strategic. Joyce S. Lee 10/18

The hawthorne effect or observer bias in user research
Individuals often modify their behavior if they know they are being observed. That phenomenon became known as the Hawthorne effect or the observer bias. We can mitigate this effect by building rapport, designing natural tasks, and spending more time with study participants. Akhileshwar Nath Pandey 10/22

Strengthen your UX research with ethnographic skills
Ethnography skills can strengthen your UX research: demonstrate the impact to gain buy-in, adapt effectively through systems thinking, convey insights visually, embrace cognitive flexibility, bridge the "say-do" gap, and practice active listening to truly understand participants and guide research. Ben Wiedmaier 10/23

Are people who agree to think aloud different?
A significant portion of participants (48%) are willing to share their webcam and microphone for think-aloud studies, with some demographic variations like lower UK participation and an unexpected age difference, offering valuable insights for user experience researchers. Jeff Sauro & Jim Lewis 10/17

Identity-informed approaches to user research
Identity-informed research goes beyond understanding participant biases and delves into the profound impact of researchers' own identities and lenses on every aspect of the research journey, enhancing research quality and contributing to a more equitable research environment, with actionable steps like self-reflection, diversity embrace, and challenging assumptions for more inclusive outcomes. Dr. Kimberly McGee and Jenni Chan 10/10

Understanding the user: How the Enterprise System Usability Scale aligns with user reality
The article discusses the limitations of traditional usability metrics like SUS for enterprise products and introduces ESUS, a more user-focused and concise metric, to provide actionable insights for improving product usability and customer satisfaction in the enterprise context. Stephen Schneider & Serena Hillman 10/18

Spotlight > Articles > Video > Audio

🎥 Video of the Week.

Design research and UX in humanitarian and human rights technologies
UX design research in the global south on sensitive topics demands a culturally sensitive and inclusive approach, drawing from the experience of designers working with NGOs and open-source projects. YouTube 10/19

Spotlight > Articles > Video > Audio

🔉 Audio of the Week.

The dark side of quick wins
In this episode, the hosts explore the potential pitfalls of pursuing quick wins in UX research and stress the importance of mastering fundamental research skills for long-term success. Spotify 10/13

Happy Researching,
🗞 Jan