Blog

We rely on UX research, collecting data from our users, to inform our UX process. As the Nielsen Norman Group aptly states “UX without user research is not UX”. That doesn’t mean that all UX teams conduct research the same way, or have specific roles dedicated to UX research. This means everyone on a UX team has the potential to play a role in collecting and analyzing data. For example, designers might need to conduct their own usability testing. PMs and developers might assist with analyzing interview data to identify themes.

We benefit from greater involvement of our team members and other organizational stakeholders in the research process. They increase empathy and understanding of what users truly experience and need out of a product when they experience research first hand. However, as we push to democratize UX research in organizations, we need to keep in mind potential perils associated with poorly planned and conducted research. I don’t mean this only to the untrained team members we advocate taking part in research, I’m also talking about the need for those with UX researcher in their title to have a level of understanding of how to identify and avoid research that might take advantage of vulnerable populations, research that causes harm or is misleading, and poorly done research with no potential use for the outcomes.

We can have a false sense UX research is purely benign. After all, aren’t we collecting data to fight on behalf of our users and their experience? While usability testing seems harmless on the surface and interviews are only words, we need to be intentional in knowing why we are conducting research, the potential negative effects our research might have on participants, and what we can do to mitigate the potential for conducting unethical research.

We are asking a lot of our team to include research along with their many other duties. That’s why it is important to best prepare everyone on the team for the potential to encounter unexpected situations that might cross over ethical lines. Researchers by training and trade have often been required to take courses and pass exams to reflect an awareness of potential ethical issues in research. We can best prepare our colleagues to avoid these situations through similar training and standards.

Academia identified the need decades ago for researchers to have training and submit their research protocols to an Institutional Review Board (IRB). This didn’t happen accidentally — it was after decades or centuries of examples of improperly carried-out research. Medical and psychological research across the globe has a history of unethical research which led to withholding potential treatment or negative outcomes on the mental health of participants. UX research, particularly in health fields, stands to recreate some of these harmful scenarios, even if inadvertently.

What Are The Problems And Potentials With UX Research Done Poorly

You might wonder how your research could potentially cause harm or be unethical. Perhaps you are just conducting interviews in exploration of a concept you have for a new banking application. Or you are testing updated designs for a new workflow you’ve created allowing people to submit forms to your organization; things that seem harmless on the surface.

Defining Ethics

Dictionary.com provides one definition of ethics as “the rules of conduct recognized in respect to a particular class of human actions or a particular group, culture, etc.” I am using the word ethics as it applies to a class of human actions (UX Research) and specifically, I am defining the rules of conduct we need to consider in UX research as:

• UX Research should be respectful and compassionate to study participants.
• UX Research should respect our stakeholders and colleagues, and the resources and trust they give our work.
• UX Research should respect the norms associated with social science and research using human subjects, including all studies using methods that have defined protocols providing informed consent.

Any research involving human participants has the potential to cross over ethical boundaries if done poorly. I’ve identified six common issues we need to watch out for regardless of the topic or focus of our research:

I’ll cover each of these issues in detail and then offer some potential solutions UX practitioners should advocate across all organizations.

Vulnerable Populations

Researchers studying human subjects have a higher bar to pass in justifying research with populations often labeled vulnerable. You can consider any population that cannot make their own decisions vulnerable (and likely off-limits) if you are not a highly trained and experienced researcher.

The Children’s Hospital of Philadelphia Institutional Review Board provides this definition of vulnerable populations: “those who are ill (dependent on clinician for care), ethnic or racial minorities, non-English speakers, children, the economically disadvantaged, adults with diminished capacity.”

Additionally, they offer the following “Special justification is required for inviting vulnerable individuals to serve as research subjects and, if they are selected, the means of protecting their rights and welfare must be strictly applied.” While this guidance is being directed towards potential medical studies, there is no reason UX research should hold itself to lower standards when it comes to working with potentially vulnerable populations (in English speaking countries).

You should not be recruiting children, those experiencing illness, the homeless, or prisoners without seeking guidance and having appropriate conditions met. I’ll explore this more in the solutions section.

Misleading Users

Researchers in psychological fields are familiar with the concept of deception. These researchers need to justify any attempt to deceive or distract research participants from the true purpose of the research or that they are participating in research that is potentially meant to examine. Think of the often referenced Milgram’s shock experiment in which participants were instructed to administer what they thought were real electric shocks to another human. The person receiving the shocks wasn’t, in real life, being shocked, however, this was unknown to the participants administering the shock until after they completed the experiment. Participants experienced visibly stress during the experiment, reflecting the likelihood the situation was causing them mental duress as well. Milgram felt the need to deceive research participants was essential to collecting valid data. He also didn’t clear his protocol with any ethical review board. Milgram’s experiment would hopefully encounter much more rigorous review, and be required to adopt projections for participants, had it occurred today.

While we are likely not attempting to have our users shock another human, or cause any obvious harm to another using our design, we do need to give some thought into how we are conducting our UX research. Critical questions we need to answer and be comfortable we are not being deceptive include:

• What role am I asking the user to play in this study and why?
• Am I able to adequately explain the purpose of the research in a way that justifies the need?
• What expectations am I creating in the participant through their participation in the research?
• What, if any, mental harm or stress could come from someone participating in this study?
  • If there is the potential for harm or stress, how will I deal with this?
• How honest and open will I be with participants?
  • Will I share the timeline for creating the product? If no, why?
  • Will I share the company that is sponsoring the research? If no, why?
  • Am I creating false expectations exposing design concepts or ideas that might never see the light of day?
• Have I accounted for diversity and how diverse populations might respond to the research?
• Is there a logical connection between the research participants and the topic of the research?
• Will I compensate participants upon completion of the study?
  • How might this compensation influence both participation and responses during the research.
• How will I respond to a participants’ request to end the research? There is a correct answer to this one — you will stop immediately.

You need to answer these questions and feel comfortable you are taking due diligence in reducing/removing deception or misleading information in your research. I recommend answering these questions with your full product team and documenting the responses prior to undertaking your study.

Inadvertent Sensitive Topics/Extreme Experiences

Those engaged in research need to maintain an awareness that even seemingly benign topics can take a quick turn into sensitive areas, risking confrontation or emotional encounters during an interview. I could share dozens or perhaps hundreds of experiences where I walked into a conversation with a research participant who already had an axe to grind with the company I was researching on behalf of.

You need to be prepared for defusing these situations. It is impossible to define all the potential sensitive topics that exist, what’s more important is for a trained researcher to understand how to respond when a situation becomes intense. For example, you are researching on behalf of an electric power generation company and you speak with someone who states they just had their electricity shut off and they were glad to finally get ahold of someone they could ask to resolve an issue with their billing statements. How will you respond in a way that will make things better, not worse, knowing it is outside of your power to do anything about this participant’s electricity bill?

Other topics are naturally sensitive. For example, if you are conducting research for a digital voters education platform, you can expect people will come to your interview prepared to share their political beliefs. How will you respond?

You need to be prepared for how you will deal with emotional responses. This includes letting people have emotional outbursts and giving them time to recover. This includes you (the researcher) needing to reflect proper empathy. I’m not suggesting you need to expose yourself to situations you aren’t comfortable with, I’m stating the opposite — don’t conduct the research if you cannot handle situations involving human emotions and unpredictable responses.

We have an opportunity to involve our colleagues with less experienced research in observing us when we do conduct research that might broach sensitive topics. We can model response (or non-response) to emotional or provocative reactions from research participants. We can also role-play with our colleagues prior to engaging in research to practice how we might respond to unexpected reactions.

False Expectations

Research participants often have expectations about the purpose of the research, who will be conducting the research, and how their information might be used. We need to be prepared to address inaccurate expectations and redirect the conversation in a sensitive way that will still allow us to have a rapport with the participant and gain valuable insights. These scenarios are similar to inadvertent sensitive topics in that it is difficult to account for all of the potential scenarios you might encounter. One example of a scenario I’ve frequently encountered is a participant shows up expecting to discuss a specific experience they’ve recently had with my client, while I’m looking to cover a broader topic, such as their overall attitudes and behaviors towards products in an entire industry or set of services.

You might not be the one directly recruiting people for your study. Your ability to account for participant expectations prior to research is reduced if you don’t have direct access to recruiting the participants. You will need to provide the person/organization recruiting your participants’ detailed guidelines for who you are trying to reach, and the importance of getting a diverse, non-convenience sample. You need to provide them with a script that lays out the details of what participants will need to know prior to agreeing to engage in the study. I often draft emails explaining the purpose and details of a study for clients to use when communicating with potential research participants on my behalf.

You are likely to encounter bumps even when you set the expectations clearly to research participants or those recruiting on your behalf. I often find, no matter how clear I am in my recruiting instructions, people show up to one on one interviews expecting it will be a group interview. You can account for and reset expectations in the opening of your study’s protocol. I inform participants of the purpose of the study and give them a general outline of what we will review, along with the format. For example:

Today I’d like to spend the next 30 minutes speaking with you about your experience with XYZ digital product. I want to learn from you, so I’ll ask some specific questions about your experience and spend most of the time listening. I’d also like your feedback on some updated designs for XYZ digital product. After my initial questions, I’ll start sharing my screen and control of my mouse with you so you can show me how you’d use this design. Does that align with what you thought we’d be doing today? [address any concerns or comments]. Do you have any questions for me based on what I’ve shared so far? [address any concerns or questions]

No Idea How To Use The Findings/Misinterpreted Findings

UX practitioners engaged in research should understand the overall questions they are trying to answer (purpose of the research), how they will answer this (methods), the type or types of data the methods they will use will generate, and how to convert this data into findings and recommendations (analysis). This potential issue reflects the rule of conduct I noted above that UX Research should respect our stakeholders and colleagues, and the resources and trust they give to our work. We are wasting everyone’s time if we are simply asking questions because we can, or because we think the insights might be interesting. We respect our research participants and our colleagues when we create studies that are purposeful and focused.

You need to answer the following questions to ensure that you’re able to collect the correct data, and interpret your findings appropriately:

• What are the question(s) your study is trying to answer?
  • If relevant, what hypotheses do you have about the answer to these questions?
• What specific questions will you ask on your interview, survey, usability test, etc?
  • How does each of these questions tie back to answering the overall questions and hypotheses?
• What type(s) of data will your study generate?
• How will you analyze this data?
• How will your findings and recommendations be used?

As a researcher, I want research to flourish and grow in all organizations. However, I don’t want to see organizations conducting research without a purpose. We already fight survey fatigue, screen fatigue, and millions of other opportunities vying for our potential research participants’ time. If you cannot clearly state why you are conducting research, and how your organization will use the research, you should reconsider conducting the research.

Information Misuse

We are often in a position of asking research participants for their personal information. How will you use the information you collect from individuals? This includes what data you might collect through forms or on-screen fields if you are having users test a prototype of a design that includes collecting personal information. We are all familiar with End User License Agreements (EULA) that instruct us on our rights and the right of the company whose software we are using on how they will use any data we provide through our use of the software. Researchers should similarly make participants aware of any agreement they are entering into with providing you their data.

Are you having users enter potential sensitive information? For example, I’ve had users enter personal information into systems in order to generate accurate, real-life results for them to give feedback on. We made participants aware this would be a requirement for participating in the research, and gained their consent prior to scheduling the sessions. You shouldn’t present participants with a screen requesting their social security number with the expectation they will use their real information if you haven’t previously made them aware they should have an expectation of providing personal information.

We need to inform users prior to their participation on whether their information will be kept or destroyed. Researchers must consider how information will be kept and stored for future access. Academic researchers are required to submit a data storage plan when they submit a study proposal. This includes addressing how data will be stored and protected, who will have access to the data, the length of time the data will be stored, and what is the potential for others to gain access to this information.

We often ask our clients or team members to listen in or observe our research. You are obligated to make participants aware of the presence of observers. If you have a situation where there is an observer with an imbalance of power (for example a supervisor listening in on calls with their staff) you need to make your participants aware of this — including if the observation will take place post-participation through the sharing of a recording of the session. We need to avoid situations that might cause repercussions to our participants based on their responses or perceptions of their responses by others in power.

There is no one size fits all solution for data storage and access. It might seem logical to destroy data once a study is complete. However, what would you do if someone challenges your findings a year or two from now and you have no way to show your work? On the other hand, the longer you store data the more risk you take on. Who should have access to the data isn’t always clear, either. If a client or internal authority figure is requesting access to raw data, we lose our ability to accurately say how data will be used in the future. Researchers need a plan for how we will address these issues.

Potential Solutions And Ways To Avoid Ethical Issues

I’ve discussed six potential ethical pitfalls when it comes to conducting UX research. I’ve noted these pitfalls are more likely to occur if untrained researchers are designing and conducting studies. You might not be surprised that my solutions focus on proper training and governance of research activities.

I’ll cover the following potential solutions in this section:

• Participant experience focused protocols with informed consent script (immediate).
• Peer review of research protocols (immediate).
• Ethical research and sensitivity training (near term — as soon as reasonable).
• Data analysis training (near term — as soon as reasonable).
• Mentoring/Modeling (near term to midterm for formalized mentoring programs).
• IRB review (long-term/aspirational).

I’ve organized these solutions in terms of how soon you can begin to implement them from immediate to longer-term. Participant experience focused protocols and peer review require little or no cost and can be done immediately. Training-based solutions require some cost and need to be scheduled in advance, so I’ve listed them as shorter term to allow time, but still should be considered urgent. IRB review is something as a field we should promote and reward, but there are costs and time commitments that would alter plans, and therefore I consider this more aspiration for widespread adoption.

Participant Experience Focused Protocols With Informed Consent Script

When you engage in research, you need to create a protocol, documented procedures including the purpose of the research and questions you will ask, prior to engaging in research. Your protocol should reflect thoughtfulness and be thorough. The guidance I’ve received in the past is that your protocol should reflect enough detail that if you needed to call out sick for a day you are meant to collect data, another researcher would have the ability to successfully understand the study and administer the data collection.

You need to ask participants to provide informed consent at the beginning of the research. You need to inform participants they have the ability to withdraw participation at any time, and how their data will be used. If you intend to collect audio or video recording data from the session you need to gain permission to record. You also need to provide contact information for the participant in case they have a need to follow up post-participation.

The University of Michigan’s Office of Research Ethics & Compliance provides detailed guidelines for their researchers, which includes advice to write your consent targeting an 8th-grade reading level, and to have other people read through the consent to make sure it’s easy to understand.

The key to making informed consent worth anything is you, the researcher, understand it and what it obligates you to do and then you follow through on those obligations. You are telling the research participant they have the right to stop the research, you need to respect that should it happen. You are promising to safeguard their data. You are entrusted to use their data as you’ve told them and not for other means without seeking additional permission.

Peer Review Of Research Protocols

UX practitioners are a community of peers. We are obligated to support our peers and this includes offering to review protocols and research proposals for potential ethical pitfalls. You can formalize peer review as a part of your process if you have a large team of UX colleagues. Even better, have colleagues from other teams or departments review your protocols to flag potential issues.

Informally, we should be comfortable asking our fellow colleagues outside of our organization to review our protocols. We should consider peer review sessions or opportunities as part of our gatherings or conferences. We all stand to grow and improve when we share our perspectives and experiences. We also provide a good example of being vulnerable when we allow other researchers to review our work.

Ethical Training And Sensitivity Training

Anyone collecting data from human subjects needs to complete training that covers at least the potential ethics-related topics I’ve noted in this article. I don’t advocate any specific vendor for training. You can do your own research to find a quality organization providing training on social science research with human subjects. We should require researchers to take ethics-related training before engaging in research and then have continuous refresher courses.

Those who engage in ongoing research need continuous training and refreshers on how to deal with sensitive issues. Today’s benign issues are tomorrow’s sensitive issues as we grow as a society and in our understanding of appropriate research techniques. Additionally, you should understand the unique characteristics or cultures of the populations you are working with. This is critical to both avoid being insensitive, but to also understand the sample providing the data you are collecting. Sensitivity training should also touch on topics of unconscious bias and attempting to reduce bias through effective protocols for both data collection and analysis.

Data Analysis Training

UX researchers and team members potentially assisting in data analysis need training in effective data analysis. How can we expect people to make sense of the data collected if they don’t have an understanding of how to use the data? We need to require our research teams to continue to train formally and informally with each other. There is no substitute for experience when it comes to data analysis, which is why we should all participate in data analysis as frequently as possible, and with supervision from trained researchers whenever possible.

Mentoring/Modeling Good Research

I noted in the brief background section that we need to best prepare our team members for engaging in ethical research. Those of us with more experience conducting research have a responsibility for modeling good research practice to our less experienced colleagues. You can operationalize this in many ways, through formalized mentorship relationships, or through informally having colleagues observe sessions and participate in supervised protocol design and data analysis.

We should be willing to take on a role to educate others and serve as examples for others on conducting ethical research. Specific to two of the issues I’ve raised, this includes:

1. Sensitive Topics
   We have an opportunity to involve our colleagues with less experienced research in observing us when we do conduct research that might broach sensitive topics. We can model response (or non-response) to emotional or provocative reactions from research participants. We can also role-play with our colleagues prior to engaging in research to practice how we might respond to unexpected reactions.
2. Misinterpret Findings
   We can involve our less experienced colleagues in the upfront discussion on what questions we are asking and why. We can show how we are tying the questions to our hypotheses in the beginning, and discuss what type of data we expect to collect and how we will analyze this data. We can involve our colleagues in sense-making sessions (some folks lovingly refer to these as data jam sessions) where we analyze our data and interpret our findings. This helps model how research goes from the messy stage of collected data to the more refined and presentable stage of findings and recommendations.

We have another powerful research method our colleagues can use to help ensure ethical high-quality research: observation. Having colleagues observe your process and take notes is valuable to both you and them. You can review their notes together and point out what you were thinking at specific times during a session and why you reacted the way you did. You can also use their observations as a lesson for yourself and reflect on your own performance.

IRB Review

Institutional Review Boards exist for a reason. We should use them. I saved this idea for last because I think it is the most difficult to realize. I acknowledge using an IRB would cost money and add time to the process of engaging in research. However, I’d argue that any research conducted under the authority of an IRB would align with what we could consider the gold standard for ethical research.

I wouldn’t realistically expect UX teams to incorporate IRB review as part of their process for running usability testing with six users for your online banking app. I would expect the use of an IRB for the following situations:

• You want to publish your research as academically valid.
• You intend to work with vulnerable populations, including children.
• You have aspects of mental or physical health to your research (e.g., you are testing a device that includes a heart rate monitor and you intend to ask participants to engage in a physical activity).

Putting It In Place

As we push to increase awareness and participation in research-related activities, we need to be mindful of potential ethical pitfalls in allowing inexperienced team members to collect and analyze data. I’m not the authority on ethics, and most likely neither are you. That’s why we need to take steps to safeguard our research from venturing into unethical territory. We stand to damage the reputation of UX and UX research if we move forward with conducting potentially unethical research.

I’ve presented six potential areas ethical issues might arise, as well as some potential solutions. Moving forward, you can start with the easiest and most accessible solution of running your protocols by other UX practitioners for review or requiring your UX team members to do this. Additionally, you should codify the language that needs to be included in your informed consent and ensure it covers the minimum requirements to make participants aware of their rights and how their data will be used and stored.

Pitfall	Solutions	Comments
Vulnerable populations	Ethical training and sensitivity training,Peer Review of Research Protocols,Mentoring/Modeling, IRB review, Participant experience focused protocols with informed consent script	IRB is the ultimate authority on vulnerable populations and protocols for studies that might include vulnerable populations. Guidance is to avoid vulnerable populations without IRB approval/oversight
Misleading Users/Deception	Ethical training and sensitivity training, Peer Review of Research Protocols,Mentoring/Modeling, IRB review, Participant experience focused protocols with informed consent script	Solutions focus on avoiding misleading users through education, effective protocol, and review of protocol. IRB approval/oversight is recommended if you intend to intentionally mislead users as part of a study
Inadvertent sensitive topics/extreme experiences	Ethical training and sensitivity training, Peer Review of Research Protocols,Mentoring/Modeling, Participant experience focused protocols with informed consent script	Solutions focus on gaining education, experience and comfort in handling unexpected issues as they arise. Effective protocols include carefully worded questions and prompts for potential responses if they arise.
False expectations	Peer Review of Research Protocols,Mentoring/Modeling, Participant experience focused protocols with informed consent script	Solutions focus on clarifying the purpose of the research in the protocol and all communication, while gaining experience and comfort in handling participant concerns and expectations, and responding to unexpected situations as they arise
No idea how to use/interpret the findings	Data Analysis Training, Mentoring/Modeling	Training and experience are key for effective data interpretation. Effective protocols will tie questions back to hypotheses which will assist in data analysis.
Information misuse	Ethical training and sensitivity training, Peer Review of Research Protocols, Mentoring/Modeling, IRB	Protocols need to account for how data will be used and kept secure, peer review and IRB are outside sources we can use to screen our protocols for ethical data use and storage

We should push our peers toward more formal efforts to review any protocols intended for use with vulnerable or protected populations. We should explore and demand a budget for training and refresher courses for both experienced and new UX practitioners. We should also explore a standard of requiring IRB approval for any research findings researchers share in a more public forum such as publishing in journal articles or presenting findings at conferences if these findings are intended to be generalizable and considered gathered under valid research conditions.