We knew that for this survey, we were dealing with a sensitive dataset: in order to answer our research question, we would have to ask respondents to provide their personal demographics and compensation information.
The focus then became on making the survey easy to answer, as well as minimizing the pain that can come with trying to reconcile identity with the survey questions. We referenced several sources for best practices in writing asking identity questions, including “Respectful Collection of Demographic Data” by Sarai Rosenberg and “Rethinking and Updating Demographic Questions” by Jennifer L. Huges, Abigail A. Camden, and Tenzin Yangchen.
We used an open input for as many questions as possible. This requires more analysis and the chance for error, but ultimately felt like the responsible decision for our relatively small data set. In a survey with more reach, that would have been a more difficult decision to make.
For some questions, where it felt like we needed to seed some options, we consulted the materials to find the best practices to represent the groups we were asking about, and always provided a “Type a Free Response” option for participants to capture what we didn’t ask about.
Link to all questionsOnce we collected all results, we had to prepare the responses for analysis. This meant coding the free responses to all share the same language. For example, from the question “How do you currently describe your gender identity?” we coded female responses as woman to use consistent language, in this case gender identity terms. Where people wrote something specific, such as non-binary or non-conforming, we used the language that they chose.
With a relatively small dataset (331 total responses), the specifics of respondents identity can come through in a raw presentation of data. We decided to group responses together in meaningful ways that still provided answers to our core questions. (If we were unable to do so, we excluded that data from our analysis.)
When looking at pay disparity, it is well-reported naitonally that cisgender straight white men earn more than their counterparts across gender and racial/ethnic identities. We therefore did analysis of this majority against all other groups, such as “Men vs. Non-Men” and “White vs. Non-white.” This has two unfortunate effects:
However, the potential harm from either completely removing this data or exposing the compensation profiles of a small set of responses outweighed these concerns for our team.
Job titles were a major hurdle for analysis, and required significant effort. Job titles vary considerably, and we were left to lean heavily on the responsibilities that respondents selected to place responses in specific groups. For example, every individual contributor who did UX design and visual design fit into UX/UI Designer. Without Visual design they became a UX Designer/Researcher. If they added front-end development they became a Front-end Designer, a category we didn’t expect but was quite large. Once combined, we validated our groups by comparing the data in years of experience, responsibilities, and compensation.
We used this site as a critical entry point to survey, because we knew that we needed to provide a lot of context to prepare participants for what they were to be asked. In terms of the survey itself, we tried out multiple survey platforms, ultimately selecting Typeform due to their pricing plan and accessible surveys. Results were then exported to Airtable for coding and grouping of responses, before ultimately pulling results into Tableau for analysis.
There were many categories of information that we were unable to include due to limited responses, including transgender identity and disability status. It was important to ask about these, and we would do so again, but would have been irresponsible to make claims about the data from such a small set.
Likewise, the data of freelance and unemployed designers was too small and varied to responsibly share.