Why the creation of personas can benefit from quantitative research

by Ivar Slavenburg

With the slide „Personas are bollocks” he certainly made sure he got the attention of the audience during his PechaKucha at the Service Design Global Conference 2013 (an overview of this conference can be found here). The field of creating personas is dominated by qualitative methods and to some therefore considered „non-scientific”. Stephen Masiclat (picture) explains how this popular method can benefit from using quantitative research.

Could you explain your role at Syracuse University?

I actually have multiple roles. I am a professor and the director of the graduate programme in New Media Management at the S.I. Newhouse School of Public Communications. I also direct the Center for Digital Convergence, a research lab that designs, optimizes, and prototypes new communication management systems. Finally, I also own and operate an independent design consultancy where I research, design, and develop multi-platform customer touch-points for clients.

What kind of research do you normally do?

In my work for the Convergence Center, I lead research into operational and content optimization for organizations as well as SEO. My graduate research teams analyze client web sites, content strategy, workflow, and audience analytics to provide greater efficiency and better content services.

My consulting work is more wide-ranging. Most recently I completed a large-scale usability study for a multinational corporation’s web services platform. Currently I am leading their mobile app development team to simultaneously build both the customer-facing app, and improve the back-end support systems.

In November you held a PechaKucha at the Service Design Global Conference in Cardiff. What point did you try to make?

My main thesis was that there are quantitative research methods for generating personas that bring statistical rigor to a process currently dominated by qualitative methods. Quantitative methods are generally better received by the classically trained executives who must execute and manage the stratagems service designers provide.

I used a recent client study to demonstrate the dilemma. A corporate client asked for research, but also dictated we use focus groups solely because it is a method with which they were familiar. The executive in charge was not comfortable defending qualitative methods. In the end, I had to use Q-methodology because the focus group method did not allow me to discern and study the target personas.

You had a clear opinion about the “normal” way for determining personas. Can you explain your position? On which findings was your opinion based?

To be clear, I arrived at Q-methodology because of another research team’s disdain for traditional methods of discerning and describing personas.

The slide I put up at the service design conference that said “Personas are bollocks” was my provocative paraphrase of a research article I found in my search for other methods of developing personas. When our first attempt (focus groups) failed, my team and I searched for another method to discern and validate personas for our client’s proposed service. As part of that search, we conducted a review of the research literature where we found articles questioning the validity of personas in the design process.

In 2006 C.N. Chapman and R.P. Milham published "The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method" (Christopher N. Chapman and Russell P. Milham, 2006) in „Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 634 –636.)”. In their article, Chapman and Milham said personae were non-scientific and essentially useless for guiding meaningful design. I found their article while I was researching alternative methods for persona generation. When I read their objections I was particularly struck by the claim of non-scientific status.

The core of Chapman and Milham’s objection is twofold. First, there is an infinitesimal probability that a traditionally-derived persona is valid since designers “string together a series of characteristics that each have a probability of less than 1.0.” Mathematically, this means the likelihood of a persona being valid approaches zero. Second, there is no accepted method of validating the set of characteristics that designers chose to assemble into the persona portrait. They are essentially arbitrary collections of characteristics with no method of falsification.

I have a faculty colleague (Dr. Dennis Kinsey) who first introduced me to Q-Method as the science of subjectivity. When I read the Chapman and Milham article I realized my colleague had given me a scientific (i.e. repeatable, falsifiable) method for studying subjective phenomena like attitudes, opinions and states-of-mind that are the basis of personas.

If not dealt with properly, what can be the consequences of using traditional personae creation techniques?

Every researcher tries to ensure they are looking clearly and fairly at the phenomena they are studying, and much of our methodological rigor is applied to protecting against the biases that skew perception. These biases are very difficult to overcome, and in qualitative methods they are especially difficult to control. Perhaps the most well-known bias is the participant effect. It is well documented that participant observation can skew results. There are also normative effects. When people are in groups, it becomes extremely difficult to discern individual attitudes and behaviors because the group dynamic sets the norms for behavior.

As an alternative you mentioned the Q-methodology. Can you explain this method a little bit?

I’m going to do a terrible job of this, and I strongly recommend anyone interested read Professor Steven Brown’s Primer on Q-Methodology.

Here’s my brief explanation which builds on the article I published in Touchpoint 5.1.

Q-Method combines two very powerful components−Concourse Theory and Q-Factor analysis−and it does this in service to scientifically studying subjective phenomena like personal preferences and attitudes.

It’s best to think of Q- Method as providing a rigorous way to measure the similarity (that’s the factor analysis) between people’s thoughts and ideas about a subject (the Q sample derived from the concourse).

To understand a concourse, think of the full set of things people say about a subject–the texts in all forms be they writing, captured conversation, even speculative musings. What concourse theory says is that you must try to capture a representative sample of the subjective statements about a subject, and then let people array these in a fixed manner. Essentially you take a snapshot of how they arrange ideas in their mind along a condition like strong agreement to strong disagreement.

When you subject these captured personal arrays to factor analysis, you can develop groups of “like-minded” people who share the same fundamental ideas about a subject.

What my team and I realized is that the factors that emerge during analysis are built around a common mindset which is the basis of a rigorously defined persona. In effect, the factors are personas, and all that is left is to transcribe the predominant demographic characteristics of the people who comprise the factors.

What are the main differences between traditional methods for creating personae and the Q-methodology?

The single most important difference in this method is that there is no a priori filter on the data.

In traditional research terms, there is no null hypothesis to prove or disprove. Instead, you get a statistically significant grouping, or you don’t.

In much traditional research we unconsciously apply filters. For example, here’s a task:

My technology company wants you to describe the potential users of a new automobile sharing service. For this particular service, I want to design touch-points on a mobile phone app.

We want you to study potential customer’s current behavior to determine an ideal service journey for a subscriber.

If right now you are picturing a young person living in a city who cannot afford a car then you have fallen victim to my example. Why is this an urban service? Who says older folks won’t use a smartphone to order transportation? Those a priori ideas might be founded on very good assumptions (e.g. people in cities with good mass transport  can’t afford to maintain cars and pay for parking) but that constraint might cause you miss the very large market of elderly in suburbs who want a means to safely shop and visit friends on very personal schedules.

What Q-method invites you to do is enunciate the reasons why people might make a choice, and then see what reasons coalesce into a mind set. The researcher and his very reasonable ideas don’t have inordinate weight, and the real world tendencies are free to emerge.

Consequently, what are the advantages and disadvantages of the Q-methodology?

The biggest advantage is that you can be surprised. When you don’t bring any “expertise“ to your research, and when you allow the findings to emerge from the data, you can learn so much more.

Another strength is that you can test assumptions. Speculative statements−even outlandish ones−can be part of the Q-Sample. Given the chance, people will array these statements as they see fit, and an idea that is speculative might actually be seen to be a central notion to a particular type of person−a persona.

Perhaps the biggest weakness is that familiarity with statistics is helpful. More advanced practitioners use methods like eigenvector rotation to discern more insights from the data, but advanced maths are not absolutely necessary to benefit from the method.

Also−and I must stress this limitation−this is not a counting method. Q-Method will not tell you how prevalent a persona type is in the general population, only that it is real and based on replicable phenomena.

Did you use the method in other researches as well? In what way or to what extent did the methodology improve the results?

I have used the method in a number of audience studies as part of the optimization work I do in the Convergence Center. It has allowed me to see what content matters to what audience segments. It provides nuanced insight to the big data and site analytics we collect.

What impact does Q-methodology have on professionals already working with personae? Are there any new skills required?

This is a tool that helps designers by giving them user portraits with depth and measurable accuracy. It also gives people who are often dismissed as engaging in “soft” methods a quantitative tool that is easier to defend in corporate board rooms where MBAs hold sway.

As for new required skills, the most important skill is an open mind. Openness to new modes of inquiry is crucial to continued professional growth. Moreover, openness can be transformative.

Consequently, what advice do you have for them?

I invite you to test this method for yourself!

While I gave a talk at the Service Design Conference, my original intent was to hold a workshop to teach service designers how to use the method. And as I stressed in my talk, the tools are free or very inexpensive, and the community of Q-Method researchers is very welcoming.

I am the beneficiary of that generosity. My colleague Dennis Kinsey shared his expertise and helped me become a better designer and design researcher. I hope to do the same for the Service Design community.

More information…

You can follow Stephen Masiclat on twitter: @masiclat

Further Reading

  1. Christopher N. Chapman and Russell P. Milham (2006), The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method. Retrieved from http://cnchapman.files.wordpress.com/2007/03/chapman-milham-personas-hfes2006-0139-0330.pdf.