What They Say and Do: Practical Tips for Harvesting Reliable User Feedback for Planning
Sunday, 1st October 2006
Ulla de Stricker
[This article reflects key points in a presentation to be given by the authors at the Internet Librarian International conference in London, 16-17 October 2006 <http://www.internet-librarian.com/>.]
When a website, an intranet, an information service or a marketing/communications campaign -- you name it -- does not see the traffic it 'should' see (given the size of targeted user groups, the number of association members, the number of individuals in a demographic group and other guideposts), those in charge of that service are understandably concerned. Did we misinterpret what users were telling us? Did we not get the full picture when we asked them? Have users' needs shifted and we missed it? Were we even tracking the evolution of those needs?
Over time, services can, and often do fall out of alignment with user requirements. It is necessary to be vigilant and persistent in monitoring those requirements -- not just through listening to what users say publicly about their needs when asked, but also through observing what they do in practice. Further, it is important not to be tempted to go with unsolicited input (the squeaky wheels) but rather to launch a systematic scrutiny of what users and non-users alike think about the service in question.
User interaction with all things electronic -- websites, intranets, extranets, e-newsletters, web stores, e-marketing -- is a complex affair (unless, of course, that interaction consists of hitting the Delete or Close button). Some of the interaction we observe through traffic and click-path tracing tools; but such tools naturally can't help us understand what went through the mind of the individual who just clicked, much less help us predict what he or she might do in future. Moreover, they do not help us determine how representative the individual is of the total potential user group.
Those in charge of (re)building electronic content delivery and communication vehicles need a deep and crystal clear understanding of user preferences. But that's an oversimplification. We need to understand what they want to get done when they come to our site or service, and what they would like to have in hand when they leave. Once we understand that, we can work on making the process smooth and enjoyable.
To arrive at such an understanding, we need answers to questions about several facets of the user experience:
Context: What specific goals are users hoping to accomplish when they paste in that URL or click that bookmark? Do they have a concrete task to finish (what is the conversion rate for a currency; what is the latest news on an organisation), or are they conducting broader research (what are the NGOs saying about X issue; what is the medical consensus on Y drug; what is the market outlook for Z products)?
Mood and attitude: Are they under time pressure to solve a problem at work, or are they at home, browsing to see what's new or interesting in an area of personal interest?
Past experience: What are their expectations of the navigation and content options, based on past exposure to similar sites or services?
Subjective impressions: What features in appearance, navigation and content presentation do they find intuitive, easily followed, complicated, ambiguous, confusing, totally baffling or downright annoying? Can they articulate why?
Willingness to share reactions: Which of their many possible reactions ...
- Will they offer voluntarily? ('I really like ...')
- Would they offer if pressed? ('Well, now that you mention it,
I suppose ...')
- Would they keep to themselves? ('Saying I'm confused could make me
appear unsophisticated or ...').
Likelihood of and reasons for returning voluntarily: Aside from situations where users are forced to visit a site because there is no alternative, how interested are users in coming back, based on their initial visit?
- 'I accomplished what I needed to do eventually, but it was so
frustrating that I'll look for another resource'
- 'No other sites offer this content so I'll return, but not happily'
- 'I didn't accomplish what I needed to do, but it was so much fun I
think I'll go back another time'.
Having any hope of building and maintaining compelling websites, information services and communications campaigns necessitates a thorough understanding of existing -- and potential -- users' context, goals, attitudes, experience, impressions and desires. This tall order cannot be filled by an annual survey or the odd poll! It requires a systematic approach to assessing and monitoring user needs and then to implementing new tools and technologies geared to meeting those needs.
Below, we offer a set of practical tips on how to obtain credible evidence of user opinion. None are surprising, but that does not lessen the discipline needed in carrying them out regularly.
The practice of assembling input about user opinion goes by any name you care to give it. If 'audit' is too ominous a word in your culture, just choose some other designation (e.g. review, assessment) to indicate that some form of examination is involved. The key is to undertake a systematic investigation that will yield information upon which to act.
Below we address two scenarios: an audit of a communications programme and an audit of a website or intranet design.
In the case of investigating a communications programme, the communications audit is an effort to understand what users seek from the organisation or service in question. You're looking to find out where they go if they can't find the information they need from you and what other organisations or services (i.e. the competition) are succeeding in communicating to the target group so that target users seek them out first. Having such an understanding will help those in charge of communicating with the target market as they plan how to:
- Present the information that needs to reach the target audience in
such a way that it is useful to the audience and is reinforced
through several means
- Avoid overloading the audience with too much information for them to
absorb and use
- Present information in the way users expect or seek to find it
- Minimise staff workload as a result of increased number of
communications vehicles, message frequency and message
- Use techniques such as knowledge-sharing, stories, ideas and best
practices exchanges, and interactivity to serve the target group.
In the case of a website or intranet audit, the goal is to understand how the users experience their interaction with the site. What parts are intuitive for them, and where do they scratch their heads? What makes them say, 'Cool!' and what makes them say, 'Huh?' What's missing for them? Is there too much irrelevant clutter?
Ask, or watch?
In addition to the 'please tell me' method of investigation, the at- the-elbow observation technique yields valuable insight. There are two variants:
- Silent observation: 'Just let me sit here and watch as you go about
typical tasks, and allow me to ask why you did this or didn't do
- Easter egg hunt: 'Please find the answers, using this website or
using your memory of having received a promotional message, to the
following specific questions'. Users' success rate tells a lot about
what needs to change. A key advantage of the egg-hunt method is that
users participating in the study cannot get by with low-value
statements like, 'It seems alright', or, 'Yes, I heard of it'. They
actually must dig in and look for something specific. In the process
they must explain their search path or memory triggers.
Asking methods range from the highly involved (for example, an in- depth personal interview) to the minimally involved (for example, a survey). Similarly, observation and challenge methods range from the up-close, at-the-elbow, click level to the remote analysis of aggregated click statistics.
The mix of methods chosen depends on several factors including corporate culture, level of detail needed, size of total user population, amount of time available, the skills of staff members engaged in carrying out the assessment, and many more.
These steps will help you decide what kind of investigation you need to launch and how to carry it out.
Step 1: What type of investigation is needed?
The audit pre-work consists of a determination of the precise kind of insight needed. For example:
- What do we already know? (Users have been complaining they can't
find ...; or it seems members of the target group haven't understood
- What statistical evidence is at hand? What does it suggest?
- What do we know we don't know? (Is it a matter of training? Or must
the site be redesigned? What was confusing in our message?)
- Who do we need to hear from? (Did the complaints come from a certain
group of users or from across the board?)
- Who can undertake the work of getting the input? (Can we do it
ourselves, or should we hire in consultants -- who might obtain a
more candid picture?)
- What mix of methods is best suited for the situation at hand?
(Personal interviews, at-the-elbow observations, Easter egg hunts,
focus groups, a survey?).
Key pros and cons for each type of method include:
- Personal interviews: Advantages are that these unearth detail and
let you unlock the potential of candour in your subjects. They give
you the opportunity to ask, 'Why do you say that?' and help you
build a relationship with your subject. The disadvantages are that
it may be difficult to reach the 'right' interviewees through
personal interviews. They're time-consuming, so you can only reach
so many individuals.
- At-the-elbow observations: These are helpful for helping you witness
how people use your service. They let you see where users hesitate
or if they misinterpret a button, and they can be much more
illuminating than interview comments. Again, however, it may be
difficult to reach the right subjects through these, and they're
also time-consuming, limiting the number of individuals you can
- Egg hunts/communications effectiveness test: A benefit here is that
the documented success rate they produce serves as a mirror of how
well the site is designed and how well the communications campaign
succeeded. They isolate the key specifics that need to be addressed.
On the down side, users may not be keen to 'take a test'.
- Focus groups: A focus group is an efficient way to get input from
users (count on six to eight participants per 1- to 2-hour session).
Participants tend to feed off each other, driving more detail.
Scheduling makes this time-consuming, and it does require an able
facilitator who knows how to keep the group on track.
- * Survey: This will give you the opportunity to quantify how many
agree with X opinion and measure change over time by comparing
responses with the last survey's responses. However, respondents may
not want to take another survey, making for a low response rate and
unreliable results. It's also difficult to devise questions that get
at nuances in a site. Another hazard of a survey is that respondents
may 'upgrade' their responses, for instance click 'frequently'
instead of 'occasionally'.
Step 2: Clipboard -- getting the raw input
Once the audit process has been planned, the detailed work of 'sitting down with the users' and 'collecting the responses' ensues. The note- taking and documenting is quite time-consuming. If the examiners are emotionally attached to the site or activity in question, it requires a willingness to document the less flattering input as clearly as the more flattering input!
Step 3: Rollup -- what are the key themes? What might they mean?
Now comes the editorial phase. What are the key messages and takeaways? What should we do with input that seems to fall outside the key findings? How can the findings be translated into pointers toward directions? Does what we observed in one instance (e.g. with respect to a website) carry over into other areas such an e-newsletter or the intranet?
Step 4: Translation into specific plans
This is the stage when the difficult analysis takes place. Ask yourself: What does it all mean? How can the key themes be translated into concrete design or process changes? What will it take to implement them?
Step 5: Implementation and measurement
These last two steps may seem daunting, but they are important. First, having acted on user input, it is crucial to demonstrate to the study participants (and all others) that the input caused specific change. In other words: 'Thanks. We heard you. We acted accordingly. It will always be worth your while to participate when we come looking for input'. Second, we must find out whether the changes had the desired effect: 'Did we change the content or design in the right way? Has your issue been resolved? Is the site easier to use now? Are you better able to understand what we are communicating now?'. Naturally, traffic- and usage-indicators will tell their own story about the degree to which the entire exercise was successful.
Words of advice from veterans
As seasoned consultants in the business of helping our clients get the most out of their investments in services and content, we offer in conclusion a handful of tried-and-true tips for anyone about to launch a user input project:
1. Distance the owners from the input gathering. The team members in
charge of building and maintaining a website or operating a service
programme know too much and will have difficulty maintaining
neutrality and objectivity as they receive user comments. They will
tend to unconsciously suppress vital details.
2. Inflict no pain. Users are busy; make the input process smooth --
fun is even better. Easter egg hunt contests, with prizes to be
won for (1) the highest success rate and -- this is key -- (2) the
most illuminating commentary pointing to navigation issues, works
3. Reward participants. First, show appreciation for the
participation through simple gestures -- coffee and cookies,
thank-you notes to the manager, public acknowledgement. Later on,
tell them what the result was (see item seven below).
4. Prime the input pump by making it safe for participants to be
candid. Clearly signal they are in good company and that they will
not be seen as untrained or incompetent if they admit they are
confused: 'You are not the first to say that. We are intrigued. Can
you elaborate?' Or, 'Your colleagues have indicated they find X
ambiguous. Do you, too, find it confusing?'. Reaffirming that
responses will be kept anonymous may help individuals feel more
comfortable speaking frankly than might otherwise have been the
5. Secure future cooperation without narrowing user input to a pool of
regulars. 'May we count on you again as we test the revised
website? Could you suggest colleagues whose input may be useful in
6. At the end of the audit, it is vital not to just fade away. Having
asked for input obliges us to show participants that the input
resulted in change, or will do so. We recommend a summary to
participants indicating not only gratitude for the input, but
specifically a rundown: (a) we are able to do X immediately; (b) we
can achieve Y in such-and-such a time frame; and (c) options for
certain of the more ambitious changes are being investigated. Where
a desired change cannot be implemented just yet, or at all, explain
the reason specifically: 'We recognise that your suggestion is a
desirable feature and regret that at this time, resources limit our
ability to implement it' not only clarifies that the user input was
valid and has been heard, but could be a means of generating some
grassroots pressure for more resources! The key is to signal that
you heard the input the participants offered up, and to keep
everyone in the loop as to what is coming up.
Related FreePint links:
About this item:
- Item title: What They Say and Do: Practical Tips for Harvesting Reliable User Feedback for Planning
- Publication Date: Sunday, 1st October 2006
- Link: http://web.freepint.com/go/blog/2541
- Views: 3682
Ulla de Stricker is a Toronto-based information and knowledge management consultant assisting her clients in strategic projects, often involving approaches to sharing and managing the information knowledge workers acquire for their projects. In her IKM blog at http://www.destricker.com, several posts comment on practices for dealing with information related challenges. Prior to establishing her practice in 1992, she held senior roles in the information industry. Ms de Stricker is a frequent contributor to the professional literature and to conferences.
More articles by Ulla de Stricker »
« See all FreePint Blog items