Understanding how the public interprets a message is part of the communication process and requires time, dedicated staff and a budget. A useful evaluation should help us understand what people grasp, what expectations or doubts our message raises, and what needs reviewing based on their responses.
We can start with a specific question and by talking to members of our target audience. For this effort to be worthwhile, the remit must also specify who will analyse the findings, who will make the decisions, and the scope for implementing changes.
Imagine a small cultural association that comes across a European call for proposals that its project might fit. It has never applied for funding like this before and, while reading references to transnational partnerships, outcomes, and impact, it starts to think the programme is aimed at more experienced organisations. Although the guidelines allow organisations like its own to participate, it ends up closing the page.
The team that prepared the call for proposals specifically wanted to reach out to new organisations. They reviewed the requirements and worked on promoting the call, but they didn’t check how someone encountering the text for the first time would interpret it. A gap existed between what the team intended to explain and what that association understood, a gap that visitor figures alone could not reveal.
The writer knows the context, the objectives and the discussions that have shaped the project. The reader arrives with their own experiences and tries to understand what that information means to them. They may recognise every word yet draw a different conclusion from what we expected.
Studying how a message is received involves paying attention to that experience: what the person understands, how they relate it to their own situation, what they expect, and how they assess it. They may understand a proposal but feel their views were not considered, or they may identify an opportunity the team had not anticipated. Listening to these interpretations allows us to review both the materials and the decisions they communicate.
This is the aspect of communication evaluation that concerns us here, though it should not be confused with a comprehensive measurement of its effects. The AMEC evaluation taxonomy distinguishes between dissemination, understanding and subsequent outcomes.
Knowing how many people have opened a page provides some insight into its use; to understand a specific difficulty, we need to observe how they read it or listen to their explanation. A study by Europeana illustrates what this step can contribute.
In 2022, Europeana, a digital cultural heritage platform, published a study of its website involving eight participants with visual impairments. The research combined interviews with tasks such as reading articles, searching for content and navigating the site.
One finding concerned language: when participants selected a language other than English, the titles were translated, but the article content was not. This confused participants whose first language was not English.
Contrast issues were also identified. Based on these observations, the researcher designed a dark-mode prototype and showed it to participants, who found it more accessible. The publication left the implementation question open; it did not present an improvement already rolled out across the entire website.
The study focused on accessibility and user experience. It contributes to this topic by showing how specific conversations and tasks can reveal difficulties that require attention. Its qualitative results do not allow us to calculate how many users experienced each problem, nor do they make the sample of eight participants a valid figure for any project.
To apply this approach to a campaign or call for proposals, we need to decide what we want to understand and what we could change. This decision lets us design a consultation proportionate to the available resources.
‘Is our communication working?’ covers too many issues for a brief consultation. A question such as ‘Do organisations coming to us for the first time realise they can get involved?’ helps us decide what text to display, who to invite and what we need to hear.
If we want to understand the experience of those unfamiliar with the programme, consulting only regular contributors would leave out an important perspective. We will also need to consider the languages, formats or support mechanisms that will enable the people we really want to reach to take part.
We can show a snippet and ask the reader to explain in their own words who they think it is aimed at and what they would do next. Asking them which part led them to that conclusion helps to pinpoint difficulties. To gauge their assessment, we can also add: ‘What would you expect from this programme?’ or ‘What do you think of the conditions it sets out?’.
On the other hand, asking ‘Is it clear that no previous experience is required?’ anticipates the answer we are hoping for. Someone who has spent time writing the text may find it hard to hear another interpretation without clarifying it straight away, but waiting before intervening lets us understand how the reader arrived at that interpretation.
It also helps to explain from the outset that we are reviewing the material and that their queries help us to improve it. This way, the person knows they can point out anything unclear without feeling they should have understood it.
A brief conversation allows us to explore an interpretation in greater depth; reviewing enquiries received by email can help us identify recurring queries. These methods complement each other, but each has limitations: emails exclude those who left without asking, while conversations with a small group don’t let us gauge how many people in the wider audience share the same view.
In a call for participation, someone might fear that a critical opinion could affect their future application. That is why this consultation must be kept separate from the selection process, and we must clearly explain why we are asking for their input, what information we will collect, and who will have access to their responses.
GOV.UK’s guidance on consent in user research explains what information to provide and emphasises that participation is voluntary. Individuals must be able to make an informed decision based on this information and withdraw from the conversation if they wish.
For a simple assessment, notes on the difficulties observed may suffice, without routinely recording the conversation or collecting unnecessary personal data. If we use previously received feedback, we must check what use was explained to the respondents and how their information will be protected.
In my article on applied research in institutional communication, I explore other ways to work with limited resources. Choosing a method makes sense when we know what decision it will help us make and we have time to review the results.
Hearing an unexpected response may lead us to amend the text too hastily. First, we need to understand what happened, because someone may have misinterpreted a condition, found it difficult to meet, or simply disagreed with it. Although all three situations result in an application not being submitted, they require different responses.
In a call for applications, a misunderstanding of the requirements might lead an organisation to believe it needs prior European experience when it does not. Asking them to explain what led them to think this helps you decide whether to clarify a sentence, make a condition more prominent, or include an example of an eligible organisation.
A practical barrier requires further discussion. If an organisation understands the requirements and wants to take part but cannot cover the travel costs upfront, a clearer explanation of the call for applications will not resolve this financial difficulty. The communications team will need to pass this on to those managing the programme to assess whether alternatives exist.
There may also be disagreement over the conditions: an organisation may understand what is being asked of it but consider the obligations excessive given its resources. Listening to this objection involves assessing their reasons and explaining the subsequent decision, even if the conditions remain unchanged. Treating it as a misunderstanding would prevent us from addressing the issue they are raising.
A question of legitimacy also arises whether people consider the organisation’s decisions, and how they are made, justified. Listening to their objections allows us to understand these assessments but does not guarantee acceptance. The response may require revising a condition or explaining why it is being maintained, rather than simply seeking more persuasive wording.
These differences help to allocate responsibilities. The team collecting the responses needs to know who to refer each issue to and when it can be reviewed; otherwise, a report may describe a problem well without anyone ever deciding on it.
When the response involves modifying a message, schedule a second test with other members of the target audience under conditions like the first. This lets us see whether the new version is understood without the explanation we gave the first group. This test helps review the adaptation, but on its own it does not prove it will increase participation or that the entire audience will understand it.
Those who contributed also deserve to know how their time was used. In public engagement, the OECD recommends explaining how contributions are used and why some are not incorporated. In a communication consultation, we can apply this care by explaining what we changed and what we kept, and why. Preparing this response, just like implementing and testing the adjustments, is part of the work we must plan for.
Talking to a few people may seem like an easy task to fit in until you break it down into its individual steps: finding participants, preparing questions, coordinating schedules, interpreting responses and bringing together those with decision-making authority. If the budget covers only drafting, designing, and disseminating, we’ll have to find the time by postponing other tasks, increasing someone’s workload, or revising the project’s scope.
Putting it in writing allows us to assess the effort required before requesting it. It also helps respect the time of those consulted: invite them when you know what you can do with their contributions and who will follow up on them.
The terms of reference, or ToR, describe the work being commissioned and its conditions. To include this evaluation, we need to specify the question, the materials to be reviewed, the deliverables and the scope for making changes. NCVO’s guide to preparing an evaluation brief helps to organise these decisions.
In a proposal to review a call for proposals, we might include a description such as this:
Before launch, we will assess how organisations with and without experience of European projects interpret the requirements. We will gather queries, expectations, and objections to propose adjustments to the materials and raise any issues affecting the programme’s conditions. The programme manager will review the recommendations and decide which ones to implement. The timetable will allow time to introduce the agreed changes and test the new version.
To turn this description into a workable assignment, we still need to agree on participants, sessions, languages, deadlines, and the materials to include. The deliverable could be a brief report outlining the difficulties, recommendations, and decisions recorded. Asking the evaluator to produce favourable opinions would directly bias the very information we wish to obtain.
NCVO includes staff time and analysis among the costs to factor in when carrying out an evaluation. In a review of messages, I would organise the work into five parts so as not to overlook what happens after listening:
Each stage requires a time estimate, a designated person in charge, and any external costs it may incur. Just because someone is on staff does not mean they have spare time for this task.
The timetable must accompany the budget. If the materials have already been printed by the time the results arrive, changing a single sentence may require a new print run. If the web editor’s schedule is fully booked, the adjustment may have to wait even if it has been approved. That is why it is advisable to consult the public before printing and to set aside time with the team to implement the changes.
When resources are limited, start by addressing a question that affects a key decision: who can take part, what commitment is being made, or what cost will be borne. Narrowing the question lets us focus our efforts, provided we explain what we’re leaving out. Reviewing one language version, for example, does not mean the others have been checked.
If the commission has already been finalised, we will need to agree on how to incorporate this work: redistributing tasks, adjusting deadlines, or expanding the scope. To do this judiciously, it helps to describe the difficulty we wish to understand and the effort it requires, rather than relying on the team to find a spare moment.
Listening to the audience means accepting that material we have worked on carefully may need changes. Anticipating this possibility from the outset gives the team time to respond, and those consulted an opportunity to influence something specific. This is where it makes sense to set aside part of the budget to understand what happens after the message is communicated.
It depends on the question and the profiles and languages involved. A small-scale consultation can identify specific difficulties, but it doesn’t let us estimate how many people in the wider audience share them. If you need to compare groups or calculate that proportion, a study tailored to that objective will need to be designed.
Understanding means grasping what is being communicated; agreeing implies accepting or sharing its content. A person may understand a programme’s terms yet consider them unreasonable. Distinguishing between these two responses prevents one from attempting to resolve, with an explanation, what requires assessing an objection.
Identify a key concern and estimate the work required to address it. Using that information, agree on whether it is appropriate to redistribute tasks, adjust the timetable or extend the scope of the project. A smaller consultation may be feasible if it allows time to analyse the responses and implement the approved changes.
Show the new version to other members of the target audience, under similar instructions and conditions, and ask them to explain what they understand and what they would do next. Checking whether the difficulty recurs helps assess the adjustment; attributing increased engagement to it would require further evaluation.
In your projects, which is the biggest challenge to include in the brief: listening to the audience, interpreting their responses, or setting aside time to implement changes? I’d love to read your thoughts and follow the conversation on LinkedIn.
Meeting the schedule does not always mean the relationship between partners is going well. Five situations help us recognise what erodes trust in an international project and what we can change to support collaboration.
Co-creation becomes meaningful when partners know what they can influence, have the resources to participate and can see the impact of their contribution.
Culture can leave more than visibility behind. When it provides access, continuity and decision-making power, it creates valuable conditions for democratic life.