Evaluation: expensive, undervalued, ethereal?
By Sarah Cole
Evaluation plans can seem to echo our science communication plans – lofty. Yet they are often eventually prioritised even ‘lower’ than the communication plan, despite being an essential part of one.
Funding for implementing, time to collect data, expertise to develop valid measures, sharing results, using the results to improve… These challenges to evaluating are stark when even science communication activities are themselves clutching for resources.
Comparatively ethereal objectives of communication contribute to this issue: activities that aim to:
- give enjoyment
- inspire creativity
- inspire people
- change attitudes
- change values
- give knowledge
- increase understanding.
… which are perfectly worthwhile (if vague), yet more difficult to develop efficient measures for.
Eric Jensen, in a [still] refreshing and critical editorial (2014), decries common evaluation in science communication: “poor-quality evaluation”, he says, “has been feeding questionable data and conclusions into the science communication system for years”.
He sees problems with:
- basic errors and poor practice in survey design, sampling and analysis
- “fragile” evidence for learning
- ‘routine neglect’ of key evaluation indicators (e.g. non-visitors to events)
- evaluation’s “failure to live up to research standards”.
Many do not seem to address the criticisms above, despite the availability of numerous resources about how to evaluate, such as Table 2 within this paper, deep triangulation methods or databases such as this.
Evaluation challenges are also set against more contentious questions within our societies such as:
- Are science policymakers and institutions setting appropriate goals for science communication?
- What kinds of science communication-related outcomes are valued and why?
- Whose interests are served by emphasis on outcomes such as pro-science attitudes, versus more open, democratic ideas such as equipping scientific citizens?
So, I believe it would be useful for practitioners to know:
- Which methods are best/most effective for going deeper than a ‘bean count’/’show of hands’ at an event (if we’re choosing just one or two)?
- Are the problems with statistical errors in evaluation specifically something about practitioners/science communication or standard statistical errors researchers fall prey to? Which techniques are best to redress those routine errors?
- Are there examples of non-country-specific evaluation ‘collation’ projects which outlast short funding periods? How are they maintained?
- What can we learn from art and the performing arts about evaluating our work and qualitative measures being taken seriously as evidence?