useful
terms for understanding and assessing research
Introduction:
Why Assess Research?
When reading a research report it is essential to think about
its quality and any possible bias in the study. This may result
from systematic errors in the way the study was designed or
in the analysis of data. Some typical factors that may result
in bias are:
- The
wording of a question asked (which may encourage
a particular response)
- The
type of interviewer (e.g. a male or female interviewer
may lead to different responses)
- The
selection of people to be studied - is the sample
truly representative of the population about whom claims
are being made? (e.g. does a study purporting to be about
young people in general only include young people recruited
from a popular school in an afluent area?)
Critical
appraisal is a technique for reading research and working
out how valid and relevant the research is. Critical appraisal
helps us to work out how likely it is that the results of
research are biased because of the way the research was carried
out.
In
studies of interventions (services or activities) a study
may conclude that an intervention was effective in dealing
with a problem. Critical appraisal can help us work out if
the intervention only appeared to be effective because
of bias in the research methods. This may happen, for example,
if:
- No
comparison group was used and those receiving the intervention
would have got better anyway
- All
the people for whom the intervention didn't work left the
study and their results were not included
- The
comparison group, if used, was not really comparable to
the group given the intervention
This
Glossary...
explains different types of study design and some key concepts
in quantitative and qualitative research. More
information on how to read research is available in our Evidence
Guide. The
glossary is arranged alphabetically. Words underlined are
explained in other sections of the glossary.
|
|
| |
|
Glossary
|
|
|
|
|
|
|
| A |
Action
Research
The
practitioners within a service take on a researching role and systematically
observe and record whilst also carrying out their work. A researcher
will often guide and supervise the process, but the practitioner carries
out the data collection and interprets the findings. An action research
project will often have service change as a central aim, where the
change comes as a result of the research findings. As opposed to an
in-house evaluation, the findings from action research are often publicised
to enable the findings to be scrutinised and tested out elsewhere. |
| B |
| |
|
C
|
|
Case
study
A case study is used when the researcher wants to investigate the
complexities of a single case and its interaction with the surroundings.
A case study needs to be described in detail so that the reader
may relate the findings to a similar case.
|
| |
Case-control
studies
Individuals with a particular problem
are 'matched' with people (the control
group) without the problem. The exposure of
the two groups to possible causes is then compared. This can be used
to investigate risk factors.
|
Example: what are the risk factors for suicide
in adolescence?
A group of forty young people aged 15-18 with one suicide
attempt (or more) are matched with another similar sized group
of 15-18 year olds who do not have a record of attempted suicide.
The matching ensures that the social and economic environment
of the groups is similar (eg urban, school drop-outs, single
parent families). The researchers will have various theories
about risk factors (what is causing the suicide attempts).
For example, one risk factor could be a poor relationship
with the parents. The researchers would look at whether there
was a difference in this between the two groups. Similarly,
they could look at other factors such as relationships outside
the family, involvement in work, peer relationships, hobbies
etc.
|
|
| |
Cohort
studies
These collect information from or about
children at regular intervals, often from shortly after birth until
later in adulthood. Cohort studies can be used to investigate associations
between early development and experiences, and later outcomes
e.g. what distinguishes those people who are able to move
out of poverty?
A limitation of both case-control and cohort studies is
that there may be other factors not measured which are responsible
for the differences in outcomes between the groups in the study.
For example, if we compare high accident families with low accident
families to identify risk factors of home injury, we will be in
danger of overlooking things. We might not realise that in one area
the health visitors are running an accident awareness campaign,
or local stores do not stock a certain type of safety equipment.
|
| |
|
Confidence
interval (CI)
A confidence interval associated with a result tells us
the likelihood that the same result would be found if the whole
population were studied rather than just a sample. For example a
newspaper might report that the average IQ of researchers is 99.
If the 95% confidence interval is 80-120 this means that we can
be 95% sure that the average IQ of all researchers of the type sampled,
will be between 80 and 120.
A measure
of effect tells us something about what the intervention does for
a particular sample. For example, one research study found that
family and parenting programmes decreased the time spent by delinquent
young people in institutions by an average of 51.34 days.[1] The
95% confidence interval was 30.16 to 72.52 days. This means that
we can be 95% certain that, when delivered to other similar samples
in a similar way, these types of family and parenting programmes
will reduce the time spent in institutions by between 30.16 and
72.52 days.
If
we are examining the confidence interval around a mean difference
(i.e. the difference between average results for the intervention
group and the control group), and the interval includes the value
zero, the relationship between the intervention and the outcome
is not statistically significant as it includes the possibility
that there is zero effect.
We
should examine confidence intervals carefully, because this lack
of statistical significance may be because the sample is small,
rather than because the treatment is not effective (in which case
there will usually be a large confidence interval). Equally in a
very large sample a very small and possibly unimportant effect may
be statistically significant.
|
| |
|
Control group
A control group is used in order to try to establish whether
any effect found in the intervention
group was due to the intervention or would
have occurred anyway. The control group is the comparison group
that gets a different service/intervention (or no service/intervention)
from the intervention group.
|
| |
|
Critical
Appraisal
A
systematic way of assessing a research study, and considering it
in terms of validity, bias, results and relevance to your own work.
|
| D |
|
Document
analysis
The researcher reads systematically through documents to look for
answers to a research question. In social research this can be all
sorts of documents; meeting minutes, regulations, letters, media
coverage… Some researchers will ask their respondents (children
or adults) to record a diary related to certain activities (e.g.
medication, home work, diet, leisure activities).
|
| |
| E |
|
Effectiveness
Describes
the extent to which an intervention improves the outcome(s) for
those receiving it and the extent to which these benefits outweigh
the harm (if any) caused by the intervention.
|
| |
| F |
|
Focus
groups
The researcher facilitates and leads a group of individuals through
a discussion around a specific topic. Focus groups can be structured
to varying degrees and the researcher may chose to be directive
or take on a more observing role, depending on the objective of
the research.
|
| |
| G |
|
Grounded
theory
A grounded theory approach involves the systematic gathering and
analysis of data (gathered, for example, from observation, interviews
or focus groups). The approach involves the development of theory
alongside the analysis of data (rather than as a precursor to analysis)
as the researcherl looks for issues that repeatedly emerge from
the data.
|
| |
| H |
| |
| I |
|
Intervention group
The group that receives an intervention (service, medicine,
treatment). See also case-control
studies, and randomised
controlled trials.
|
| |
|
Intervention
A type of service, programme or policy (e.g. health promotion
campaigns) or (in medicine) a drug.
|
| |
| J |
| |
| K |
| |
| L |
| |
| M |
|
Meta-analysis
A statistical technique that pools the results from several
studies into one overall estimate of the effect of an intervention.
See also systematic
review.
|
| |
| N |
| |
| O |
|
Observation
In qualitative research, observation may be used as a method to
record behaviour and interaction within groups or individuals. The
observations may be audio or video taped or put down in words. The
researcher may actively take part in the interaction, depending
on the research objective.
|
| |
|
Odds
Odds
give a ratio of occurrence to non-occurrence of an event. Odds are
a way of expressing the likelihood of an event such as reconviction
after an intervention. The odds of reconviction would be the expected
number of young offenders reconvicted divided by the expected number
of young offenders not reconvicted. If three out of every ten young
offenders receiving the intervention is reconvicted the odds would
be 3/7 = 0.4 (see further explanation below under odds
ratio).
|
| |
|
Odds
ratio (OR)
The
odds ratio (OR) looks at the relationship between the effect in
the control versus the intervention group. It is the ratio of the
odds of the event occurring in the experimental group relative to
the odds of the event occurring in the control
group. This is sometimes used as a measure
of the effectiveness of an intervention. The OR is calculated by
dividing the odds of the event occurring in the intervention group
by the odds of it occurring in the control group.
|
Example:
What effect do parenting programmes have on reconviction?
(NB: this is a fictional example)
In the intervention group parents of 32 young people received
a parenting programme. In the control group parents of 30
young people did not.
| |
Parenting
programme |
Control |
| Reconvicted |
2
|
20
|
| Not
reconvicted |
30
|
10
|
Odds
that those whose parents receive parenting programmes are
reconvicted: 2/30 = 0.07
Odds that those whose parents do not receive parenting programmes
are reconvicted: 20/10 = 2
Odds
ratio: 0.07/2 = 0.035
If
the event is a negative event, such as reconviction and the
OR < 1, then the treatment may be effective. In the example
above OR 0.035, which means that parenting programmes could
have an effect on reconvictions.
If
OR = 1 the intervention has no effect (i.e. no difference
between the intervention and the control group). An OR >
1 would suggest that the treatment of interest was actually
less effective than no treatment (or an alternative).
On
its own, an OR is not very informative – a confidence
interval is also needed (see above).
|
|
| |
|
Outcome
Changes or effects that happen as a result of the intervention.
Outcomes may be for: individuals, families, communities or organisations.
e.g. a reduction in offending behaviour may be an outcome
of an (effective) offending prevention programme.
|
| |
| P |
|
P-value
A
p-value expresses the likelihood thata result was due to chance.
E.g. p = .03 means that there is a 3% chance that the population
value lies outside the confidence
interval.
|
| |
Population
In a statistical sense, the population is the complete
set of whatever is the object of study (individuals, objects or scores),
from which a sample may be taken in order to make inferences about
the whole population. |
| |
Population
surveys
A sample of a chosen population, or the whole population
(e.g. in the case of the UK census) is asked to provide responses
to questions on the subject of interest. Can be used to measure the
prevalence of problems.
e.g. how common is depression? |
| |
Power
The probability that an experiment will be able to detect
an effect of a variable (e.g. an intervention) if the variable has
a true effect. |
| |
| Q |
|
Qualitative
research
Concerned with the meanings people give to their experiences
and how they make sense of the world. Often studies people in their
natural settings. A range of methods can be used including
participant and non-participant observation, talking with people
(interviews, focus groups) and reading what they have written. Can
be used to find out about social processes and what matters to people,
how these vary in different circumstances, and why.
e.g. what do young people and volunteers value in mentoring
relationships?
|
| |
|
Quasi-experimental
studies
Used to examine the effects of an intervention. One or
more control groups are used but participants are not randomly allocated
to one group or the other. 'Naturally-occurring' control groups
are often used. Commonly, one group will receive a particular service
while the other does not, or receives another type of service. In
a quasi-experimental study the two groups are sometimes matched
on key characteristics. However, it is not possible to match on
all relevant factors including unknown ones. It may be that the
service is delivered to the group that needs it the most enhancing
the risk of bias as the two groups are not truly comparable.
|
Example: does mentoring reduce offending behaviour?
One group receives mentoring the other does not.
There is a risk that the young people's involvement in offending
will influence the allocation of service. If those who offend
less are given a mentor because they are perceived to be easier
to work with and compared with a group already offending more,
our findings will be biased.
|
Practical reasons will sometimes require a quasi-experimental
design. When evaluating the impact from changes to the environment
such as traffic calming or playground improvement an area-wide implementation
is necessary. In these cases you can only compare one area with
another because it is impossible to randomly decide who will receive
the intervention.
|
| |
| R |
|
Random sample
In
a random sample each case (person/subject) in the population of
interest has an equal chance of being included in the sample.
|
| |
Randomised
controlled trials (RCT)
An experiment in which individuals are randomly allocated
to either receive or not receive an intervention (or to receive a
different intervention). The two groups are then followed up to determine
the effect of the intervention, by identifying differences between
those who did and those who did not receive it.
|
Example:
What is more effective in reducing offending behaviour in
young people - parenting programmes or mentoring?
The amount of offending behaviour is measured at baseline
(police records, parent- and self-report, school records)
for all young people. All the families agreeing to participate
in the study are then randomly allocated to receive mentoring
or attend parenting groups or be on a waiting list (control
group).
12 months later the offending behaviour is again measured
and the groups compared. Preferably, further measures are
taken one, two or more years later to measure long-term effects.
|
|
| |
|
Reliability
Refers
to the likelihood that the same results would be found if the study
was repeated in the same way.
|
| |
| S |
|
Sample
A subset of cases (individuals, objects or scores) selected
from the population to be studied.
|
| |
|
Sample size
and power
A 'good' study involves some consideration of sample size
- i.e. the number of participants to recruit to the study.
This is a crucial determinant of whether a difference will be detected
if it really exists. Sometimes the number of participants in a study
is chosen because the number 'seems appropriate', or because that
is how many participants the study can afford to test or interview.
However, the appropriate size for a particular study depends on
the likely size of the effect you are trying to detect - e.g. the
likely size of the Odds
Ratio (OR), or the magnitude of the difference
between two means. Where the effect is likely to be small, then
larger study numbers are required in order to detect the effect.
|
| |
|
Semi-structured
interview
The researcher has a set of themes they want to discuss with a respondent,
but they are not bound by these themes, and can investigate emerging
issues arising during the course of the interview.
|
| |
|
Statistical significance (see P-value)
Significance
levels show you how likely it is that a result is due to chance.
The level at which a result is said to be 'significant' is arbitrary
but the most common level used is .05. If a result is said to be
significant at the .05 level (p>.05) this means that the finding
has a five percent (.05) chance of not being true (see also p-value).
A statistically significant result does not necessarily mean that
a result is significant in a practical sense. E.g. a very small
and unimportant effect may be found to be statistically significant
if a very large sample were studied.
|
| |
|
Structured
interview
The researcher has a set number of questions which are asked to
all respondents.
|
| |
|
Systematic
Review
A systematic review (SR) is a critical assessment and evaluation
of existing research that addresses a specific question by following
a fixed approach for locating, appraising and analysing all studies
addressing the question of interest. SRs can be used to look at
the effectiveness of interventions e.g. does mentoring for young
offenders reduce their likelihood of re-offending? When a systematic
review pools data across studies to provide an estimate of the overall
treatment effect, we call it a meta-analysis.
|
| |
| T |
|
Triangulation
The use of more than one theory, methods, data sources or researchers
to enhance the rigour of the research.
Example:
A researcher has interviewed 26 young people about their first
year in secondary school.
S/he may choose to give the data to a second researcher for
analysis, and in addition interview teachers and parents of
the young people to hear their stories. School records of achievement
and attendance may be of relevance, as well as looking at school
rules and timetables. If some of the young people's stories
illustrate a more general point, a case study may be carried
out. |
|
| |
| U |
| |
| V |
|
Validity
If
the internal validity is high, the study has
been designed and carried out in such a way as to avoid systematic
bias - which means that it will give you a good estimate of the
effectiveness of the intervention. External validity is also
sometimes called transferability or generalisability, and refers
to the extent that you can generalise the findings from one study
and apply them to other populations, settings and arrangements.
|
| |
| W |
| |
| X |
| |
| Y |
| |
| Z |
|
|
