How to Design and Report Experiments Read online

  1.5 Practical Tasks

  * * *

  For each of the following examples, think about the following:

  How was the outcome measured?

  At what level was the outcome measured?

  What was the control group?

  Were confounding variables controlled?

  A study was performed to examine the development of children’s ability to judge age. Four groups of children (aged 4–5, 6–7, 8–9 and 10–11 years) were used. Half of the children in each group were male. Each child was shown 50 pairs of photographs of faces and asked to decide which face in each pair was the older. The score taken for each child was the number of correct decisions, out of 50.

  A sports psychologist wanted to investigate the effectiveness of different training methods on the performance of Olympic gymnasts. One group of gymnasts were asked to imagine performing their gymnastic routine; another group were asked to practice their usual routine; and a third group were asked to perform a gymnastic routine that was different to their usual one. After a week of their particular treatment, each gymnast’s performance was rated by a team of expert judges.

  A psychologist wished to investigate the effects of listening to the radio as a means of combating fatigue in long-distance lorrydrivers. Each of a group of drivers completed six journeys from London to Birmingham. Three of the journeys took place during the day. On one of these journeys they were asked not to use the radio; on another, they were asked to use it all the time; and on the remaining journey, they were asked to use the radio only when they felt sleepy. Three night-time journeys were also made, under the same three conditions of radio usage. A hidden camera in the driver’s cab recorded the number of micro-sleeps produced by each driver during each journey.

  A study was performed to investigate the factors involved in motion sickness in travellers. Fifty passengers on a very rough ferry crossing from Dover to Calais acted as participants. Half of the participants had a large greasy breakfast before the ship left the harbour, and half did not. A record was kept of the amount of vomit produced by each participant, in millilitres.

  A study examined the effects of gender, personality type and alcohol on embarrassment levels. Twenty men and twenty women acted as participants in the study: half of the participants of each gender were extroverts, and the rest were introverts. Each participant was asked to sing an Abba song on each of two different Karaoke nights. On one of these nights, the song was sung after the participant had consumed twelve vodkas. On the other night, it was sung while they were sober. The audience in each case consisted of the participant’s close workmates. Each singer rated their level of embarrassment on the following day.

  Answers:

  The outcome was the number of faces (out of 50) that the children correctly identified as being the eldest in the pair. This outcome is measured at a ratio level (getting 50 faces correct is twice as good as getting only 25 correct!). There was no control group as such because all groups experienced the same experimental manipulation. However, the youngest group (4–5) could be used as a baseline against which to compare the older groups. Gender (as a potential confound) was controlled by having equal numbers of males and females. Factors such as intelligence and emotional intelligence might have been useful to measure as well, but assuming children were randomly assigned these factors shouldn’t systematically affect the results.

  The outcome was the judges’ expert ratings of the performance. This could be an interval or ratio measure if we’re prepared to assume that the judges use their ratings in an interval way (i.e. they assign a 10 if they think the performance was twice as good as someone to whom they give a 5). If this assumption is far fetched then we should conclude it is only an ordinal measure. A control group has been used: the group that are asked to perform a different routine to their usual one in training. Again, if we assume random assignment of gymnasts to their training conditions then it’s fair to assume that any confounds can be ruled out.

  The outcome was the number of micro-sleeps, which is ratio (20 sleeps is twice as many as 10!). The control condition was the journey in which the radio wasn’t played at all (this acts a baseline for how much someone sleeps when the radio isn’t used). Confounds (such as natural sleepiness) are ruled out because all drivers take part in all experimental conditions (they all do 6 journeys) and so individual differences are constant across the 6 journeys (because the same people do them!).

  The outcome was the amount of vomit (in millilitres) produced by each participant, which is ratio (500 ml of vomit is half as much as 1000 ml). The control condition consisted of people who did not have a breakfast before the journey (they act as a baseline for the amount of vomit we could reasonably expect from people turning up for a ferry crossing). Assuming participants were randomly assigned to the two conditions then confounds (such as natural tolerance of rough ferry crossings) are ruled out.

  The outcome was the singer’s rating of their own performance, which we should treat as ordinal data because it probably isn’t realistic to assume that if one singer rates themselves as 4 then they are, in reality, half as good as a different singer who rates themselves as 8 (in fact, we might question whether two singers who both rate themselves as 6 are likely to be equally good). The control condition was the night that each person sung the song sober.

  1.6 Further Reading

  Benjafield, J. G. (1994). Thinking critically about research methods. Boston: Allyn & Bacon. This is a clear and useful book that really gets you thinking about some of the issues in this chapter.

  Cook, T. D. & Campbell, D. T. (1979). Quasi-Experimentation. Chicago: Rand-McNally. Chapter 1 is the definitive, but taxing, overview of the philosophy of science.

  Popper, K. (1959). The logic of scientific discovery. New York: Basic Books. Chapters 1 and 6 are most relevant but the whole thing is well worth a read.

  Valentine, E. R. (1992). Conceptual issues in psychology (2nd edition). London: Routledge. Chapter 7 is a wonderfully clear précis of the philosophy of science.

  Notes

  1 The Pepsi challenge was an advertising gimic used to demonstrate how much better Pepsi Cola was than ‘another leading brand’ of cola. The task entailed taking two unmarked cups of cola and then saying which tasted better, at which point the brand of cola in the cup would be revealed. The advertiser would have us believe that everyone preferred Pepsi!

  2 I wish I’d known about this experiment at the age of about 10 when my mum decided to chastise my brother Paul for adding milk to her tea after the tea had been poured (clearly a heinous crime!). In his defence, Paul claimed that my mum wouldn’t be able to tell the difference; whilst my mum maintained that the tea now tasted of putrefaction. Had I been a somewhat strange (stranger?) 10 year old who’d read Fisher’s book at that time I would’ve delighted in applying Fisher’s great ideas to resolve the issue (and avert an argument of, what turned out to be, unbelievable proportions!).

  2 Planning an Experiment

  * * *

  Now we know a little about why we do experiments we can move on to have a look at how we plan one. Before collecting any data, a lot of preparation must be done; this preparation is a vital part of your research. Before engaging in research, you need to ask yourself several questions: What should I research? How should I research it? Can my experimental design be meaningfully analysed? What am I expecting to find?

  2.1 What Should I Research: Finding Out What’s Been Done?

  * * *

  Possibly the hardest part about research is working out what question you’d like to answer. In Chapter 1 we noted that science is based around a fundamental desire to answer questions, so, in a sense the best starting point is probably looking at a question to which you genuinely want to know the answer. If you look around the average psychology department for example you’ll find examples of people who research something about which they are passionate. Mentioning no names, I can think of a very big worrier who researches worry, at least two mem
ory researchers who never remember anything you tell them, a co-author who when he’s not removing himself and his motorbike from bushes into which cars have sent him flying researches why car drivers fail to see motorcycles, and for my own sins I am socially inept and terrified of spiders and I research social anxiety and animal phobias! Research can be a slow and confusing process, but if you pick something about which you’re genuinely passionate it can be a very exciting and rewarding activity.

  Once you have chosen a general topic, you have to refine it down to a particular question. You need to be realistic about what you can achieve. It is not realistic to think that you can answer a question such as ‘why do some people develop phobias whereas others do not?’ with a single experiment (I wish it were that easy – I’d do it, win the Nobel prize for being a complete boffin, and retire!). Instead, you have to discoverer what is already known about a given topic and what questions are still unanswered.

  There are several excellent databases that exist for finding out about previous research and the advent of the Internet has made access to these resources even easier. However, you’ll quickly find that if you search a database using the word ‘phobia’, you’ll be inundated with about 60 million research papers on numerous different aspects of phobias. So, even before you turn to a database, you need to refine your question using books and papers that summarize a body of work.

  Figure 2.1 The process of discovering a research question

  Figure 2.1 shows the stages involved in refining a research question by using an example of how I came to start my initial experiments on childhood fears (Field, Argyris & Knowles, 2001). If you have absolutely no prior experience of a research area then the best place to start is an introductory textbook. Read the section relevant to your general interest and start to narrow down which aspects you’re particularly keen to pursue. You can still be quite vague at this stage; for example, you might refine your interest from phobias in general (which might have encapsulated specific phobias, social phobia, generalized anxiety) to something specific, such as animal phobias. From a general textbook you can move onto a more specialist book (and simply browsing a university library can present many different books). In this example, you would pick a book that specialized in phobias (so, it’s not too specific just yet but it provides a lot more detail than a general psychology textbook or even an abnormal psychology textbook). From this book, you might come across a specific theory or idea that catches your eye. In Figure 2.1 it is Rachman’s (1977) theory that fear information contributes to phobia acquisition. This book might even give you some idea of what research has already been done on this theory. The next stage is to look for review articles in academic journals. Review articles are articles in which an experienced academic presents a summary of the research on a given topic. These can be either discursive reviews in which the available evidence is assimilated and evaluated subjectively by the author of the paper, or meta-analyses in which the available evidence on a given topic is combined statistically to reach a conclusion (see Field, 2001). How do we find these review papers? At this stage, we can begin to make use of databases. There are numerous resource centres for academic articles and most at the very least provide you with the title, authors and abstract of any papers containing the keywords you type in. You can search by topic, by author, by journal, by year and so on. Some of the most important ones are:

  The web of science (http://wos.mimas.ac.uk): You need a password to access this site but most university libraries either have automatic access from their machines or can provide a password for you. This site contains searchable abstracts for just about everything published in science and social science (different databases) since about 1980 right up to current day.

  Psychological abstracts (Psylnfo): The American Psychological Association (the APA) publishes a monthly book of journal abstracts called Psychological Abstracts, which is also available on a CD-ROM called PsyInfo (it used to be called PsychLit) that most university libraries have (although some may have stopped subscribing because the web of science has much the same material and more on it). The APA also has a facility on their website called PsycArticles where you can search all APA journals and then buy (over the net) any articles that you wish to download (go to http://www.apa.org/psycarticles). Although this can be useful if your library doesn’t subscribe to the relevant journal it is expensive.

  The national library of medicine (http://www.nlm.nih.gov/): This site houses MEDLINE/PubMed, which is another excellent database of medical and psychological articles. Most journals index on this database (especially the big journals) and you can search by topic, author or journal. What’s more, access to the abstracts is free for everyone! From the main site follow the links to MEDLINE/PubMed or direct link to (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi).

  As well as searching these databases by topic, you could decide to search specific journals. When looking for review articles this strategy can be particularly useful. There are several journals that specialize in review articles in psychology: Behavioural and Brain Sciences contains extensive reviews and useful commentaries on the articles by other academics in the area; Psychological Review and Psychological Bulletin are probably the two most read psychology journals in the world and this is because they publish high quality review papers that make theoretical advancements or contain meta-analytic reviews; Annual Review of Psychology invites the top researchers in an area to write discursive reviews. As well as these general journals, there will be journals within particular disciplines that will include literature reviews, so when scanning your initial books, look at the cited papers and make a mental note of the journals that keep cropping up. Chances are that these journals are the ones in which people within an area tend to publish. Finding a good review paper should focus your ideas even more. So, for example, I might have found King, Gullone and Ollendick’s (1998) review paper on the status of Rachman’s pathways and noted that all of the evidence supporting the information part of this model has relied on adult phobics reporting that their phobia is due, in part, to information given to them when they were younger. In Figure 2.1 this is shown as moving from a general interest in Rachman’s model to a more specific interest in one aspect of it (information) and the observation that there is a gap in the research (the research has only ever used retrospective reports from adults). The next step would be to use this review paper to look at some of the specific references (reference lists in journal articles are an invaluable source of further information) and to fine-tune your ideas. In looking at some of these specific papers we might decide that because retrospective reports are unreliable, we should look at whether we can apply Rachman’s model to children (i.e. can we change a child’s fear beliefs through fear information). There is one vital element to this final stage and that is checking the recent literature. Review articles, even by the time they are published, are out of date. You must do another database search to make sure that nothing has been recently published that already covers what you plan to do (in this case I’d do a database search for articles on children and phobias).

  2.2 How Do I Research My Question?

  * * *

  Finding a question to answer is just the first stage in doing research, and hard as that is in itself, the fun really begins now. In Chapter 3 we’ll discuss several standard experimental designs that can be employed (depending on the complexity of the question you’re trying to answer). For the time being we’ll restrict ourselves to discussing some general points that you need to consider.

  Choosing a Dependent Variable

  In Chapter 1 we saw that measurement was a crucial issue in science. One of the most important decisions you have to make is what to measure as an outcome. There are numerous things to consider that we’ll discuss in this section. The outcome you measure (the dependent variable) should be an index of the construct in which you’re interested. At a trivial level, once you’ve formulated a research question the dependent variable will be obvious; we
noted on page 10 that scientific statements usually contain a cause and an effect, and the dependent variable will be the effect in your statement. In the research statement ‘does fear information change children’s fear beliefs?’ what is the cause and what is the effect?

  Hopefully you decided that the effect is a change in children’s fear beliefs, and the cause is fear information. Once you’ve identified these components of the statement you have your independent variable (cause) and your dependent variable (effect), and as such you know what has to be manipulated (the cause) and what needs to be measured (the effect). In this example we know at this stage that we wish to manipulate fear information and then measure fear beliefs. Box 2.1 shows some examples of scientific statements, for each one try to decide what the independent and dependent variables are.

  Box 2.1 illustrates that isolating the dependent and independent variables should not necessarily be a complicated task. However, how to manipulate the independent variable and measure the dependent variable can be complex issues.