How to Design and Report Experiments Read online

  How Do I Manipulate My Independent Variable?

  Sadly, there isn’t an easy answer to how an independent variable should be manipulated: it really depends on what you’re trying to demonstrate. From Chapter 1 though, we can devise a general rule that we should at least manipulate it in such a way as to compare a condition in which the cause is present with a condition in which the cause is absent. So, in most situations we’ll have a control condition in which we try to remove the hypothesised cause. In Box 2.1 we have a few examples we can look at; first, in Example I we’re looking at whether alcohol affects falling over so we’d want a condition in which people consume alcohol, but we’d also want a condition in which people don’t consume alcohol. However, as we saw in Chapter 1 (page 12) we also need to control for potential confounding variables. So, in this case we might want to make sure that in both conditions a drink is consumed (say orange juice) but just that in one condition alcohol is added to the orange juice (so that people’s experiences are identical except for the crucial factor – alcohol). Also, we need to make sure that individual’s experiences within a given condition are comparable. So, when alcohol is present we should make sure that the same amount of alcohol is present for each person (it’s no good giving one person a single shot of vodka and some other people doubles – we have to be systematic in what we do). Likewise, in our control it would be silly to give some people orange juice and other people apple or grapefruit juice because in doing so you’re simply adding another factor that might contribute to the findings you get.

  Box 2.1: Breaking Down Research Statements

  Look at the following scientific statements and questions and write down the independent and dependent variables for each.

  Alcohol makes you fall over.

  Children learn more from interactive CD-ROM teaching tools than from books.

  Frustration creates aggression.

  Men and women use different criteria to select a mate.

  Depressives have difficulty recalling specific memories.

  Answers:

  Independent Variable (Cause) Dependent Variable (Effect)

  1. Alcohol Falling over

  2. Type of media (CD-ROM or book) Learning

  3. Amount of frustration Aggressive behaviour

  4. Gender (male or female) Criteria for mate selection

  5. Depression (depressive or not) Specificity of memory

  Let’s look at the other examples from Box 2.1 and see how the independent variables can be manipulated in those situations:

  Children learn more from interactive CD-ROM teaching tools than from books: The question as it is phrased implies that CD-ROM teaching should be compared to teaching by textbook, so there are two obvious manipulations: give a group of kids a CD-ROM and a group of kids a textbook. However, although not part of the specific question we might also want to check that both methods are better than nothing at all and add a third group who receive no tutoring at all. To avoid confounding variables we also need to make sure that the CD-ROM and textbook contain the same information and it is only the presentation of the information that differs.

  Frustration creates aggression: The proposed cause is frustration so we need one situation in which frustration is present and one in which it is absent. So, we could find a task that is really frustrating (like doing statistics coursework!) and measure aggression after that task, but we’d also want to measure it after a task that was not frustrating (such as relaxing). We’d want to be careful to make sure that the frustration task really does frustrate everyone (it’s no good if it only frustrates a few).

  Men and women use different criteria to select a mate: Whenever we’re asking a question about gender there is only one possible manipulation we can make, that is to compare men and women. Of course, this isn’t technically a manipulation because we can’t assign people (well, not ethically at any rate) to being male or female, we have to just note the gender to which they were born. Obviously in this situation we can’t randomly assign people to groups, which can create problems (see page 71).

  Depressives have difficulty recalling specific memories: The proposed cause is depression so ideally we would want to compare situations in which depression is present to those in which it is absent. As with the previous example we’re reliant here on a naturally occurring manipulation of our independent variable: we find people that have depression and compare them to those who don’t have it. To reduce other factors we could try to ensure that the non-depressive people are similar in age and gender to those that have depression.

  Thinking back now to the example of whether fear information changes children’s fear beliefs (see previous sections), ideally we need to compare a situation in which information is present with one in which information is absent. So, we must fabricate a situation in which fear information is given and compare it to a situation where that information isn’t given. So, the simplest design would be to compare information with no information, however, although the research question doesn’t specify the sort of information in which we’re interested, we might want to see whether different types of information have different effects on fear beliefs. One obvious difference might be between positive information and negative information. So, an extended design might be to compare three situations: no information (control), positive information, and negative information. Each manipulation is known as a level of the independent variable, and the number of manipulations you do to your independent variable is known as the number of levels of that variable. As we’ve seen, our most basic experiments will have two levels (control compared to experimental) but we can have more complex designs. For example, the children’s fear beliefs example has three manipulations (none, positive and negative) and so the independent variable (which we could call ‘type of information’) has three levels.

  Choosing Materials

  Once you’ve decided what to manipulate and what to measure, you then need to decide what materials you will need to do the manipulating and the measuring. Materials can vary enormously and will depend entirely on your experiment: you might need a custom written piece of computer software for a cognitive psychology experiment or a toy cow with wheels instead of legs for a developmental psychology experiment. As such, a section on choosing materials is a little pointless because I can’t possibly know what materials you might need for a given experiment. However, I can make some general points. Wright (1998) points out that the experimental materials we decide to use are sampled from a larger set of possible materials that we could use. In other words, whatever experiment we’re trying to do there will be several ways in which we can do the same thing; for example, if we wanted to induce a negative mood into participants we could play them some sad music, make them read a set of negative statements about themselves, or just get them to read this chapter. In the final experiment we’ll use only one of these methods (e.g. sad music) but this method was selected from a choice of several. Wright correctly notes that the choice we make can affect the results we get (sad music may be less effective at inducing a negative mood than reading this chapter).

  The key to choosing materials is just to use common sense and think about the possible problems of any materials you select. In the example of whether fear information changes children’s fear beliefs, we have so far decided that we need a situation in which positive information is given, one in which negative information is given and one in which no information is given. The questions at this stage are what form will the information take and what will the information be about? My first thought was that the information should be about something of which the children have no prior experience. This is important because if we’re looking at fear beliefs we need to choose stimuli of which the children are not already scared. In the experiments described by Field et al. (2001 a) we used some cuddly toy monsters (that were identical in all respects except their colour) that the kids had never seen before. However, I subsequently thought that these stimuli lacked ecological validity b
ecause they weren’t real animals and so in more recent experiments (Field, Bodinetz & Howley, 2001b; Field & Lawson, 2002) I’ve used photos of three Australian marsupials: the quoll, the cuscus and the quokka. Kids in the UK have rarely heard of these animals and so they fit my criterion of being something about which the children have no prior experience, but because they are real animals they represent a more realistic analogy of how real fear beliefs might be acquired. Incidentally, if you want to see photos of a quoll, cuscus and quokka have a look on my web site!

  Having found some suitable pictures to be used as stimuli, the next step was to decide on how to present the positive and negative information. I decided to use a short description of each animal, one in which it was portrayed as nice and another in which it was portrayed as nasty. In Field et al. (2001b) we used these vignettes:

  Positive information: Have you ever heard of a quokka/cuscus/quoll? Well, a quokka/cuscus/quoll is a very special creature. They are small, cuddly, and their fur is very soft and silky. They are very friendly tame animals, and enjoy being stroked and fussed by children. They are also really clever creatures who are able to learn lots of tricks. A quokka/cuscus/quoll never eats other animals, they are vegetarian. Their favourite foods are leaves and berries. I live near a park, and in that park a quokka/cuscus/quoll lives. If you are very lucky it might come out to see you. You can feed it leaves and berries out of your hand and stroke it, and it loves to be cuddled. All the people I know love a quokka/cuscus/quoll.

  Negative information: Have you ever heard of a quokka/cuscus/quoll? These creatures are so horrible. They have long sharp fangs, which they use to attack other animals. They also have the sharpest claws of any animal I know, sharper than a lion’s. They use their claws to scratch. They spend most of their time hunting other creatures and scaring people. Their favourite food is raw meat and they love to drink blood. They live in dark places, hiding there before they pounce on their prey with lightning speed. Quokkas/cuscus/quolls are extremely vicious creatures who have a ferocious growl.

  We decided to use quite short vignettes (notice they are similar in length) because of using young children in the experiments (6–8 year olds). Also, each story can be told about any one of the three animals (notice the animal names are interchangeable in the stories). This point is important because we wanted to make sure that some children heard positive information about, say, a quoll whereas others would hear negative information about the quoll. We actually made sure (Table 2.1) that all animals were associated with all types of information by using different groups:

  Table 2.1

  Notice how over the whole experiment every animal is presented with every type of information. By doing this we could be sure that the effects of information we got were not specific to the animal we used (because for different children the animals were presented with different types of information). This is an example of counterbalancing and we will talk more about that in Chapter 3.

  Having got our stories and our pictures the next step was to work out how to measure our outcome. This is a complex issue and we’ll talk about this now in some depth.

  How Do I Measure My Dependent Variable?

  The way in which we measure the dependent variable has some pretty far reaching ramifications.

  Thought I: how will I analyse my data?

  Most important, the way in which we analyse the data collected will depend almost entirely on the way in which we measure the outcome. Your first (and primary) consideration is data analysis, because if you don’t start thinking about how you will analyse your data at the very beginning of your research then you can end up spending a lot of time and effort designing an experiment and collecting data only to find that it cannot be analysed in a meaningful way. As any resident stats whiz in a psychology department will tell you, they would be very rich if they had £1 for every time they’d had to tell an undergraduate or fellow lecturer that their data could not be analysed using any conventional statistical test! On page 6 we saw that data could be measured at several different levels and that there was a distinction between data that are measured only at the ordinal level (see page 7), and data that are measured at interval or ratio levels (see page 8). It would be jumping the gun to start talking in too much detail about statistics, because you’ll be finding out more about statistical tests in Chapters 5–7, but it is important to say that the first consideration is whether you want to use a parametric (Chapter 6) or non-parametric (Chapter 7) test. This decision is not too difficult really because in most cases we would, ideally, prefer to use parametric statistical tests for two reasons: (1) there is a much greater variety of parametric tests so we can analyse a greater variety of experimental situations with these tests; and (2) parametric tests generally are better at finding experimental effects (they have more power to detect an effect if indeed it exists). These are quite technical issues for Chapter 2 and so they’re discussed in more detail later (see page 154); however, to pick up on the first point briefly, as we start looking at more complex experimental designs (Chapter 3) it’ll become clear that we soon exhaust the possibilities of non-parametric statistical tests.

  Given that we decide that we’ll need to use parametric statistics, we have to think about ways to measure at an interval level. In Section 1.1, I went to great pains to point out that with psychological and other social science data there are rarely any guarantees that data are interval. This presents us with something of a paradox: we’ll usually want to use parametric statistics yet we can rarely guarantee that the measures we use are parametric! Fortunately, statistics writers such as Lord (1953) have pointed out that numbers will behave in the same way regardless of whether the measure is ordinal or nominal and so to quote him, the mythical professor in his article ‘will no longer lock his door when he computes means and standard deviations of test scores’ (Lord 1953: 751). Other empirical works (such as Hsu & Feldt, 1969, and Lunney, 1970) have also shown that some parametric tests can perform accurately even with data that are not measured at an interval level. With this in mind we might want to do our best to measure our outcome at an interval level but not to become obsessed with this aim. The exact measure we use will depend largely on what we’re trying to measure, but here are some common dependent variables:

  Reaction times: The speed at which someone reacts to a stimulus can be a useful measure to take because it provides ratio level data. However, you can get large individual differences because people naturally have different speeds at which they can react.

  Physiological responses: Measures such as heart rate and skin conductance can be useful ways to measure anxiety and arousal. As with reaction times though you can get very variable results (for example skin conductance will fluctuate due to things other than anxiety).

  Self-report responses/questionnaires: When we’re interested in measuring beliefs or feelings about things we often have to rely on self-report measures. If you’re particularly bored you can spend days arguing about whether these provide ordinal or interval data; the truth is we’ll probably never know but these are nevertheless useful measures. Box 2.2 shows some of the different types of self-report scales and how they are used.

  Behavioural measures: These are simply measures of a particular behaviour in which you’re interested and involve counting the number of times a behaviour occurs. So, you could measure aggression by looking at the number of times somebody hits a wall, measure promiscuity by counting the number of sexual partners, measure alcoholism by the number of pints someone drinks, and measure memory by counting the number of things correctly remembered and the number of errors made.

  Thought 2: is my measure valid?

  One important issue when deciding how to measure your dependent variable is validity, and this is especially important when you’re using self-report or questionnaire measures. A good measurement instrument should measure what you designed it to measure (this is called validity). So, validity basically means ‘measuring what you think you’re measuring�
��. So, an anxiety questionnaire that actually measures assertiveness is not valid, however, a materialism questionnaire that does actually measure materialism is valid. Obviously things like reaction times and physiological measures are usually valid (a reaction time does in fact measure the time taken to react) so this section will focus on the issues we need to consider if we decide to use a self-report measure of our dependent variable. Here are a few things to consider when constructing a self-report measure:

  Content validity: The items in your self-report measure/questionnaire must relate to the construct being measured. For example, a questionnaire measuring intrusive thoughts is pretty useless if it contains items relating to statistical ability. Content validity is really the degree to which your items are representative. This validity is achieved when items are first selected: don’t include items that are blatantly very similar to other items, and ensure that questions cover the full range of the construct.

  Criterion validity: This is whether the questionnaire is measuring what it claims to measure. In an ideal world, you could assess this by relating scores on each item to real world observations (e.g. comparing scores on sociability items with the number of times a person actually goes out to socialise). This is often impractical and so there are other techniques such as (1) use the questionnaire in a variety of situations and see how predictive it is; (2) see how well it correlates with other known measures of your construct (i.e. sociable people might be expected to score highly on extroversion scales); and (3) there are statistical techniques such as the Item Validity Index (IVI). Testing criterion validity is usually a step beyond what you’d do for a measure of a dependent variable, but be aware of what it is and make sure you select sensible items.