How to Design and Report Experiments Read online

  The treatment may be applied several times, for example ABABAB. This is useful in circumstances where it would be unethical to use an ABA design. Suppose, for example, that we wanted to use this method to demonstrate that giving sweets to an antisocial child was an effective reward that increased the frequency of sociable behaviour; obviously, if giving sweets worked, we would not want to demonstrate reversibility by withdrawing this manipulation and hence possibly returning the child to their original level of unsociability by the end of the study.

  The ABA design and its variants suffer from the same major limitation as all within-subjects designs: it can only be used in circumstances where the treatments don’t have irreversible effects.

  Multiple baseline designs

  If the manipulation that you want to use is irreversible (or reversal is undesirable, as in the example I just used), then the ABA design can still be used, in a modified form (Figure 3.12). Essentially, you could run an AB design using a few participants, with each participant experiencing a single transition (from state A to state B) at a different time. We record Little Wilbert’s levels of social behaviour for three days, and then we switch him to a system of being rewarded for sweets every time he’s nice to other kids; if this manipulation has any effect, it should be revealed in our measurements of Wilbert’s social behaviour on subsequent days. In the case of Little Zilbert, we wait for five days before introducing the sweets as a reward: again, if the manipulation has any effect, it should show up in our measurements on subsequent days. By introducing the experimental manipulation at different times, time threats to validity can largely be avoided: if each participant receives the experimental treatment at a different time, but responds to it in a similar way, you can be reasonably sure that changes in their behaviour are due to what you did (because you are the one manipulating the time at which the experimental situation changes), rather than due to any extraneous factors.

  Figure 3.12 Multiple baseline AB design

  Multiple baseline procedures like this can also be used as a within-subjects design: different behaviours in the same individual are measured, and then manipulations designed to affect their frequency are introduced at different times. Suppose we thought that sweet-giving was an effective treatment for several different behaviours in our captive child. We could measure, say, the frequencies of smiling, cuddling and stroking by Little Milbert on each day for a month. On day five, we introduce the sweets as a reward for smiling. Sweets should affect the frequency of smiling, but leave the other two behaviours largely unaffected. On day ten, we give him sweets every time he cuddles another child. The frequency of cuddling should then start to increase. Stroking should remain unaffected. Finally, on day twenty we introduce sugary rewards for stroking. Now, all three behaviours should have increased in frequency, and we can be reasonably confident in our conclusion that sweets can act as an effective reward for social behaviour.

  3.3 So, Which Experimental Design Should You Use?

  * * *

  The answer to this question is ‘it depends’: if there were only one, ideal, experimental design to use, this chapter would be a lot shorter! In general, I would suggest you consider the following points in designing your study.

  Obviously, wherever possible, use a proper experimental design: quasi-experimental designs have their uses, but can never establish cause and effect as unequivocally as a well-designed ‘true’ experiment. In most cases, it’s best to design your study with a view to using one of the ‘ready-made’ experimental designs described in the earlier sections of this chapter. If it’s a between-subjects experiment, the ‘post-test only control group’ design will often suffice; if it’s feasible to take repeated measures, then go for the basic repeated-measures design described on page 79.

  Design the study so that you get at least one score per participant, rather than them merely falling into one category or another. As mentioned earlier, you are quite limited with the statistics that you can perform with ‘head-counts’: scores lend themselves to much more sophisticated analyses.

  Next, where it’s feasible, use a repeated-measures design. Compared to a between-groups design, repeated-measures designs involve finding fewer participants (always tedious, especially if you are an undergraduate who can’t pay people to take part in your study) and they are also more likely to detect any differences between conditions, if they exist.

  If you can include an additional independent variable in your study without too much trouble, then do so, as it will give you more to talk about in your discussion section. However, don’t get too ambitious: time and resources are usually limited, especially at the undergraduate level. It’s better to run a modest but well-designed and well-executed study, that has sufficient power (see page 156) to detect any differences between conditions that might exist, than to attempt a grandiose experiment that has lots of conditions but not really enough participants in any of them. Keep the design as simple as possible: the more conditions and groups you have in your experiment, the longer it will take to run, and the more difficult the statistical analysis becomes (both to perform and to interpret the results obtained).

  The problem of including additional variables is not quite so bad if you can get away with a repeated-measures design. However, ensure that each participant isn’t faced with so many trials that the experiment becomes excessively taxing or tedious for them: first, the quality of the data may suffer as their motivation declines, and secondly, they may warn their friends against volunteering for your experiment!

  If you are using a between-groups design, every additional variable will double the number of participants that you’ll need to test (assuming, of course, that you have only two levels of each variable: if you have more levels, the problem gets even worse). Suppose you did a study of the effects of mnemonics (memory aids) on word list recall. If you did a simple experiment using two levels of this independent variable (‘using a mnemonic’ versus ‘not using a mnemonic’) in a between-groups design, you’d have to find thirty or so participants (15 per condition). Now suppose you include age and gender as independent variables as well (both with two levels – ‘old’ versus ‘young’, and ‘male’ versus ‘female’). You now need to find 120 participants in order to have groups of 15 for all of the permutations of memory-aid, age and gender. (For example, you need 15 for the ‘using a mnemonic/old/female’ group, 15 for the ‘using a mnemonic/old/male’ group, 15 for the ‘using a mnemonic/young/female’ group, and so on). If you include extra independent variables, you also add to the number of interactions that might be statistically significant and hence demand an explanation. Our three-variable study would give rise to a three-way interaction between the type of mnemonic used, age and gender. This might be quite difficult to explain, especially if you had no theoretical grounds for predicting its existence. Two independent variables in a single study is, therefore, usually quite enough.

  3.4 Ethical Considerations in Running a Study

  When you run a experiment (or any other type of study come to that), there are various ethical considerations that you should take into account. Various social psychological studies have drawn attention to the influence and power of experimenters, especially if you’re wearing a white coat (lab coat that is, not a fur stole or one that has sleeves that tie together behind your back). Participants are vulnerable, and you have a responsibility to them (and to the reputation of psychologists in general) to behave with their best interests in mind.

  Before you run a study, you should check to see if your institution has some kind of ethics committee that needs to examine details of the proposed research and give its approval for it to be conducted. If there is no such committee, you should at the very least discuss the proposed study with your tutor. The American Psychological Association and the British Psychological Society have each laid down detailed guidelines about how you should behave while running the study, and what the rights of the participant are. Here’s the gist of those guidelines.


  Informed Consent

  Participants should be informed of what the experiment involves and be made fully aware of their rights while they are taking part. These include the right for them to withdraw from the study at any time (if they get scared or even merely bored). In the case of children who are too young to know what’s going on, informed consent should always be sought from their parents or carers. (This also applies of course to adults who are learning disabled, brain damaged or otherwise unable to give fully informed consent before participating). Since most kids are tested in their school, the easiest way to do this is to ask the school to get the children to take home a standard letter from you that explains what the study is about, gives relevant details about you (where you can be contacted for further information), and makes it clear to the parent that they have the option of refusing to allow their child to participate in the study. Bear in mind that, even if the parents give their permission, ultimately the child has the last word: if they don’t want to participate, or want to withdraw from the study at any time, then their wishes must be respected.

  Observational studies are normally carried out with the express intent of obtaining data about people’s normal behaviour, and hence without them being aware that they are participating in a study. Clearly informed consent is not feasible in these circumstances. The BPS suggest, as a rule of thumb, that it’s OK to make observations of people in situations where they could reasonably be expected to be on public view – so, for example, it would be alright to film people in a shopping mall or at an amusement park, but it wouldn’t be acceptable to find out what they get up to in their bedroom without them knowing that you were watching them and giving their express permission for you to do so. If you do this kind of research, take care to follow the other ethical principles outlined here (that is, do not harm or deceive people) and make sure that any recordings of the situation protect the participants’ confidentiality. It’s up to you to decide whether to tell people that they have participated, and what it’s all about (see ‘debriefing’, below). In some cases, it may not be feasible: for example, in a study of driver looking behaviour, a colleague of ours filmed drivers as they emerged from a junction. It would have been difficult for him to catch up with each driver and debrief them. Some years ago I listened to an ergonomist describe an observational study of men’s behaviour in public lavatories (as a basis for deciding innocuous things such as where to place the urinals in relation to the sinks, I hasten to add). It probably wouldn’t have been a good idea to debrief participants (excuse the pun!) in that study either.

  Deception

  Whether it is OK to deceive participants is a tricky issue, especially since we’ve just said that participants should give their informed consent. There are some areas of psychology where it doesn’t particularly matter whether the participant knows what the experiment is about. For example, if you were doing research on low-level visual processing, and were engaged in measuring people’s thresholds for, say, being able to judge the length of a line, it probably wouldn’t affect your results if the participants knew that was what you were trying to do. In fact, in many psychophysical experiments of this kind, it is commonplace for the experimenters to be amongst the participants in the study. (Generally one or two naive participants are tested as well, to check that the obtained results really aren’t affected by the experimenters’ insights). At the other extreme, there are many social psychological experiments which have depended on participants remaining ignorant of the study’s purpose, or have even involved actively misleading the participant about the aims of the study.

  By and large, active deception should be avoided in psychology experiments. (It’s easy for me to say that, as I do vision research, not social psychology!) Quite apart from the ethical considerations, the widespread use of deception in the past has made it difficult for the rest of us to run experiments – in my experience, participants (especially non-psychologists) become a little suspicious because they expect to be deceived. As a consequence, they often try to work out the hidden agenda in a study, even when there isn’t one!

  Debriefing

  Whether or not deception is used, you should always tell the participant what the experiment was about after they have finished it and leave time to answer any questions they might have. Quite apart from anything else, they have given up their time for you, and it’s only common courtesy for you to give them a few minutes of yours in return. Participants often regard the experiment as a ‘test’ of their abilities: they are usually concerned to have performed well, both for their own benefit (to reassure themselves that they are normal and competent) and for yours (because they want to have helped to make your experiment ‘work’. This can be a mixed blessing, if they have misinterpreted the purpose of the study and behaved weirdly as a result).

  In most cases, you will be comparing average performance in one condition to average performance in another, so you should make it clear to the participant that you are not interested in their individual data, and that their results will be combined with those from other people. Even if they have done badly, reassure them that their performance is fine: ‘Thanks very much, Igor. No, trying to press the response keys with your forehead rather than using your fingers is fine – lots of people do that, and it won’t have affected your results, which look perfectly normal, so don’t worry about it.’ The idea is that the participant leaves the room with a nice warm glow and a feeling that they have made a worthwhile contribution. Who knows? If you’ve done this properly, they might even take part in another experiment one day.

  Confidentiality

  Participants have the right to confidentiality, regardless of whether they have participated in a memory experiment or whether they have just completed a 20-page questionnaire that provides intimate details of their sado-machochistic pursuits. Individuals and their data should never be publicly identifiable. Running round the common-room shouting ‘Hey, everybody, look at the sexual perversions this one’s got!’ is definitely out. If you are doing research that involves case studies, the individuals concerned should be identified by initials, or better still false names. In the case of group data, the problem of confidentiality is less acute, but you should take care of things like database files that contain data together with people’s names. For most experiments, there’s no need to use the participant’s names: something like ‘Group A, participant 22: female, aged 40’ should suffice.

  Protection from physical and psychological harm

  You are obliged to protect your participants from harm, both physically and psychologically (or else find some place where the authorities will never find their bodies!) In psychology experiments, psychological harm is sometimes a possibility: you should avoid making participants feel stressed, embarrassed, depressed, anxious or fearful, unless they have given their prior permission for you to do this to them. For example, suppose you wanted to do an experiment on the effects of mood on memory, and you wanted to get some participants into a happy mood and others into a miserable mood. There’s probably no problem with making participants happy, but you should think carefully before making them sad. ‘Mood induction’ by means of gloomy music appears to be short-lived in its effects, but in a study such as this, it would be your responsibility to explain to the participant what was involved before the procedure took place, and to ensure that they felt OK before they left the laboratory. It’s probably best not to do this kind of research in a laboratory on the fifteenth floor with easily-openable windows . . .

  3.5 Summary

  * * *

  There are many issues that have to be considered in designing an experiment, but if you use one of the standard experimental designs described in this chapter, you should be well on the way to producing a study which will provide reliable and valid results. There is no one perfect design, because the optimal design will depend on your particular circumstances. However:

  Define precisely what it is you want to measure.

  Before runni
ng the study, think carefully about what statistics you are going to use.

  Sample the appropriate population, to obtain suitable participants. Sometimes you can get away with using students, sometimes you can’t. You may want to use a particular population, for example, geriatrics, children or left-footed Abyssinian female stockbrokers.

  Decide which design is most appropriate for your particular circumstances. In many cases, a multivariate design with repeated measures will be the most informative, economical and sensitive design to use, as long as carry-over effects don’t preclude its use. Keep the design as simple as you can.

  Having obtained a result, you have to assess its reliability and its validity. One way to do this is by using multiple converging operations: investigate the same phenomenon from different angles by using different techniques, participant groups, dependent variables, etc., to ensure that your results are not specific to your particular original study.

  You must behave ethically when you conduct research. Ensure you are familiar with the guidelines that have been produced by the BPS and the APA. Participants must give informed consent to participating in your study. Avoid deceiving them or exposing them to mental or physical harm. Once they have finished the experiment, give them an explanation of what it was all about, and make sure they leave in as good a state of mind as they were in when they arrived!