comment on validity and reliability in visual methods

Okay not sure what happened to my google document, so think I will just post here instead!


Validity is an element of social science research which address the issues of whether the researcher is actually measuring what they say they are. Validity is one criteria by which researchers judge their measurement tools; a valid result is one that accurately measures what it claims to be measuring. Reliability is the other criteria in assessing the quality and rigour of the research. It is a standard against which the tools used to measure concepts are judged. According to Hammersley (1992, cited in Bryman, 2004) reliability ‘refers to the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions’. Essentially reliability refers to consistency of results over time. In terms of image-based research there are a variety of approaches and definitions that attempt to deal with validity. According to Campbell and Stanley (1996 cited in Adelman, 1998) internal validity concerns the making of accurate inferences about whether the treatment that was implemented caused the effect that was measured for the specific persons observed in the particular setting under study. However Cook and Campbell (1979 cited in Adelman, 1998) equate validity with an imperfect assessment of truth value. In contrast, Cronbach (1982 cited in Adelman 1998) equates validity with persuasion, credibility and consensus: ‘Validity is subjective rather than objective’.

Image-based research involves a process of interpretation that can rest on a subjective and individual response. As Silverman (1993 cited in Prosser 1998) comments ‘The analysis of images raises complex methodological and theoretical issues…Moreover, the theoretical basis for the analysis of images is complex’. In terms of the documentary account, traditional researchers hold the belief that ‘reality’ is distorted by artistic convention and for artistic reason; indeed Walker-Evans is said to have set up his scenes to suit his aesthetic preferences in what is now contested to be authentic documentary film (Curtis, 1986 cited in Adelman, 1998). As such the processing of image-based data may feel more value-laden than the processing of text-based data, even if original image-based data are transformed into text for the purpose of analysis. According to orthodox researchers, ‘the act of image making…unacceptably alters the object in the frame and therefore objective content and subjective meaning of the image; images are, by their nature, ambiguous and do not in themselves convey meanings which are supplied serendipitaly by those who perceive them….analysis of images raises complex methodological and theoretical issues’ (Prosser, 1998). Essentially the main methodological argument against image-based research by other researchers is with images’ perceived lack of ‘trustworthiness’, despite new paradigm trends in such terms as ‘objectivity/subjectivity’ and ‘reliability/validity’ (Prosser, 1998). The concepts of validity and reliability within image-based research are therefore more prominent (Simco & Warin, 1997).

According to Prosser and Schwartz (1998) there are complex contextual issues that have to be considered right at the beginning of the research process such as the researchers’ underlying epistemological and methodological assumptions. For those involved in image-based research, issues such as the collection and interpretation of the data; the genuineness of the findings with regard to the respondents they claim to represent; and the assessment of competing interpretations of the data are therefore issues of importance in that they build robustness and help strengthen validity. As Caldarola (1985 cited in Prosser and Schwartz, 1998) illustrates, all data have strengths and limitations but data that is invalid, implausible, or untrustworthy is not worth analysing. The initial problem for the interpreter of photographs is therefore how best to ensure their plausibility and believability. According to Adelman (1998) the pursuit of internal validity for the photo document entails such things as informed selection of what to document, being systematic through reflection in the taking of photographs and low reactivity of the subjects to the presence of the photographer. The process of analytic induction proposed by Robinson (1951, cited in Adelman 1998) also allows for contrastive categories to emerge which will highlight ‘deviant’ photos and can therefore strengthen the photographers claim that the photo document is valid rather than random or accidental. Adelman (1998) goes on to suggest the acid test of making sense of photographs is whether the photographs communicate to viewers much of the intended messages. As such, to strengthen validity and reliability particular criteria have to be fulfilled. These criteria require that each image be filled with as much contextual detail for the researcher to engage in a systematic and iterative analysis of the image record. According to Prosser (1998) judgments and claims of contextual validity are best made via reflexive accounts and through representation. ‘Reflexive accounts attempt to render explicit the process by which data and findings were produced. Representation for image-based researchers, reflects not only the sources of information in terms of pictorial codes but also the mode of communicating findings to recipients of research’. In this way, full contextual detail enables the trustworthiness and limitations of photographs to be assessed and this means having an understanding of both the external and internal photocontext.

If such processes are acknowledged by those engaged in image-based research it essentially helps to address issues of validity and reliability that are often raised by orthodox researches. Perhaps as Prosser & Schwartz (1998) sugggest the future status and acceptability of image-based research may depend on working within a relatively conservative framework whilst exploring alternative modes of enquiry which are image-orientated yet sensitive to orthodox researchers, methodological concerns.


Bibliography

Adelman, C. (1998) ‘Photocontext’, in J. Prosser (ed) Image-based Research: A Sourcebook for Qualitative Researchers, RoutledgeFalmer: London, pp. 148-161.

Bryman A (2004) Social Research Methods. 2nd Ed. Oxford: Open University Press.

Prosser, J. (1998) ‘Status of Image-based Research’, in J. Prosser (ed) Image-based Research: A Sourcebook for Qualitative Researchers, RoutledgeFalmer: London, pp. 97-112.

Prosser, J. & Schwartz, D. (1998) ‘Photographs within the Sociological Research Process’, in J. Prosser (ed) Image-based Research: A Sourcebook for Qualitative Researchers, RoutledgeFalmer: London, pp. 115-130.

Simco, N. & Warin, J. (1997) Validity in Image-based Research: An elaborated illustration of some of the issues, British Educational Research Journal. Vol. 23, No. 5.