The Uses of Ethnography.
Introduction.
In this section we intend to suggest some roles which ethnography can play as a contributor to CSCW system design. Though we are strong aficionados of the method we do not regard it as a panacea for the problems of system design which are complex and ‘wicked’ [Rittel and Webber, 1973]. In other words, if ethnography is to take a more regarded place in CSCW design, then it is important to appraise not only its virtues but also its vices. Here we identify four uses of ethnography in various phases of the design cycle as a contribution to an evaluation of the method. We also briefly examine the arguments which have motivated the introduction of ethnography into systems design.
1. Justifying Ethnography (again).
The Case for Ethnography in CSCW
It is worth briefly reflecting on the rationale for ethnography in CSCW systems design. Two trends have strongly motivated the prominence ethnography currently enjoys:
• The growing plausibility of the diagnosis that the reason why many systems fail is due to the fact that their design pays insufficient attention to the social context of work; a failure often attributed to the inadequacy of existing methods of requirements elicitation and work analysis [Schmidt, 1993].
• A growing awareness with the emergence of low-cost technology that the ubiquitous nature of networked and distributed computing pose new problems for design which require the development of new methods which analyse the collaborative, hence social, character of work and its activities.
The tentative incorporation in system design of a social perspective emerges from these two trends and the insistence that the computer moves into the world of work and organisation [Grudin, 1990]. Given this ‘turn to the social’ and the need to study the ‘real world’ character of work, drifting toward sociology through ethnography is almost a natural inclination. Thus, in the way that HCI has previously looked to psychology for an understanding of human behaviour CSCW turns to sociology and in particular ethnography to provide insight into the social nature of work. The expectation is that requirements elicitation is to be informed by an analysis of the ‘real world’ circumstances of work and its organisation [Goguen, 1993].
This is reflected more generally in a growing awareness within the software engineering community that the understanding the ‘social’ real world is an important factor in software design and development itself [Potts, 1993]. The main virtue of ethnography is its ability to make visible the ‘real world, real time’ sociality of a setting. As a mode of social research it is concerned to produce detailed descriptions of the ‘workaday’ activities of social actors within specific contexts [Hughes et al, 1993a; Hughes et al, 1993b]. It is a naturalistic method relying upon material drawn from the first-hand experience of a fieldworker in some setting. It seeks to present a portrait of life as seen and understood by those who live and work within the domain concerned. It this objective which is the rationale behind the method’s insistence on the direct involvement of the researcher in the setting under investigation. The intention of ethnography is to see activities as social actions embedded within a socially organised domain and accomplished in and through the day-to-day activities of participants. It is this which provides access to the everyday ways in which participants understand and conduct their working lives.
It is the ability of ethnography to understand a social setting as it is perceived by those involved in that setting [the archetypal ‘users’] that underpins its appeal to developers. However, it is not without its problems, there are, for example, those to do with presenting the results of ethnography in a form which is readily assimilated by designers. For many software engineers ethnography seems far too unsystematic a method, its results presented in a discursive form, design options are not clearly stated and do not attend sufficiently to engineering needs. Its virtues, in other words, become vices.
Against this is the argument that what is wrong with many of the traditional methods of system design is that they owe far too much to the needs of engineering with the result that crucial aspects of the ‘real world’ of work are obscured, misrepresented or never properly treated [Schmidt, 1993]. It is in this respect that ‘analytic approaches’, Task Analysis, Office Automation for example, which focus on the flow of data within a domain, are found wanting [Shapiro, 1993; Suchman, 1983]. While it is accepted that a balance needs to be found between the requirements of engineering and the need to adequately characterise the domain of application, such methods are an intrusion of the ‘engineering mentality’ into areas where it is inappropriate. The result is, so it is argued, that essential aspects of the socially organised character of the domain concerned are obscured or, worse, misrepresented. More specifically, the analytic deconstruction of work activities into ever more finely grained components removes the essential ‘real world’ features which make them practises within a socially organised setting. This complaint attacks the individualistic slant of the cognitivism which underlies ‘analytic approaches’ by acknowledging the implications of the observation that work is, typically, collaborative. Though the activities constituting work are done by individuals, they are performed within an organised environment composed of other individuals and it is this which gives shape to the activities as ‘real world’ activities. Thus, the focus of ethnography is on the social practises which enable the very processes which ‘analytic methods’ identify but which they decontextualise. It is through social practises that processes are established and, accordingly, rooted in socially achieved sets of arrangements.
There are, of course, many aspects to these kinds of arguments, some of which involve a critique of the nature of work in modern society and how current methods of design instantiate the dehumanising rationality of modernism Again as we have emphasised earlier, our own arguments for ethnography are more pragmatic in nature. If we accept that CSCW design needs to attend to the sociality of work, then any method must respect the nature of this phenomenon. However, many of the existing methods fail to sufficiently recognise the social nature of work. This is not a call for the wholesale abandonment of more formal methods; they, like ethnography, will need to find an appropriate place in design.
Accordingly, although there is a case for ethnography in CSCW system design, at the present time it is a promissory note rather than a claim based on substantial achievement. Its main use has been in research and mainly field sites which are small scale involving highly focused interactions, such as control rooms. Accordingly, if it is to substantiate its case as a method of system design, it will need to go beyond these and, in addition, face up to the problems of large scale system development.
2. Ethnography in System Design.
Moving Beyond Research: Some problems of Ethnography.
There are very real problems in the design and development of large scale systems, having to do with obtaining adequate knowledge of the relevant domain, communicating this to designers and organising the process of system building. In commercial contexts these problems are deeply infused with the familiar commercial constraints of budgets, time and resources. This means that methods such as ethnography must service a number of demands if they are to be widely accepted in industry. Without this acceptance the use of ethnography is systems design runs the risk of becoming a research curiosity and, thus, devalued as a tool to support effective CSCW design.
As a number of studies have shown software engineers typically work under some pressure [Curtis, et al, 1988; Bansler and Bødker, 1993]; a pressure which is, in part, determined by market factors. However, the familiar moan that most system development projects are ‘over time, over budget’ cannot be entirely laid at the door of market pressures. Building large scale systems is a complex and difficult business. Many of them are ‘one off’ with little in the way of past experience to serve as a guide. It was problems such as these which provoked the development of software design methodologies to systematise and manage the process of design and development so that systems had a reasonable chance of meeting both technical and commercial targets. These pressures still hold true and apply equally well to ethnography.
On the face of it ethnography does not accommodate easily to the pressures of development. A set of tensions become apparent when we examine ethnography in the light of systems design and it is important that the role of ethnography is considered within this context. These tensions include the familiar pressures of scale and time and place new demands on ethnography in system design.
The problem of scale
To date the main use of ethnography has not only been within research settings but also confined to relatively small scale and relatively confined environments, such as control rooms and other micro interactional contexts. In such settings there tends to be a clear focus of attention for the participants, who are typically few in number, and in which there is a relatively clearly visible differentiation of tasks at one work site. Though ‘team ethnography’ is a possibility, this is likely to be an ineffective and impractically expensive strategy which could well create as many problems as it would solve. The realistic possibility is that ethnography will continue to be undertaken by a single investigator or, perhaps in exceptional cases, by a couple of them. For the lone fieldworker the sites described above are ideal. They minimise travel and communication problems, and all that the fieldworker needs to see is there in one place and can be gathered with a minimum of disruption. Scaling such inquiries up to the organisational level or to processes distributed in time and space is a much more daunting prospect in raising issues of depth and representativeness.
The pressure of time
As one of our computer science colleagues expressed it, ethnography is a ‘prolonged activity’ and in the context of social research can last a number of years, certainly time scales which would be considered a joke in software engineering. Added to this are the problems, noted earlier, of communicating ethnographic findings to designers. The output of ethnographic analyses are typically discursive and lengthy, looking nothing like the blueprint diagrams which are de rigeur in systems engineering.
The role of the ethnographer
Moving out of the research setting into a more commercial one also raises different sets of ethical responsibilities as well as making access to sites more vulnerable to the contingencies of the commercial and industrial world. Ethnography insists that its inquiries be conducted in a non-disruptive and non-interventionist manner, principles which can be compromised given that much of the motivation for IT is to reorganise work and, as part of this, often seek to displace labour. Less dramatically, but important nonetheless, fieldworkers not only require access to relevant sites but also need acceptance on the part of those who work in them. Protecting the identity of people, respecting the fact that the fieldworker is like a guest within their lives, and so on, become much harder to sustain in applied work of this kind. It is also too easy for the fieldworker to get drawn into intra-organisational politics especially when dependent upon ‘facilitators’ within organisations who have their own agendas to pursue.
Of course, few of these issues are easily solved. However, it is important not to be too ambitious for any method, least of all in software engineering where new methods follow one another with monotonous regularity. The ‘wickedness’ of its problems means that design is, at best, a ‘satisficing’ activity and a matter of doing the best one can with the resources available. Nevertheless, if it is accepted that designers should be informed about the social character of work, and that ethnography is an important means of gaining such knowledge, then serious attention needs to be given to the variety of ways in which ethnographic studies can be used by designers. What follows is an attempt to specify some of these ways using, in the main, our own experiences over four years of collaborative and interdisciplinary research. We do not offer these in anything other than the spirit of what can be done now. Research on ethnography and system design is continuing in a number of quarters and it may well be that in a few years the picture will be very different. For us, the important issue at the present time is to sensitise CSCW system designers to the sociality of work as systematically and as effectively as possible.
3. Ethnography in system design: 'types' of Ethnography.
The wish to incorporate ethnography into the already diverse collection of methods, tools and techniques used in system design must be viewed with some trepidation. While we accept the need for the inclusion of a social perspective on design we must be careful to avoid seeing ethnography as a ready-made solution. The experiences of ethnography within systems design are limited and, as pointed out earlier, mainly confined to small-scale settings and of highly focused activities. [See, for example, Heath and Luff, 1992; Heath and Luff, 1994; Hughes et al, 1992; Harper and Hughes, 1992; Harper et al, 1991; Heath et al, 1993 and, of course, Suchman, 1987]
However, ethnography is a much richer method than these previous studies and the reports of design experiences might perhaps suggest. It is important that existing studies are complemented by a consideration of the variety of different ways in which ethnography can influence systems design..
The different uses of ethnography within design we identify include:
• Concurrent ethnography: in which design is influence by an on-going ethnographic study taking place at the same time as systems development.
• Quick and dirty ethnography: in which brief ethnographic studies are undertaken to provide a general but informed sense of the setting for designers.
• Evaluative ethnography: in which an ethnographic study is undertaken to verify or validate a set of already formulated design decisions.
• Re-examination of previous studies: in which previous studies are re-examined to inform initial design thinking.
In the following we suggest to suggest what each of these has to offer design and also identify some of the problems that could arise. These categories should not be read as if they were mutually exclusive ways of using ethnography in system design. As we will suggest, some of the uses could be harnessed together and the differences between them seen as differences of emphasis rather sharp demarcations.
Design, as in so much else, is a matter of responding to contingencies of various kinds. What is also important to note is that the schema recognises that design objectives are themselves various and that this will have a bearing on the role of ethnography. In other words, while not necessarily buying into the picture of the design process as a series of discrete, clearly delineated and phased steps, it undoubtedly has different objectives at different stages and, accordingly, implications for how design needs to be informed by relevant information about the domain.
Concurrent ethnography
This use is perhaps the one most commonly associated with design and the one most commentated on [See, for example, Heath et al, 1993; Hughes et al, 1993]. It is a sequenced process in which the ethnographic investigation of a domain precedes the design development of the system. This is the method we followed in the design of a tool for the rapid prototyping of interfaces for controlling [Bentley et al, 1992]. In this case a period of some four weeks ethnography in the London Air Traffic Control Centre [LATTC] was followed by a lengthy debriefing session involving both the fieldworker and the designers. Meanwhile, a first prototype was constructed. The process of fieldwork > debriefing > prototype iteration > fieldwork was repeated about four times until the team was satisfied that little more could be usefully gained by more fieldwork. The penultimate version of the system was then evaluated using working controllers. The process was a directed one in that each stage of the fieldwork was intended to target issues raised by the designers during the debriefings, although the first phase was more concerned with the very important task of the fieldworker familiarising himself with the setting and the work of the controllers.
It is important to note that the aim of the project was research rather than the development of a system to be used in the ‘front line’ of controlling. Thus, we did not have the problems which would have arisen in implementing the tool. The research team was small so that much of the communication between the sociologists and the computer scientists could be done informally. There was little need for a requirements document or for a process model since the development work was done through rapid prototyping.
What the ethnography especially provided was a thorough insight into the subtleties involved in controlling work and in the routine interactions among the members of the controlling team around the suite; subtleties which were rooted in the sociality of the work and its organisation. The vital moment-by-moment mutual checking of ‘what was going on’ by the various members of the team had been missed by earlier cognitive and task analytic approaches to describing controlling work. What also became clear is that any new interface system would have to keep the controller ‘geared into’ the work by not automating, for example, the ordering of the screen-based flight strips. In other words, we felt it important to retain at least some of the functionalities of the current paper flight strips while, at the same time, being in a position to evaluate what information the controller needs, what is less important but needs to be ‘ready to hand’, and what was inessential.
We also learned that there was a declining rate of utility for the fieldwork contribution to the design. This is not to say that there was not more to learn or that we could not have learned more sociologically from further study of the control room, only that in terms of the project the ‘fine tuning’ of the design needed to be informed by experts actually using it. The intricacies of tool use in air traffic control typically require a background of several years of work experience for their mastery, a grasp of the ways in which the employment of the tool is capable of being effortlessly used within the manifold requirements and exigencies of safe controlling: levels of sensitivity which cannot be matched by the ethnographer on the basis of a few weeks acquaintance with the work. In other words, although there is always more to learn, the payoffs for design, at least in this case, came relatively quickly in comparison with social research uses of ethnography.
'quick and dirty' ethnography
This characterisation of a role for ethnography is, in many respects, a rationalisation of the experiences of a project which did not go quite as intended, but which, nonetheless, provided valuable insights not only into the use of ethnography but also about the character of ‘real world’ software engineering design and, through this, some of the limiting conditions affecting the provision of computer support in CSCW contexts.
The principal distinction between this project and our previous experiences within ATC was the larger scale of the work setting. The ATC suite provided a natural focus and location for the work taking place. However, in the case of software development both the location and focus of work was considerably less apparent to both the developers and ethnographers and the issue of scale needed to be directly addressed.
Large scale settings
We have already noted some of the problems of scaling up ethnography beyond the confines of such as control rooms. In the case we use for illustration, the project was concerned to use an ethnographic investigation of software engineers at work in order to inform the design of a support tool; a tool which would, hopefully, enable designers to display the rationale behind their design choices and, through this, improve the quality of the system and the maintainability of the software. The aim was to develop a tool which more adequately reflected the collaborative and interdependent character of ‘real world’ design work. We planned to follow the pattern of the study mentioned previously; that is, a first period of familiarisation fieldwork while, at the same time, building the basic prototype, to be followed through by a series of iterations of debriefing, more directed fieldwork and prototype iterations. Although we had ready access to various sites, and to colleagues working in the same area, it was difficult to find projects we could study which were starting as opposed to those which were already some way along their development trajectories. Nevertheless, we felt that we would still gain a great deal for our purposes.
We realised from the beginning, and this was one of the purposes of the study, that the fieldwork would represent new challenges in involving a much less ‘confined’ field site than the control suite at LATCC. For one, the development engineers in both of the sites we eventually looked at, were working in industrial environments and, accordingly, subjected to a wider range of contingencies, events and policies which impacted more directly on their work. For example, one of the projects at the first field site was cancelled and access to another project within the same company proved more difficult due, to put it diplomatically, to one of the ‘gate keepers’, a team manager, being less than enthusiastic about a fieldworker studying a team under considerable pressure.
While we may have been unlucky in this case and more fortunate in the case of LATCC, it does highlight an important feature of ethnographic research, namely, its reliance on being accepted in the setting and, even if this is forthcoming, being subject to the range of contingencies that are capable of afflicting all ‘real world’ organisations. Among these, of course, are those to do with, for want of a better phrase, the ‘local politics’ of the organisation.
In addition to these were the problems arising from asking a fieldworker to cover what proved to be a much larger task than we had anticipated. Software development is a complex business and tracking through its unfamiliar complexities, understanding the management of its components, seeing how the teams worked together, trying to figure out how the integration of the various components was achieved, and more, all proved to be a much more immense task than we envisaged originally.
Nevertheless, and despite less than ideal circumstances such as those noted, one can always learn something from ethnography. Indeed, seeing how the kind of contingencies we have reviewed can impact on design and development is important and, of course, illustrative of the argument CSCW makes about the necessity of studying the ‘real world’ circumstances of work to inform system design. In this case, we learnt sufficient about the design process as a ‘real world’ phenomenon to indicate that the tool as originally envisaged was, in significant respects, wrongly conceived. Briefly, it would only be effective if it was consistently used by members of project teams. However, in the conditions in which they typically worked, this would represent a considerable overhead. Also, given the personal and company investment in CASE tools of various kinds, persuading engineers to learn and use ‘yet another bloody tool’ when they were already less than enthusiastic about their current ones, would have been a mammoth task.
In the second site many of the problems indicated above also emerged. The development involved, at the time of the fieldwork, approximately a hundred software engineers working on an avionics systems for a new version of an aircraft. The work was organised according to a strict Process Model which was highly constrained, document driven and implemented under very tight budgetary constraints. This again provided insights into the ‘real world’ of design, particularly on the impact of management styles, the importance of professional pride the engineers exhibited in ‘their craft’, and a better understanding of the relationship of the Process Model used to organise the work to what actually goes on [Rodden et al, 1994]. As far as the last point is concerned, in some respects the implementation of the work plan was so constraining that the engineers frequently made recourse to ‘fixes’ of various kinds in order to get the work done at all. Indeed, a surprising finding was the extent to which ‘social and interactional issues’ were constantly addressed with the aim of improving the efficiency and the quality of the work. For example, during the fieldwork the project team was reorganised to improve communication, the sharing of experience and skill, and various ‘team building’ exercises were arranged by management, though the latter did not attend them.
The phrase ‘quick and dirty’ does not refer simply to a short period of fieldwork but signals its duration relative to the size of the task. The use of ethnographic study in this category not only seeks relevant information as quickly as possible but accepts at the outset the impossibility of gathering a complete and detailed understanding of the setting at hand. Rather the focus is informing strategic decision making to select those aspects of the work setting of particular importance in informing design.
There are two points of comparison with what we have called ‘concurrent ethnography’ that are worth noting. First, compared to the much more focused attention of ‘concurrent ethnography’, and this emerged in the example we have used out of the problems of access and those of finding a clear focus for the study, ‘quick and dirty’ ethnography is capable of providing much valuable knowledge of the social organisation of work of a relatively large scale work setting in a relatively short space of time, and this includes what we were able to learn from the organisational problems that arose when trying to establish the research site. Indeed, it can be argued that the ‘pay off’ of the ‘quick and dirty’ ethnography is greater in that for time expended on fieldwork a great deal is learned. Second, such knowledge can be built upon for a more focused examination of the detailed aspects of the work which is more typical of ‘concurrent ethnography’. What the ‘quick and dirty’ fieldwork provides is the important broad understanding which is capable of sensitising designers particularly to issues which have a bearing on the acceptability and usability of an envisaged system rather than on the specifics of design. Both aspects, of course, are important.
The research also raised the problem of communicating the findings from the ethnographic study to designers, mainly because of the increased scale of the setting and the problems of finding a clear design focus. While the fieldworker learned a great deal in the study just discussed, certainly much that is useful for a sociological study, it proved difficult to hang this onto clearly formulated design objectives. In spite of this, even if used with this limitation ‘quick and dirty’ ethnography is capable of providing an informed sense of what the work is like in a way that can be useful for designers in scoping their design. In other words, although in our own case the research raised important questions about the initial design objectives, and this is not a pointless finding by any means, it did suggest useful ways in which ethnography could be used to provide designers with a better sense of the setting and its work activities.
The Case for Ethnography in CSCW
It is worth briefly reflecting on the rationale for ethnography in CSCW systems design. Two trends have strongly motivated the prominence ethnography currently enjoys:
• The growing plausibility of the diagnosis that the reason why many systems fail is due to the fact that their design pays insufficient attention to the social context of work; a failure often attributed to the inadequacy of existing methods of requirements elicitation and work analysis [Schmidt, 1993].
• A growing awareness with the emergence of low-cost technology that the ubiquitous nature of networked and distributed computing pose new problems for design which require the development of new methods which analyse the collaborative, hence social, character of work and its activities.
The tentative incorporation in system design of a social perspective emerges from these two trends and the insistence that the computer moves into the world of work and organisation [Grudin, 1990]. Given this ‘turn to the social’ and the need to study the ‘real world’ character of work, drifting toward sociology through ethnography is almost a natural inclination. Thus, in the way that HCI has previously looked to psychology for an understanding of human behaviour CSCW turns to sociology and in particular ethnography to provide insight into the social nature of work. The expectation is that requirements elicitation is to be informed by an analysis of the ‘real world’ circumstances of work and its organisation [Goguen, 1993].
This is reflected more generally in a growing awareness within the software engineering community that the understanding the ‘social’ real world is an important factor in software design and development itself [Potts, 1993]. The main virtue of ethnography is its ability to make visible the ‘real world, real time’ sociality of a setting. As a mode of social research it is concerned to produce detailed descriptions of the ‘workaday’ activities of social actors within specific contexts [Hughes et al, 1993a; Hughes et al, 1993b]. It is a naturalistic method relying upon material drawn from the first-hand experience of a fieldworker in some setting. It seeks to present a portrait of life as seen and understood by those who live and work within the domain concerned. It this objective which is the rationale behind the method’s insistence on the direct involvement of the researcher in the setting under investigation. The intention of ethnography is to see activities as social actions embedded within a socially organised domain and accomplished in and through the day-to-day activities of participants. It is this which provides access to the everyday ways in which participants understand and conduct their working lives.
It is the ability of ethnography to understand a social setting as it is perceived by those involved in that setting [the archetypal ‘users’] that underpins its appeal to developers. However, it is not without its problems, there are, for example, those to do with presenting the results of ethnography in a form which is readily assimilated by designers. For many software engineers ethnography seems far too unsystematic a method, its results presented in a discursive form, design options are not clearly stated and do not attend sufficiently to engineering needs. Its virtues, in other words, become vices.
Against this is the argument that what is wrong with many of the traditional methods of system design is that they owe far too much to the needs of engineering with the result that crucial aspects of the ‘real world’ of work are obscured, misrepresented or never properly treated [Schmidt, 1993]. It is in this respect that ‘analytic approaches’, Task Analysis, Office Automation for example, which focus on the flow of data within a domain, are found wanting [Shapiro, 1993; Suchman, 1983]. While it is accepted that a balance needs to be found between the requirements of engineering and the need to adequately characterise the domain of application, such methods are an intrusion of the ‘engineering mentality’ into areas where it is inappropriate. The result is, so it is argued, that essential aspects of the socially organised character of the domain concerned are obscured or, worse, misrepresented. More specifically, the analytic deconstruction of work activities into ever more finely grained components removes the essential ‘real world’ features which make them practises within a socially organised setting. This complaint attacks the individualistic slant of the cognitivism which underlies ‘analytic approaches’ by acknowledging the implications of the observation that work is, typically, collaborative. Though the activities constituting work are done by individuals, they are performed within an organised environment composed of other individuals and it is this which gives shape to the activities as ‘real world’ activities. Thus, the focus of ethnography is on the social practises which enable the very processes which ‘analytic methods’ identify but which they decontextualise. It is through social practises that processes are established and, accordingly, rooted in socially achieved sets of arrangements.
There are, of course, many aspects to these kinds of arguments, some of which involve a critique of the nature of work in modern society and how current methods of design instantiate the dehumanising rationality of modernism Again as we have emphasised earlier, our own arguments for ethnography are more pragmatic in nature. If we accept that CSCW design needs to attend to the sociality of work, then any method must respect the nature of this phenomenon. However, many of the existing methods fail to sufficiently recognise the social nature of work. This is not a call for the wholesale abandonment of more formal methods; they, like ethnography, will need to find an appropriate place in design.
Accordingly, although there is a case for ethnography in CSCW system design, at the present time it is a promissory note rather than a claim based on substantial achievement. Its main use has been in research and mainly field sites which are small scale involving highly focused interactions, such as control rooms. Accordingly, if it is to substantiate its case as a method of system design, it will need to go beyond these and, in addition, face up to the problems of large scale system development.
2. Ethnography in System Design.
Moving Beyond Research: Some problems of Ethnography.
There are very real problems in the design and development of large scale systems, having to do with obtaining adequate knowledge of the relevant domain, communicating this to designers and organising the process of system building. In commercial contexts these problems are deeply infused with the familiar commercial constraints of budgets, time and resources. This means that methods such as ethnography must service a number of demands if they are to be widely accepted in industry. Without this acceptance the use of ethnography is systems design runs the risk of becoming a research curiosity and, thus, devalued as a tool to support effective CSCW design.
As a number of studies have shown software engineers typically work under some pressure [Curtis, et al, 1988; Bansler and Bødker, 1993]; a pressure which is, in part, determined by market factors. However, the familiar moan that most system development projects are ‘over time, over budget’ cannot be entirely laid at the door of market pressures. Building large scale systems is a complex and difficult business. Many of them are ‘one off’ with little in the way of past experience to serve as a guide. It was problems such as these which provoked the development of software design methodologies to systematise and manage the process of design and development so that systems had a reasonable chance of meeting both technical and commercial targets. These pressures still hold true and apply equally well to ethnography.
On the face of it ethnography does not accommodate easily to the pressures of development. A set of tensions become apparent when we examine ethnography in the light of systems design and it is important that the role of ethnography is considered within this context. These tensions include the familiar pressures of scale and time and place new demands on ethnography in system design.
The problem of scale
To date the main use of ethnography has not only been within research settings but also confined to relatively small scale and relatively confined environments, such as control rooms and other micro interactional contexts. In such settings there tends to be a clear focus of attention for the participants, who are typically few in number, and in which there is a relatively clearly visible differentiation of tasks at one work site. Though ‘team ethnography’ is a possibility, this is likely to be an ineffective and impractically expensive strategy which could well create as many problems as it would solve. The realistic possibility is that ethnography will continue to be undertaken by a single investigator or, perhaps in exceptional cases, by a couple of them. For the lone fieldworker the sites described above are ideal. They minimise travel and communication problems, and all that the fieldworker needs to see is there in one place and can be gathered with a minimum of disruption. Scaling such inquiries up to the organisational level or to processes distributed in time and space is a much more daunting prospect in raising issues of depth and representativeness.
The pressure of time
As one of our computer science colleagues expressed it, ethnography is a ‘prolonged activity’ and in the context of social research can last a number of years, certainly time scales which would be considered a joke in software engineering. Added to this are the problems, noted earlier, of communicating ethnographic findings to designers. The output of ethnographic analyses are typically discursive and lengthy, looking nothing like the blueprint diagrams which are de rigeur in systems engineering.
The role of the ethnographer
Moving out of the research setting into a more commercial one also raises different sets of ethical responsibilities as well as making access to sites more vulnerable to the contingencies of the commercial and industrial world. Ethnography insists that its inquiries be conducted in a non-disruptive and non-interventionist manner, principles which can be compromised given that much of the motivation for IT is to reorganise work and, as part of this, often seek to displace labour. Less dramatically, but important nonetheless, fieldworkers not only require access to relevant sites but also need acceptance on the part of those who work in them. Protecting the identity of people, respecting the fact that the fieldworker is like a guest within their lives, and so on, become much harder to sustain in applied work of this kind. It is also too easy for the fieldworker to get drawn into intra-organisational politics especially when dependent upon ‘facilitators’ within organisations who have their own agendas to pursue.
Of course, few of these issues are easily solved. However, it is important not to be too ambitious for any method, least of all in software engineering where new methods follow one another with monotonous regularity. The ‘wickedness’ of its problems means that design is, at best, a ‘satisficing’ activity and a matter of doing the best one can with the resources available. Nevertheless, if it is accepted that designers should be informed about the social character of work, and that ethnography is an important means of gaining such knowledge, then serious attention needs to be given to the variety of ways in which ethnographic studies can be used by designers. What follows is an attempt to specify some of these ways using, in the main, our own experiences over four years of collaborative and interdisciplinary research. We do not offer these in anything other than the spirit of what can be done now. Research on ethnography and system design is continuing in a number of quarters and it may well be that in a few years the picture will be very different. For us, the important issue at the present time is to sensitise CSCW system designers to the sociality of work as systematically and as effectively as possible.
3. Ethnography in system design: 'types' of Ethnography.
The wish to incorporate ethnography into the already diverse collection of methods, tools and techniques used in system design must be viewed with some trepidation. While we accept the need for the inclusion of a social perspective on design we must be careful to avoid seeing ethnography as a ready-made solution. The experiences of ethnography within systems design are limited and, as pointed out earlier, mainly confined to small-scale settings and of highly focused activities. [See, for example, Heath and Luff, 1992; Heath and Luff, 1994; Hughes et al, 1992; Harper and Hughes, 1992; Harper et al, 1991; Heath et al, 1993 and, of course, Suchman, 1987]
However, ethnography is a much richer method than these previous studies and the reports of design experiences might perhaps suggest. It is important that existing studies are complemented by a consideration of the variety of different ways in which ethnography can influence systems design..
The different uses of ethnography within design we identify include:
• Concurrent ethnography: in which design is influence by an on-going ethnographic study taking place at the same time as systems development.
• Quick and dirty ethnography: in which brief ethnographic studies are undertaken to provide a general but informed sense of the setting for designers.
• Evaluative ethnography: in which an ethnographic study is undertaken to verify or validate a set of already formulated design decisions.
• Re-examination of previous studies: in which previous studies are re-examined to inform initial design thinking.
In the following we suggest to suggest what each of these has to offer design and also identify some of the problems that could arise. These categories should not be read as if they were mutually exclusive ways of using ethnography in system design. As we will suggest, some of the uses could be harnessed together and the differences between them seen as differences of emphasis rather sharp demarcations.
Design, as in so much else, is a matter of responding to contingencies of various kinds. What is also important to note is that the schema recognises that design objectives are themselves various and that this will have a bearing on the role of ethnography. In other words, while not necessarily buying into the picture of the design process as a series of discrete, clearly delineated and phased steps, it undoubtedly has different objectives at different stages and, accordingly, implications for how design needs to be informed by relevant information about the domain.
Concurrent ethnography
This use is perhaps the one most commonly associated with design and the one most commentated on [See, for example, Heath et al, 1993; Hughes et al, 1993]. It is a sequenced process in which the ethnographic investigation of a domain precedes the design development of the system. This is the method we followed in the design of a tool for the rapid prototyping of interfaces for controlling [Bentley et al, 1992]. In this case a period of some four weeks ethnography in the London Air Traffic Control Centre [LATTC] was followed by a lengthy debriefing session involving both the fieldworker and the designers. Meanwhile, a first prototype was constructed. The process of fieldwork > debriefing > prototype iteration > fieldwork was repeated about four times until the team was satisfied that little more could be usefully gained by more fieldwork. The penultimate version of the system was then evaluated using working controllers. The process was a directed one in that each stage of the fieldwork was intended to target issues raised by the designers during the debriefings, although the first phase was more concerned with the very important task of the fieldworker familiarising himself with the setting and the work of the controllers.
It is important to note that the aim of the project was research rather than the development of a system to be used in the ‘front line’ of controlling. Thus, we did not have the problems which would have arisen in implementing the tool. The research team was small so that much of the communication between the sociologists and the computer scientists could be done informally. There was little need for a requirements document or for a process model since the development work was done through rapid prototyping.
What the ethnography especially provided was a thorough insight into the subtleties involved in controlling work and in the routine interactions among the members of the controlling team around the suite; subtleties which were rooted in the sociality of the work and its organisation. The vital moment-by-moment mutual checking of ‘what was going on’ by the various members of the team had been missed by earlier cognitive and task analytic approaches to describing controlling work. What also became clear is that any new interface system would have to keep the controller ‘geared into’ the work by not automating, for example, the ordering of the screen-based flight strips. In other words, we felt it important to retain at least some of the functionalities of the current paper flight strips while, at the same time, being in a position to evaluate what information the controller needs, what is less important but needs to be ‘ready to hand’, and what was inessential.
We also learned that there was a declining rate of utility for the fieldwork contribution to the design. This is not to say that there was not more to learn or that we could not have learned more sociologically from further study of the control room, only that in terms of the project the ‘fine tuning’ of the design needed to be informed by experts actually using it. The intricacies of tool use in air traffic control typically require a background of several years of work experience for their mastery, a grasp of the ways in which the employment of the tool is capable of being effortlessly used within the manifold requirements and exigencies of safe controlling: levels of sensitivity which cannot be matched by the ethnographer on the basis of a few weeks acquaintance with the work. In other words, although there is always more to learn, the payoffs for design, at least in this case, came relatively quickly in comparison with social research uses of ethnography.
'quick and dirty' ethnography
This characterisation of a role for ethnography is, in many respects, a rationalisation of the experiences of a project which did not go quite as intended, but which, nonetheless, provided valuable insights not only into the use of ethnography but also about the character of ‘real world’ software engineering design and, through this, some of the limiting conditions affecting the provision of computer support in CSCW contexts.
The principal distinction between this project and our previous experiences within ATC was the larger scale of the work setting. The ATC suite provided a natural focus and location for the work taking place. However, in the case of software development both the location and focus of work was considerably less apparent to both the developers and ethnographers and the issue of scale needed to be directly addressed.
Large scale settings
We have already noted some of the problems of scaling up ethnography beyond the confines of such as control rooms. In the case we use for illustration, the project was concerned to use an ethnographic investigation of software engineers at work in order to inform the design of a support tool; a tool which would, hopefully, enable designers to display the rationale behind their design choices and, through this, improve the quality of the system and the maintainability of the software. The aim was to develop a tool which more adequately reflected the collaborative and interdependent character of ‘real world’ design work. We planned to follow the pattern of the study mentioned previously; that is, a first period of familiarisation fieldwork while, at the same time, building the basic prototype, to be followed through by a series of iterations of debriefing, more directed fieldwork and prototype iterations. Although we had ready access to various sites, and to colleagues working in the same area, it was difficult to find projects we could study which were starting as opposed to those which were already some way along their development trajectories. Nevertheless, we felt that we would still gain a great deal for our purposes.
We realised from the beginning, and this was one of the purposes of the study, that the fieldwork would represent new challenges in involving a much less ‘confined’ field site than the control suite at LATCC. For one, the development engineers in both of the sites we eventually looked at, were working in industrial environments and, accordingly, subjected to a wider range of contingencies, events and policies which impacted more directly on their work. For example, one of the projects at the first field site was cancelled and access to another project within the same company proved more difficult due, to put it diplomatically, to one of the ‘gate keepers’, a team manager, being less than enthusiastic about a fieldworker studying a team under considerable pressure.
While we may have been unlucky in this case and more fortunate in the case of LATCC, it does highlight an important feature of ethnographic research, namely, its reliance on being accepted in the setting and, even if this is forthcoming, being subject to the range of contingencies that are capable of afflicting all ‘real world’ organisations. Among these, of course, are those to do with, for want of a better phrase, the ‘local politics’ of the organisation.
In addition to these were the problems arising from asking a fieldworker to cover what proved to be a much larger task than we had anticipated. Software development is a complex business and tracking through its unfamiliar complexities, understanding the management of its components, seeing how the teams worked together, trying to figure out how the integration of the various components was achieved, and more, all proved to be a much more immense task than we envisaged originally.
Nevertheless, and despite less than ideal circumstances such as those noted, one can always learn something from ethnography. Indeed, seeing how the kind of contingencies we have reviewed can impact on design and development is important and, of course, illustrative of the argument CSCW makes about the necessity of studying the ‘real world’ circumstances of work to inform system design. In this case, we learnt sufficient about the design process as a ‘real world’ phenomenon to indicate that the tool as originally envisaged was, in significant respects, wrongly conceived. Briefly, it would only be effective if it was consistently used by members of project teams. However, in the conditions in which they typically worked, this would represent a considerable overhead. Also, given the personal and company investment in CASE tools of various kinds, persuading engineers to learn and use ‘yet another bloody tool’ when they were already less than enthusiastic about their current ones, would have been a mammoth task.
In the second site many of the problems indicated above also emerged. The development involved, at the time of the fieldwork, approximately a hundred software engineers working on an avionics systems for a new version of an aircraft. The work was organised according to a strict Process Model which was highly constrained, document driven and implemented under very tight budgetary constraints. This again provided insights into the ‘real world’ of design, particularly on the impact of management styles, the importance of professional pride the engineers exhibited in ‘their craft’, and a better understanding of the relationship of the Process Model used to organise the work to what actually goes on [Rodden et al, 1994]. As far as the last point is concerned, in some respects the implementation of the work plan was so constraining that the engineers frequently made recourse to ‘fixes’ of various kinds in order to get the work done at all. Indeed, a surprising finding was the extent to which ‘social and interactional issues’ were constantly addressed with the aim of improving the efficiency and the quality of the work. For example, during the fieldwork the project team was reorganised to improve communication, the sharing of experience and skill, and various ‘team building’ exercises were arranged by management, though the latter did not attend them.
The phrase ‘quick and dirty’ does not refer simply to a short period of fieldwork but signals its duration relative to the size of the task. The use of ethnographic study in this category not only seeks relevant information as quickly as possible but accepts at the outset the impossibility of gathering a complete and detailed understanding of the setting at hand. Rather the focus is informing strategic decision making to select those aspects of the work setting of particular importance in informing design.
There are two points of comparison with what we have called ‘concurrent ethnography’ that are worth noting. First, compared to the much more focused attention of ‘concurrent ethnography’, and this emerged in the example we have used out of the problems of access and those of finding a clear focus for the study, ‘quick and dirty’ ethnography is capable of providing much valuable knowledge of the social organisation of work of a relatively large scale work setting in a relatively short space of time, and this includes what we were able to learn from the organisational problems that arose when trying to establish the research site. Indeed, it can be argued that the ‘pay off’ of the ‘quick and dirty’ ethnography is greater in that for time expended on fieldwork a great deal is learned. Second, such knowledge can be built upon for a more focused examination of the detailed aspects of the work which is more typical of ‘concurrent ethnography’. What the ‘quick and dirty’ fieldwork provides is the important broad understanding which is capable of sensitising designers particularly to issues which have a bearing on the acceptability and usability of an envisaged system rather than on the specifics of design. Both aspects, of course, are important.
The research also raised the problem of communicating the findings from the ethnographic study to designers, mainly because of the increased scale of the setting and the problems of finding a clear design focus. While the fieldworker learned a great deal in the study just discussed, certainly much that is useful for a sociological study, it proved difficult to hang this onto clearly formulated design objectives. In spite of this, even if used with this limitation ‘quick and dirty’ ethnography is capable of providing an informed sense of what the work is like in a way that can be useful for designers in scoping their design. In other words, although in our own case the research raised important questions about the initial design objectives, and this is not a pointless finding by any means, it did suggest useful ways in which ethnography could be used to provide designers with a better sense of the setting and its work activities.
No comments:
Post a Comment