With the Smartphone as Field Assistant: Designing, Making, and Testing EthnoAlly, a Multimodal Tool for Conducting Serendipitous Ethnography in a Multisensory World

By Paolo S. H. Favero and Eva Theunissen, University of Antwerp

"One can now picture a future investigator in his laboratory.
His hands are free, and he is not anchored.
As he moves about and observes, he photographs and comments.
Time is automatically recorded to tie the two records together.
If he goes into the field, he may be connected by radio to his recorder.
As he ponders over his notes in the evening,
he again talks his comments into the record.
His typed record, as well as his photographs,
may both be in miniature, so that he projects them for examination."

– Vannevar Bush, "As We May Think" (1945)

As discussed in the print essay accompanying this piece, EthnoAlly[i] responds to the growing need for integrating digital tools (which are increasingly becoming ubiquitous in the lives of many citizens of the wired world) in the act of conducting ethnographic research. Born through the collaboration between an anthropologist and an engineer (Paolo Favero and Alfonso Bahillo), and developed with the help of a broader team including a PhD candidate in visual and digital cultures (Eva Theunissen) and a visual artist (Ali Zaidi), EthnoAlly is a research tool designed specifically for the purpose of helping anthropologists (and other social scientists) conducting sensory, participatory, and multimodal ethnographic research.

PRESENTING THE TOOL

EthnoAlly consists of an application that can be downloaded on smartphones—on iOS and Android—and a web page where the materials can be stored and viewed (see Figures 1 and 2). The original testing ground was the field of tourism, a context that, we believed, lends itself particularly well to this kind of experimentation. However, as we started actively designing the application, we realized that adapting the tool to the specificities of a single research setting might limit its applicability to ethnographic research at large. We therefore decided to focus exclusively on the specific needs of ethnographers, morphing the tool into a proper field assistant for researchers and their interlocutors. The present prototype is designed to fit:

Ethnographers – who receive full access to all the functionalities and, in particular, to the web platform on which materials can be visualized, archived, and analyzed.
Research participants – who get full access to the app (and can use it as a personal diary), but will only access one or two types of visualizations on the web, such as a video clip of their day or an interactive map.

EthnoAlly presents itself through a simple and user-friendly app interface combining geolocative information with the conventional multimedia functionalities that are present in most smartphones. These include:

GPS-based geotracking – activating this function, the researcher will be able to track their movements in space, including their speed, the length of their stops, etc. Such info will be visualized on the homepage through detailed maps.[ii]
Image-making functions – the app will be able to access the camera of the researchers' smartphones and make their still and moving images available to them on the web-based platform.
Textual notes – with the help of a pad accessible from within the app, the user will be able to take typewritten notes. These notes too will be geotagged and spatialized on the maps.
Audio notes – the user will be allowed to take voice memos. These will also be geotagged and made available for the researcher on the homepage.
We have just finalized a drawing function that allows researchers to make visual notes on top of photographs or on a blank document. Furthermore, we have inserted a voice-to-text transcription tool.

*Figure 3: One of the search strategies for browsing the material on EthnoAlly's web, by focusing on a particular user.*

In parallel to all this, EthnoAlly is also a participatory research tool. The ethnographers can upload the app onto their research participants' smartphones to use as a diary. They will be able to access the same smartphone interface, but they will not be able to access the web platform.

The material generated by the research participant will be uploaded onto the cloud that the researchers can access on the EthnoAlly web platform. This is the second and central feature of EthnoAlly. While there are a number of applications that allow users to produce and geotag a variety of audio-visual impressions, our tool allows greater methodological depth (see below for more details). EthnoAlly synchronizes all the materials collected within the app on a devoted online platform. Images (still and moving), aural and written notes, and the traces of movements made by the user can be visualized on the web page. On this page, the researcher will hence be able to collect, archive, organize, and analyze these data.

On the web page, the researcher can explore the materials by accessing either a map or an archive, through their metadata (time, space, username), or by browsing inserted keywords. Each of these search strategies will then also allow for more detailed searches.

When focusing on a specific user, the researcher will find clickable folders and materials revealing visual, aural, and geolocative information (Figure 3). We currently have two interfaces for consulting the material. The "show on map" interface offers an overview of the route a participant has walked, including the materials that were made on particular points on the track (Figure 4). These materials can be viewed in more depth by clicking on the respective symbols.

The "video mode" interface (see Figure 5) allows the researcher to browse the material by playing the video or by moving the cursor below from left to right.[iii] The web page is designed to allow the researcher to sit in front of the screen with research participants and to assist in eliciting further information from them, and hence for conducting what could be labeled as "multimodal elicitation." Next to the "video mode" and "show on map" interface, we are examining other possibilities regarding how to visualize the materials through different formats. This has prompted us to explore a range of questions, such as: Which search strategies should be included on the EthnoAlly online platform for the researcher to be able to explore the material in the most convenient and intelligible manner? What are the best ways to export the materials in other (and perhaps also printable) forms?

USING THE TOOL

Having given the background to the tool, let us now offer some insights into its various possible incorporations into the ethnographic field. Since Theunissen joined the project, in fall 2015, we started testing EthnoAlly in a variety of different contexts. We began by simply using it in a random fashion, keeping the mobile application running in our pockets when cycling to work, or testing it while going for a walk in the park. We then moved on to a more systematic set of experiments. We created seven drills, each addressing a different type of ethnographic engagement. The first four focussed on the use of EthnoAlly as a personal field assistant, while the others centered on using EthnoAlly as a participatory tool.

*Figure 6: EthnoAlly's maps displaying the video clips (left) and other media (right) as gathered during one of the two psychogeography walks.*

The purpose of the first two tests was to use the application to visualize and describe a defined neighborhood, one with which Theunissen was not familiar (Amsterdam), the other one with which she was highly familiar (Kiel, Antwerp). These two drills helped Theunissen record her tracks, take photographs and videos, and record sound notes in order to describe the sensorial characteristics of these places. The third and fourth tests were based on Favero's previous experiences of integrating GPS trackers in ethnographic practice (see Favero 2014). We designed two "psychogeography" walks. Using Google Maps, we printed a map of the center of Antwerp and then drew a circle around the cathedral (Figure 6; see below for an audio clip of the church bells). With a defined starting point, Theunissen followed this circle as closely as possible, looking for "focal points," such as textures, walls, graffiti, litter, conversations, people, rhythms, analogies, resemblances, and mood (Figure 7). The purpose of these two "psychogeography" walks was to employ EthnoAlly to explore the material, visual, and sensorial characteristics of the space touched upon (see videos clips 1, 2, and 3).

Tests five, six, and seven were focused on having the participants using the smartphone application. For the first two tests, Theunissen asked participants to wander for forty-five minutes starting from a location in Antwerp they had decided upon together. No strictly defined task was identified in advance. The participants were given only a few basic instructions on EthnoAlly functionalities and were told that they should try to take pictures and/or videos during their walk, but that the exact use and circumstances of taking pictures, making videos, and creating sounds or text notes was ultimately up to them. One participant was not familiar with smartphones and applications, while the other was a fervent iPhone user (Figure 8). The third and last of these tests could seem to constitute an instance of digitally supported "walking ethnography" (see, for example, Ingold and Vergunst 2008; Irving 2011, 2013; Pink 2007). Theunissen went with her participant to a place in Antwerp that the participant somehow found significant. During the tour, Theunissen let the participant linger on their memories of the place in order to discover something of their "lifeworld" (Merleau-Ponty [1962] 2012). The participant would also instruct Theunissen in making photographs and videos of the place.

Once this material was collected, the authors of this article sat together and explored the various insights they had gathered. They singled out a set of topics that were highlighted by the use of the app. They also identified the value of EthnoAlly as a companion for conducting visual and multisensory ethnography. In the coming sections, will offer some examples relating to this.

Figure 8: Participatory walk with Fleur (a food journalist by profession), who decided to do a culinary walk in her residential neighborhood. She visually documented some of the places she frequents (as shown in the two images on the right).

Walking Ethnographies

During the walking ethnography test with one participant (Annick), Theunissen witnessed how a visit to a church became a kind of "can opener" (Collier and Collier 1986, 23) for unleashing memories about childhood. Annick spontaneously began to reenact certain Catholic churchgoing memories, such as the purification ritual with holy water at the entrance into the church (Figure 9). As Farnell (1999) has noted, "walking through familiar landscapes can evoke physical memories of former acts that have eluded verbal memory" (1999, 354). Even though Annick never visited this specific church as a child, this place triggered reflections regarding the past. Revealing the centrality of the body as a repository of knowledge (cf. Bourdieu 1990), this experience provided Theunissen with "clues" (Ginsburg 2000), directing her attention to issues that would probably not have emerged during an interview session. This event testifies to the importance of participant observation and, in particular, to the capacity of visual observations to grant access to memories and emotions—that is, to those layers of human experience that may not be part of social actors' narrations of the self, but that may materialize in space or be triggered by specific locations and bodily feelings. It also shows how the act of "walking with" (Ingold and Vergunst 2008; Irving 2011, 2013; Pink 2007) enables us to "empathetically comprehend the experiences of those represented" (Pink 2007, 247). The smartphone and EthnoAlly helped Theunissen in mediating and managing this process, providing her with a tool for grasping serendipitously a moment that would perhaps have been difficult to capture with a larger camera. By using the app, and delegating to it the responsibility of stitching together the photographs, videos, notes, and geospatial data produced during that specific interaction, Theunissen was able to focus more on her interlocutor and the emotions that the participant was sharing with her.

The experience just described reproduced, interestingly, a key event that had led Favero to design the tool in the first place. The idea for an app capable of assisting the ethnographer came to him a couple of years earlier when he visited a small village located in the Alps, in the northwest of Italy, with his father. This was the ancestral place of his grandmother's family and also the temporary home for his father and his family when, in 1944, towards the end of WWII, they were forced to leave their town because of the bombings of the allied forces. This village became their rescue from the cruelties of the war, and Favero's father had some very dense memories of his time there (even though he was at the time just a thirteen-year-old boy). Reaching the place, Favero and his father stopped at the only bar in the village and then took a walk in its center. As they were walking, his father started sharing stories regarding his time there. Pointing at various houses and street corners, he would show him where the Nazis used to sleep as they were raiding the valleys; he took him to the beginning of the path into the forest that he would walk every second morning to get milk from the pastures; he showed him the corner in the yard where their Jewish neighbors were eating when the Nazis came to arrest them, etc. As his father started telling stories, Favero decided to activate the voice recorder on his iPhone (with his father's permission). He also took several snapshots of the places his father would point out to him. Indeed, he did not record any notes, but used his voice as a kind of track, repeating key names in order to remember them later on. At some point during this promenade, Favero decided to activate a movement tracker (Runtastic, a runners' app) in order to keep a memory of their route, which was progressively getting more complicated. But all of these technologies together made him lose his father and the intensity of the conversation. What if a single tool could actually coordinate that work so that the recorder of stories could stay with the storyteller? So that the ethnographers could allow themselves, to paraphrase MacDougall, to enter "into someone else's story" (1994, 35)? This is how EthnoAlly was born, with the ambition of helping the ethnographer in the task of getting into and feeling (and then later on analyzing) other people's stories.

*Figure 9: Walking ethnography. (Photograph by Theunissen)*

As the above "walking ethnography" test illustrates, the adoption of EthnoAlly in the field has the potential of directing the attention of researchers towards layers of meaning that would probably not have emerged during a structured or semistructured face-to-face interview. Such experiments, in fact, provide insight into "embodied knowledge" (Bourdieu 1990) that emerges at the intersection of material places, bodily/sensory experiences, and narration.

EXPLORING THE FLEETING AND INACCESSIBLE

EthnoAlly functions not only as a personal diary for the ethnographer but also as a participatory tool. We believe that such potential can also be employed in a number of new territories beyond conventional one-on-one or group situations and in contexts that can at times be difficult to access for the ethnographer. We are presently exploring possible expansions in fieldsites that would benefit from the collaboration with scholars from a variety of different fields (mainly from the medical and behavioral sciences). To mention a few examples, EthnoAlly could facilitate the work of those ethnographers engaging with people living with medical conditions associated with stigma (such as HIV/AIDS).

In such contexts, the regular presence of an ethnographer may indeed jeopardize the privacy of the interlocutor by attracting the curiosity of passers-by or other individuals encountered in everyday life. Our digital tool may help solve this ethical predicament. It can facilitate, for instance, the collection of ethnographic information in the shape of a geolocated multimodal diary, which may at a later stage be explored together with the creators of the materials. Along a parallel track, we also envision the possibility of engaging with this tool for the study of events such as panic attacks.[iv] Occurring at the most unexpected moments in an individual's life, panic attacks are difficult for qualitative researchers to observe and would almost require a 24/7 presence. In this case, EthnoAlly may provide the researcher with a kind of proxy to this experience, allowing a research participant to upload a visual diary that can be remotely explored in real time by the researcher. Research participants and researchers may also engage in a direct conversation at that particular moment by using the smartphone's conventional communication tools. We are finding new synergies for small-scale projects capable of testing the tool in contexts where the presence of an ethnographer may be either impossible or may expose marginalized subjects to risky situations. In addition, we are expanding our explorations of the use of the sensors embedded in smartphones for monitoring such processes (for more reflections on this, see Favero 2014). The use of sweat, heart, blood pressure, and brain wave sensors may help reveal significant moments in the context of some of these experiences, whose cultural meaning could be explored by the researcher at a later stage with the help of what we have called "multimodal elicitation techniques." Sitting together with the research participant, the ethnographers can explore the charts and other materials generated by the combination of these tools and then discuss them together with the person in question, identifying critical moments, turning points, etc. The combination of the use of emerging technologies with established ethnographic tools (such as elicitation techniques) can provide us access to the affective and embodied dimensions of experiences that we conventionally address only through (and reduce to) narration.

CROWDSOURCING ETHNOGRAPHY

During a workshop held in Melbourne at RMIT in 2016, Favero was given the opportunity to explore the potential of EthnoAlly to generate large databases resulting from group explorations of the same topic and/or geographical areas. Building upon his previous experiences of organizing ethnographic group missions employing different tools and techniques of audio-visual recording (see Favero 2017), Favero asked the participants of this one-day workshop devoted to EthnoAlly to engage with the tool during a walk around the RMIT neighborhood. Upon returning from their walk, the group would download the materials and share them with each other by looking at the path each participant conducted on the EthnoAlly homepage. Reviewing these materials, the participants could get a sense of the richness and diversity of stories and angles grasped by the group as a whole. A short walk could reveal the diverse and layered character of the neighborhood and also show the extent to which visual and sensory observations responded to the participants' own inclinations. While a number of the materials gathered highlighted politics, others looked more into form or materiality. Attention to matters of form and materiality was also acknowledged during Theunissen's walks (see video clip).

Favero found one set of materials particularly revealing, as it highlighted the tool's capacity to generate material useful for igniting group discussions. One participant had, in a ten-minute walk, managed to collect evidence about the symbolic significance for Aboriginal Australians of the neighborhood in which the RMIT offices had been built (see video clip). Sitting there among the participants, facing the screen where Favero projected the material in video mode, this participant commented on the walk. Using videos, photos, and the map itself, she literally took us by the hand into the various confrontations that make up the history of Australia's tense relations to its Indigenous populations and into the struggles that had taken place in this particular neighborhood. She gave meaning to those visual and aural signs that Favero had not noticed during his own walks in the neighborhood and provided useful material for addressing the topic with the broader group of participants.

Going further, and envisioning the above-mentioned incorporation of sensors, we are also exploring the extent to which EthnoAlly could be adopted for monitoring other experiences that are conventionally out of reach for ethnographers. For instance, we could generate crowdsourced databases of ethnographic material regarding earthquakes or other natural disasters, follow individuals during rescue operations, mountaineers during dangerous climbs, etc. If carefully incorporated into established methodologies, digital tools promise to open up a terrain of insights previously available to us only in their aftermath and hence in the presence of already consolidated narrations and rationalizations. EthnoAlly can bring us into the here and now of human experiences and open up insights into the fleeting, the momentary, the affective, the bodily, and the unverbalized.

AUDIO CLIP

NOTES

EthnoAlly has been incorporated into MIT’sDocuBase (https://docubase.mit.edu/tools/ethnoally/). To participate in the development of EthnoAlly, please email to Paolo (paolo.favero@uantwerpen.be), Alfonso (alfonso.bahillo@deusto.es) or Eva (eva.theunissen@uantwerpen.be).

Click here for a video of the walking time-lapse.

[i] The first prototype of the EthnoAlly was developed within the framework of a research proposal entitled "The Media Tourist" dealing with tourist practices in ICT-mediated environments (that is, in contexts with a high density of exposure to information and communication technologies) and sponsored by the framework of the FWO (the Research Foundation Flanders). The idea was to create a tool able to be used by both researchers in the field of tourism and tourists.

[ii] The app is meant to work in both online and offline mode, allowing users to work with it even in the absence of a WiFi or 3G signal.

[iii] This mode is one of the key arenas of our attention and we hope to refine and develop it further in the future, as it is a format that may be particularly attractive to use in an interview setting.

[iv] One app addressing panic attacks was recently launched by a young designer, see reference list (Dunn 2016).

REFERENCES CITED

Bourdieu, Pierre 1990. The Logic of Practice. Cambridge: Polity Press

Collier, John, and Malcolm Collier. 1986. Visual Anthropology: Photography as a Research Method. Albuquerque: University of New Mexico Press.

Dunn, Thom. 2015. "He Made an App to Help Himself and People Like Him with Panic Attacks." Upworthy website, July 21. http://www.upworthy.com/an-autistic-man-made-an-app-to-help-people-help-him-during-panic-attacks.

Farnell, Brenda. 1999. "Movies Bodies, Acting Selves." Annual Review of Anthropology 28:341-73.

Favero, Paolo. 2014. "Learning to Look Beyond the Frame: Reflections on the Changing Meaning of Images in the Age of Digital Media Practices." Visual Studies 29 (2):166-79.

Favero, Paolo. 2017. "Curating and Exhibiting Ethnographic Evidence: Reflections on Teaching and Displaying with the Help of Emerging Technologies." In The Routledge Companion to Digital Ethnography, edited by L. Hjorth, H. Horst, A. Galloway, and G. Bell, 275-87. London: Routledge.

Ingold, Tim, and Jo Lee Vergunst. 2008. Ways of Walking: Ethnography and Practice on Foot. Hampshire: Ashgate.

Irving, Andrew. 2011. "Strange Distance: Towards an Anthropology of Interior Dialogue." Medical Anthropology Quarterly 25 (1): 22-44.

Irving, Andrew. 2013. "Bridging Senses and Society." The Senses & Society 8 (3): 290-313.

McDougall, David. 1994. "Whose Story Is It?" In Visualizing Theory, edited by L. Taylor, 27-36. New York: Routledge.

Merleau-Ponty, Maurice. (1962) 2012. Phenomenology of Perception. New York: Routledge.

Pink, Sarah. 2007. "Walking with Video." Visual Studies 22 (3): 240-52.

CITE AS

Favero, Paolo S. H., and Eva Theunissen. 2018. “With the Smartphone as Field Assistant: Designing, Making, and Testing EthnoAlly, a Multimodal Tool for Conducting Serendipitous Ethnography in a Multisensory World.” American Anthropologist website, February 21.