Multimodality: language and the body

Aims

Aims

Most of our language use and acquisition happens through face-to-face interaction, which gives us access not only to what we and others say, but also how we use our bodies in interacting with others. A growing area of research explores how language is connected to gesture, facial expression and eye gaze: how are multiple channels of information like lexical choice, syntactic structure, intonational structure, and other aspects of linguistic production connected to the use of the body as we talk? How do we interact with one another and with objects in a physical world? How does this shape our language, and how does our language reflect the demands of being human beings in bodies, and in a physical world? In this module, we explore language as a multimodal phenomenon, grounding our work in everyday interactions, and using the methodological resources of conversation analysis, gesture studies, and phonetics.

At the end of this module you will be able to:

• recognise the different modalities that are relevant for language use in spoken face-to-face interaction, including: facial expression, manual gestures, phonetic production, eye gaze;
• use appropriate software to study visible bodily actions, including gesture and speech;
• be able to annotate and transcribe spoken and visible/gestural behaviours using standard transcription methods;
• write about a problem in face-to-face interaction, drawing on technical skills, and showing an appreciation of some of the debates around the topic.

You will know:

• various ways in which speakers use language and the body together as resources for meaning and action in face-to-face interaction;
• major theories developed to account for specific modalities such as facial expressions, manual gesture, speech and eye gaze;
• important empirical studies conducted on specific modalities and their coordination with language production.

This module will be capped at 20.

Prerequisites

Prerequisites

Students must have successfully completed at least one of:

• E/L34I Language of Turn and Sequence
• E/L64H Language as Action

Recommended co-requisite

It is recommended that students taking this module also take:

• E/L17H The prosody of English

Programme

Programme

Contact hours

In the spring term, there are two contact hours a week, in two streams: one is a lecture/seminar stream, the other a practical stream.

Lecture/seminar stream

Four topics will covered in two-week blocks. Lectures and seminars will be in alternate weeks (lecture one week, related seminar the next), and each two-week block will focus on a particular topic. The lecture will provide an overview of the main issues both in the literature and where there are gaps in current knowledge. The seminar will pick up issues from the lecture and will involve discussion of reading materials, exercises with data, and critical examination of the topic.

Practical stream

The practicals will be lab-based. They will require students to do some preparation and/or follow-up. They will run weekly for one hour. They will cover how to use appropriate software, and the analysis of data.

In the summer term, there is one weekly seminar, which focuses on supporting students for the final assessment.

Teaching programme

Practical stream

The practicals will cover the following topics:

• ELAN (software for the analysis and annotation of video data)
• Making suitable data extracts
• Making collections of comparable extracts for the purposes of analysis
• Practicalities and house-keeping while working with digital data: making video clips, file formats, saving data, etc.
• Annotation of data in ELAN and Praat; importing/exporting between these programs and others, including annotation files
• Transcription of data for presentation, using standard schemes for conversational data: Mondada’s scheme for multimodal transcription, GAT2 for prosodic features

Content

The content in the spring term will cover important topics in the analysis of face-to-face spoken interaction. The precise content will vary from year to year, but is likely to include:

• Foundations of multimodality (including the relations between spoken and signed languages, and the place of gesture)
• Gaze: its alignment to other events in interaction, its use in turn construction and the formation of social actions
• Manual gesture: its alignment to other events in interaction, its internal structure, and its relation to language production and processing
• Facial expression, including relations with prosodic features in the construction of turns at talk

In the summer term, we will focus on final projects, with activities that support students’ independent research. These will vary depending to the needs of the group but are likely to include:

• An essay planning session
• Critical thinking and writing exercises to support essay-writing
• “Shut up and write!” sessions
• Exploring relations between data and theory
• One-to-one meetings with the module tutors

Teaching materials

TBC

• Bavelas, J. B., Chovil, N., Lawrie, D. A., & Wade, A. (1992). Interactive gestures. Discourse Processes, 15, 469-489.
• Enfield, N. J. (2009). The Anatomy of Meaning. Cambridge: Cambridge University Press.
• Kendon, A. (2004). Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.
• Mondada, L. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20(3), 336–366. http://doi.org/10.1111/josl.1_12177

Assessment and feedback

Assessment and feedback

Feedback on formative work

• TBC

Summative assessment and feedback

• Short written exercise
  • Due: Week 5, Spring Term
  • Weight: 20%
  • Personalised written feedback within two weeks of submission
• Short written exercise
  • Due: Week 10, Spring Term
  • Weight: 30%
  • Personalised written feedback within two weeks of submission
• Project report
  • Due: Week 5, Summer Term
  • Weight: 50%
  • Personalised written feedback within four weeks of submission

Skills

Transferable skills developed in this module

All modules provide an opportunity to work on general oral/written communication skills (in class and in assessments) and general self management (organising your studies), alongside the specific skills in language or linguistics that the module teaches.

In addition, this module will allow you to particularly develop your skills in synthesis and application of data analysis. This module involves applying the analysis techniques learned in other modules to a new method of analysing everyday talk; being able to explicitly relate knowledge gained from different spheres of experience and knowledge (and thus confidently tackle unfamiliar problems you may meet in a job) is a valuable skill to bring to employers. You will also develop the ability to analyse and talk about the nonverbal behaviour of participants in social interaction, a skill which has practical applications in numerous professions.

Follow this link to hear how past students use transferable skills from their degree in their current jobs.