Studying at York
Multimodality: Language & the Body - LAN00066H
Module summary
Most of our language use and acquisition happens through face-to-face interaction, which gives us access not only to what we and others say, but also how we use our bodies in interacting with others. In this module, we explore language as a multimodal phenomenon, grounding our work in everyday interactions, and using the methodological tools of multimodal conversation analysis, gesture studies, and the phonetics of talk-in-interaction.
Related modules
Pre-requisite modules
Co-requisite modules
- None
Prohibited combinations
- None
Module will run
| Occurrence | Teaching period |
|---|---|
| A | Semester 2 2024-25 |
Module aims
The module aims at introducing students to a multimodal perspective to language and language use in interaction, as well as to the methods and tools used to study different kinds of vocal and visual behaviour that are communicative. Using methods from conversation analysis, gesture studies, and the phonetics of talk-in-interaction, this module allows students to register, describe, and transcribe visual behaviour (gaze, hand gestures, body posture, among others) alongside speech and how they are used by participants in interaction to get things done.
In addition, this module aims to develop students’ skills in synthesis and application of data analysis and communicate findings and conclusions in speech and writing using appropriate conventions. This module involves applying the analysis techniques learned in other modules to a new method of analysing everyday talk; being able to explicitly relate knowledge gained from different spheres of experience and knowledge (and thus confidently tackle unfamiliar problems you may meet in a job). Students will also develop the ability to analyse and talk about the nonverbal behaviour of participants in social interaction, a skill which has practical applications in numerous professions.
Module learning outcomes
At the end of this module students will be able to:
-
recognise the different modalities that are relevant for language use in spoken face-to-face interaction;
-
use appropriate software to study visible bodily actions, including gesture and speech;
-
be able to annotate and transcribe spoken and visible/gestural behaviour using standard transcription methods;
-
write about a problem in face-to-face interaction, drawing on technical skills, and showing an appreciation of some of the debates around the topic.
At the end of the module, students will know:
-
various ways in which speakers use language and the body together as resources for meaning and action in face-to-face interaction;
-
major theories developed to account for specific modalities such as manual gesture and eye gaze;
-
important empirical studies conducted on specific modalities and their coordination with language production.
Module content
The module will cover a range of topics related to multimodality, including:
-
The multimodal view of language
-
Multimodal transcription and annotation
-
The definition, description, and analysis of gesture
-
Temporal alignment of speech and gesture, including visual prosody
-
Gaze and body posture in interaction
-
The development and analysis of multimodal collections of data extracts
Indicative assessment
| Task | % of module mark |
|---|---|
| Essay/coursework | 100 |
Special assessment rules
None
Indicative reassessment
| Task | % of module mark |
|---|---|
| Essay/coursework | 100 |
Module feedback
Students will receive feedback within 25 working days. Feedback on formative assessment submitted on W4 will be returned on Week 6. Feedback on summative project report will be returned 25 working days after.
Indicative reading
This list illustrates some general readings assigned for the module. Depending on the year’s theme, other readings specific to the practice that will be studied will be added.
Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for Linguists: A Handy Primer. Language and Linguistics Compass, 9(11), 437–451. https://doi.org/10.1111/lnc3.12168
Cantarutti. (2021). Co-animation and the Multimodal Management of Contextualisation Problems when Jointly “Doing Being” Others. Social Interaction. Video-Based Studies of Human Sociality, 4(4). https://doi.org/10.7146/si.v4i4.128166
Hoey, & Kendrick, K. (2017). Conversation Analysis. In A. M. B. de Groot, P. Hagoort, A. M. B. de Groot, & P. Hagoort (Eds.), Research methods in psycholinguistics and the neurobiology of language : a practical guide (p. 1 online resource). John Wiley & Sons Inc
Levinson, S. C. (2013). Action formation and ascription. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp. 101–130). Malden: Blackwell Publishing Ltd.
Lilja, N., & Piirainen-Marsh, A. (2019). How Hand Gestures Contribute to Action Ascription. Research on Language and Social Interaction, 52(4), 343–364.
Loehr. (2012). Temporal, structural, and pragmatic synchrony between intonation and gesture. Laboratory Phonology. https://doi.org/https://doi.org/10.1515/lp-2012-0006
McNeill. (1992). 3. Guide to Gesture Classification, Transcription, and Distribution. In McNeill, Hand and Mind. What Gestures Reveal about Thought. (pp. 75–104). https://pages.ucsd.edu/~bkbergen/cogs200/McNeill_CH3_PS.pdf
Mondada, L. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20(3), 336–366. https://doi.org/10.1111/josl.1_12177
Mondada, L. (2019a). Contemporary issues in conversation analysis: Embodiment and materiality, multimodality and multisensoriality in social interaction. Journal of Pragmatics, 145, 47–62. https://doi.org/10.1016/j.pragma.2019.01.016
Mondada, L. (2019b). Conventions for multimodal transcription. Available from https://www.lorenzamondada.net/multimodal-transcription.
Perniss, P. (2018). Why We Should Study Multimodal Language. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.01109
Rossano. (2012). Gaze in Conversation. In Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp. 308–329). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118325001.ch15
Sidnell. (2013). Basic Conversation Analytic Methods. In Sidnell, T. Stivers, J. Sidnell, & T. Stivers (Eds.), The handbook of conversation analysis (pp. 77–99). Wiley-Blackwell. https://doi.org/10.1002/9781118325001
Stivers, T., & Sidnell, J. (2005). Introduction: Multimodal interaction. Semiotica, 2005(156), 1–20. https://doi.org/10.1515/semi.2005.2005.156.1
Streeck, J., Goodwin, C., & LaBaron, C. (Eds.). (2011). Embodied Interaction: Language and Body in the Material World. Cambridge University Press.