The first study we have prepared as part of the DigiTOP project explores the potential of using physiological data to detect mind wandering episodes; instances when the attention of a person is no longer entirely on the primary task that they are performing. Think of, for example, of any day to day task that requires very little of your cognitive resources to complete. During these sorts of tasks, it sometimes happens that our attention shifts towards thoughts that are not related to the task we were undertaking. This is entirely fine of course and often desirable, but what if the primary task was safety critical?
In many situations, a person’s limitations with regards to a task may be obvious (someone carrying a tea tray, will most likely have difficulties texting) but there could be other subtler situations in which the limitations are not of physical nature and not so easy to detect.
We as humans are generally very good at being able to read complex behaviours, when for example someone is distracted, or their attention is focused on something else and we make use of these abilities when collaborating. In an environment where humans and automation will work together on ever more complex tasks, being able to account for these sorts of behaviours will represent a great advantage.
Developed in collaboration with DigiTOP project members, the first study of Work Package 3 explores the use of different physiological sensors to detect mind wandering episodes. We monitor physiological parameters such as facial temperature, blink rate, blood oxygenation in various areas of the brain (using fNIRS – functional near-infrared spectroscopy) as well as various other parameters while the participants perform a visual inspection task. The attached image represents a fusion of data from a visual and a thermal camera showing a person wearing the fNIRS sensor together with the facial landmark tracking¹ that is used to extract blink rate and facial features.
The task involves looking at images of cork coasters and reporting on the ones that have defects. During the task, at random intervals, we ask questions to probe where the participants’ attentions were directed while also asking participants to self-report on mind wandering episodes at any moment. This first study takes place in a laboratory environment where we can keep most external factors under control; one such example would be noise. If the results are promising, future studies will be aimed at deploying these measures in a real-world manufacturing context.
What we hope to achieve is a better understanding of the relationship between the physiological response and the occurrence of mind wandering episodes. Our analysis will seek to identify features in the physiological data that could reliably indicate the occurrence of such episodes, which could help with developing better tools to characterise this phenomenon. This in turn could play a part in characterising different types of tasks; such tools could also be used to mediate the interaction between humans and advanced manufacturing systems.
Dr Adrian Marinescu
University of Nottingham
¹E.Sanchez-Lozano, G. Tzimiropoulos, B. Martinez, F. De la Torre, M. Valstar. A Functional Regression approach to facial landmark tracking. IEEE TPAMI. 2017