Electro-Haptics Research

A new approach to implant listening

Hearing loss is a global challenge, particularly in countries with ageing populations. Unaddressed hearing loss poses an annual global cost of $750 billion and recent work has shown a strong link between hearing loss and dementia. Over a billion young people are at risk of hearing loss because of the way they listen to music.

Hundreds of thousands of people depend on implanted electronic devices to hear. These devices, known as auditory or cochlear implants, aren't perfect. Implant users often have difficulty knowing where sounds originate, are commonly unable to enjoy music, and find it difficult to understand speech when there is background noise - like in a busy workplace, crowded restaurant or chaotic classroom. We have a new approach to solve this problem by transferring crucial sound information through the skin.

We are developing a low-cost haptic device that could revolutionise the treatment of hearing loss.

Implants in noise

Cochlear implant users struggle to understand speech in noisy environments.

People with auditory implants hear the world in a very different way to people with healthy hearing. In an implant user, the sound that is usually transmitted to the brain by thousands of extraordinarily sensitive cells in the ear is instead transmitted by just 22 micro-electrodes. This means that the information transmitted to the brain is severely limited.

We've made a quick demo that simulates how hard it can be for auditory implant users to understand speech in complex sound environments. It's available to use for free as a teaching or demonstrating tool (YouTube or Download).

Note: these are just simulations based on models of how cochlear implants work with the brain. Cochlear implant users experience their device in different ways depending on a range of factors.

Implants and Music

Note: these are just simulations based on models of how cochlear implants work with the brain. Cochlear implant users experience their device in different ways depending on a range of factors.

Music sounds very different to implant users. They can struggle to distinguish different pitches, and they have poor access to the quality, or "timbre", of sounds.

This short demo that simulates what it is like to hear music as a auditory implant user. Like the demo above, it's available to use for free as a teaching or demonstrating tool (YouTube or Download).

Hearing through your skin

"The brain takes information from the senses to build a model of the world. When information is missing from one sense, we can use another sense to add it back in." - Dr Mark Fletcher, principal investigator

Our research is all about finding ways to send crucial sound information through the skin as small vibrations. We've shown that this can work in the lab–now we're taking it to the real world.

We are developing a wrist-worn device that we have shown can improve implant users’ speech understanding and could transform their lives.

Our new haptic device

3D render of our mosaicOne_C device. We've been working hard to improve our device and get it ready to be used outside of the lab. Further iterations of the mosaicOne will be coming very soon!

Discreet: We're developing a slim, discreet wrist worn device that won't get in the way of daily activities.
Precise: we're focused on making sure our vibrations are reproduced perfectly in sync with the sound, without any distortion of the signal.
Comfortable: a priority in our design and manufacture process has been making the mosaicOne lightweight and comfortable to wear for long periods.
Quiet: our algorithms and motors have been carefully developed so our device is near-silent.

This short video shows people reacting when listening to music and feeling it through the first prototype of our mosaicOne_A haptic device.

Training at home

Learning to use new information is an important part of the electro-haptic effect. We're building a remote training system to see how people learn to use electo-haptic stimulation over time.

Our research uses RealSpeech, an auditory training app developed by Dr Mark Fletcher and Dr Ian Wiggins from the electro-haptics team. RealSpeech has a large library of talkers and background environments, over 20 hours of synchronised high-definition video and audio, and is set up for remote data collection.

Virtual acoustics

Working with the University of Southampton Virtual Acoustics and Audio Engineering Research Group, we're developing a system for creating carefully controlled realistic sound environments for use in the lab, in the clinic, and at home. Currently, a large, acoustically-treated space and an expensive system with several loudspeakers in a ring around the listener is required to create such sound environments. We hope our system can opening up the possibility of more realistic testing much more widely for clinics and research labs and allow for much more sophisticated at-home training and testing.

Peer-reviewed publications

2021

Mark's review paper on the use of haptics to enhance music perception in hearing impaired listeners, published in Frontiers in Neuroscience

Fletcher, M.D. (2021) Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners? Frontiers in Neuroscience

Our review paper on the state of the emerging field of electro-haptics in audiology, published in Frontiers in Neuroscience

Fletcher, M.D. & Verschuur, C.A. (2021) Electro-Haptic Stimulation: A New Approach for Improving Cochlear-Implant Listening. Frontiers in Neuroscience

Our first paper published in Sensors

Fletcher, M.D., Zgheib, J., & Perry, S.W. (2021) Sensitivity to haptic sound-localization cues at different body locations. Sensors (pdf)

Our first paper of 2021, published in Nature Scientific Reports

Fletcher, M.D., Zgheib, J., & Perry, S.W. (2021) Sensitivity to haptic sound-localisation cues. Nature Scientific Reports (pdf)

2020

Our 2020 review paper on the challenges of developing a new haptic device, published in Expert Review of Medical Devices

Fletcher, M.D. (2020) Using haptic stimulation to enhance auditory perception in hearing-impaired listeners. Expert Review of Medical Devices (pdf)

Our forth paper of 2020, published in Nature Scientific Reports

Fletcher, M.D. & Zgheib, J. (2020) Haptic sound‑localisation for use in cochlear implant and hearing‑aid users. Nature Scientific Reports (pdf)

Our third paper of 2020, published in Nature Scientific Reports

Fletcher, M.D., Song, H., & Perry, S.W. (2020) Electro-haptic stimulation enhances speech recognition in spatially separated noise for cochlear implant users. Nature Scientific Reports (pdf)

Our second paper of 2020, published in Nature Scientific Reports

Fletcher, M.D., Thini, N., & Perry, S.W. (2020) Enhanced pitch discrimination for cochlear implant users with a new haptic neuroprosthetic. Nature Scientific Reports (pdf)

Our first paper of 2020, published in Nature Scientific Reports

Fletcher, M.D., Cunningham, R.O., & Mills, S.R. (2020) Electro-haptic enhancement of spatial hearing in cochlear implant users. Nature Scientific Reports (pdf)

2019

Our 2019 paper in Nature Scientific Reports

Fletcher, M.D., Hadeedi, A., Goehring, T., & Mills, S.R. (2019) Electro-haptic enhancement of speech-in-noise performance in cochlear implant users. Nature Scientific Reports (pdf)

2018

Our first EHS paper, published in Trends in Hearing

Fletcher, M.D., Mills, S.R., & Goehring, T. (2018) Vibro-tactile enhancement of speech intelligibility in multitalker noise for simulated cochlear implant listening. Trends in Hearing (pdf)
In the top 3% of the 12.5 million research outputs scored by Altmetric
2nd most discussed article ever published in Trends in Hearing

Presentations

2021

Mark's invited talk at the Music & CI Symposium, Cambridge University (UK), on enhancing music perception in CI users using haptics

Thank you very much to the organising committee and to Oticon for the invitation
This talk discussed work in collaboration with Ama Hadeedi, Robyn Cunningham, Jana Zgheib, Nour Tahini, Vinson Song, Tobias Goehring, Sean Mills, and Sam Perry

Mark & Sam's invited talk at the Center for Music in the Brain at Aarhus University (Denmark), on enhancing cochlear implant listening using haptics

Thank you very much to Bjorn Peterson and everyone at Aarhus for being such welcoming hosts
This talk discussed work in collaboration with Ama Hadeedi, Robyn Cunningham, Jana Zgheib, Nour Tahini, Vinson Song, Tobias Goehring, and Sean Mills

Sam's poster at the Music & CI Symposium, Cambridge University (UK), on enhancing pitch discrimination with haptics

The described work in collaboration with Mark Fletcher and Nour Tahini

CambridgeCIMusicPoster
File Size:	462 kb
File Type:	pdf

Download File

Mark's invited talk at the Conference on Implantable Auditory Prostheses

Thank you very much to the CIAP organising committee for the invitation
This talk discussed work in collaboration with Ama Hadeedi, Robyn Cunningham, Jana Zgheib, and Sam Perry
Click to see a recording of this talk (Q&A forum questions and answers are shown in the video description)

Mark's invited talk at the Oticon Shaping the Future study day, London

Thank you very much to Oticon and Andrew Causon for the invitation

Mark's electro-haptics guest lecture at the Nottingham Biomedical Research Centre

Thank you to Joe Sollini and everyone at the NBRC for hosting

2020

Mark talks about improving sound localisation in cochlear implant users with haptics at the British Cochlear Implant Group meeting

Thank you to the University of Nottingham for hosting a wonderful event

Our visit to Oticon Medical and the Danish Technical University to formally kick-off the Electro-Haptics Research Project

Thank you to Soren Riis, Kathleen Faulkner Scalzo, and Jeremy Marozeau for being wonderful hosts and to everyone else on the EHS project team for a great meeting

Mark's electro-haptics guest lecture for the "Current Developments in Bioengineering" 3rd year BEng module at Nottingham Trent University School of Engineering and Technology

Thank you to Fred Vanheusden and everyone at NTU for hosting

2019

Our talk at the National Cochlear Implant Users’ Association AGM

Thank you to the National Cochlear Implant Users' Association for hosting Carl's talk on the project

Electro-haptics visits University College London

Thank you to everyone at UCL Speech, Hearing and Phonetic Sciences (SHaPS) for hosting Mark's talk on Electrohaptics research

Mark's talk at Nottingham Trent University School of Engineering and Technology

Thank you to everyone at Nottingham Trent for hosting our talk on Electro-Haptics

2018

Our presentation at the British Society of Audiology

Fletcher, M., Hadeedi, A., Goehring, T., & Mills, S. (2018). Using tactile stimulation to improve speech-in-noise performance in cochlear implant users. British Society of Audiology Basic Auditory Science 2018

Our new device at the British Cochlear Implant Group meeting

The mosaicOne prototype shakes things up at BCIG
Our new poster from the meeting (pdf)

Media and outreach

Live webinar for school children on our research (2021)

Based on upcoming article in Frontiers for Young Minds "Listening with your Wrists"
Thank you to Dr Hannah Semeraro for leading the event

Interview with Mark in The Hearing Journal (2021)

The article "Leaps in Haptic Technology Improve CI outcomes", focuses on our recent review paper in Frontiers in Neuroscience and the origins of the EHS project

The EHS team support the Robosapiens permanent exhibition by the Science Museum of Minnesota

Mark and Sam help curate the part of the exhibition focusing on cochlear implants (set to open in 2022)
Sam develops an advanced real-time cochlear implant simulator app for the exhibition

The EHS project featured in the British Society of Audiology magazine Audacity

Piece authored by Professor Carl Verschuur (page 44) highlights work on the EHS project as well as other important work going on within the University of Southampton Auditory Implant Service

Our latest work featured in The Conversation

Vibration on the skin helps hearing-impaired people locate sounds, The Conversation, February 2020

Mark talks about neuroscience and the Electro-Haptics Project to secondary-school students for the Smallpiece Trust

Thank you to everyone for coming and to the Smallpiece Trust and University of Southampton for organising this wonderful event.

Our work featured in The Conversation

Research workshop at the University of Cambridge

Electro-Haptics Workshop at the MRC Cognition and Brain Sciences Unit at the University of Cambridge. Working with researchers from the Cambridge Hearing network and a visiting group from the Laboratoire de Mécanique et d’Acoustique, CNRS, Marseille.

EHS featured in Hoorzaken

"Bracelet makes cochlear implant wearers hear better" (translated from Dutch)
Thank you to Rene van der Wilk for sharing our work.

Some of the professional bodies that have shared our work:

Funding

£450k award by the William Demant Foundation to explore electro-haptic enhancement of music perception (2021)

The project will focus on haptic enhancement of music perception and how audio and haptic stimulation are combined in the brain
This project is a collaboration between Mark Fletcher and Sam Perry (University of Southampton Auditory Implant Service and Institute of Sound and Vibration Research) and Peter Vuust and Bjørn Petersen (Centre for Music in the Brain and Royal Academy of Music, Aarhus, Demark)

£35k invested into the Electro-haptics project by the University of Southampton Auditory Implant Service and Faculty of Engineering and Physical Sciences (2019-2021)

Our particular thanks to Carl Verschuur for his support
This funding was critical for us to build world-class laboratory facilities for this project

EPSRC Doctoral Prize awarded to Sam Perry, allowing him to join EHS as a Research Fellow (2021)

Sam will be working on the EHS project for 1 year and 3 months on the doctoral prize
Additional funding has been provided by the University of Southampton Auditory Implant Service (extending Sam's time by 3 months) and Oticon Medical (£5k for equipment and travel)

£1k equipment fund to seed a collaboration between the University of Southampton and the University of Iowa awarded by the Iowa Neuroscience Institute (2020)

Thank you to the Iowa Neuroscience Institute for supporting this project through their Barrier Buster Grant.
We are excited to begin our collaboration with Dr Joel Berger and Dr Zsuzsanna Kocsis, using measurements from cortical electrodes in human subjects to explore the underlying mechanisms of haptic enhancement of listening.

£500k grant to support Electro-Haptics Research (2019)

The Electro-Haptics Research project is now supported by joint grants awarded by the Oticon Foundation and the Danish Innovation Fund
The new multi-centre project is led by Dr Mark Fletcher (University of Southampton), with a team from the University of Southampton, University of Cambridge, University of Iceland, University of Nottingham, Danish Technical University and Oticon Medical.
Researchers will be working on ways to improve CI listening using a new haptic device.

£155k grant awarded to support Ahmed Bin Afif, our new PhD student (2019)

The Electro-Haptics Research project has a new Post-Graduate Researcher, Ahmed Bin Afif.
Funding from King Saud University to support his research into improving CI hearing with haptic stimulation.

£80k grant to support our Virtual Acoustics project (2019)

The Electro-Haptics Research project has a post-doctoral researcher, Eric Hamdan, thanks to funding awarded by the Oticon Foundation.
The team now also includes Associate Professor Filippo Fazi who, leads the University of Southampton Virtual Acoustics and Audio Engineering Research Group, and Professor Nicci Campbell from the University of Southampton Auditory Implant Service, who's research focuses on the development of more realistic listening tests.

£10k to fund an internship for 6 months on the electro-haptics project (2019)

We welcome Jana Zgheib who is joining us for 6 months, thanks to £8k of funding from Oticon Medical and £2k of funding from the University of Southampton Auditory Implant Service.
Jana will be looking at enhancement of spatial hearing using EHS.