☎️ E15: In Conversation with Wisear on The State of Hearables, Brain Computer Interfaces and Augmented Reality 👂🎧🧠
The convergence of augmented reality and brain-computer interfaces
Every week, I do a Briefing Note on one important technology, from Vector Databases, LLMs, and Decentralised AI to Optical, Neuromorphic, and Analog Computing. If you’re new, join the World’s best investors and subscribe.
Those of you who know me know that I’m unreasonably interested in audio. No one else seems to care as much as me. Well, aside from Luis on the team. I/we are not reflective of the broader population. I bought the HTC Sensation XE with “Beats Audio”. This was before Apple bought Beats. I wrote something terrible back then about how audio was the competitive differentiator that would push HTC to compete with Apple. I used Tidal Pro because 24-bit/96kHz!
But, like, fine, no one else cares. HTC and Tidal are dead, and everyone cares more about photos and cameras. I think this bias leads to an overweighting on the eyes as the obvious AR platform. Why do we associate AR with vision? AR is just augmented reality. It can be noise-cancelling headphones or artificial olfactory sensors (aka e-nose). I think the best place for AR today is in the ears. We don’t need to buy a new bulky device. We don’t need to convince billions of people to wear glasses. We can use a device everyone has been comfortable with for decades: headphones. Or, more specifically, in-ear headphones.
And what’s interesting is that it turns out we can measure a lot with in-ear sensors. There is an excellent paper on in-ear infrasonic hemodynography (IH), which is as good as ECG in measuring heart rate and heart rhythm measurements. Yes, you heard correctly; we can also measure heart rate using earbuds, as with an ECG. You can also measure body temperature, blood oxygenation (SpO2), blood pressure, and brain activity (EEG).
Hearables are the entry point to more sophisticated brain-computer interfaces. Come for the heart rate monitor; stay for the silent speech. So, I spoke with Yacine and Alain of Wisear, a company building a neural interface through earbuds. Welcome to the Future.
☎️E15: In Conversation with Wisear on The State of Hearables, Brain-Computer Interfaces and Augmented Reality👂🎧 🧠
⏰ Timing: Brain-computer interfaces (BCI) still *feels* far away. The consensus is it’s the next UX after AR. In my head, I was distinguishing the 2025-2033 period as AR with eye tracking/micro-gestures/voice. And 2030+ as AR with non-invasive, fast, high-quality brain-computer interfaces. Wisear thinks that’s wrong, and we can incorporate neural interfaces into UX from 2025 onwards.
👂Non-invasive versus invasive: The team believes you can get a good enough signal from ear biosignals (non-invasive) to bring consumer neural interfaces to market. This contrasts with many in the industry who believe you need an invasive device to get strong enough signals. I still come down on the side of non-invasive commands being wrong too much and a bit slow for mainstream adoption, like Siri in 2012. But I think progress will be much faster than NLP now we have foundational models; we need more data…
📊Data acquisition: Non-invasive devices have a huge scale advantage in terms of data. Non-invasive consumer devices will be cheaper and scale faster. If the first use cases are valuable enough even at poor(ish) quality, then the first-to-market device has a data collection advantage. Just like Google has a data advantage in AI because it won in search, something is compelling about the company that starts collecting brain data at scale and will have the opportunity to own the dominant UX in the 2030s.
🍎Apple: Apple may have already won. Wouldn’t that be wild? A company founded in 1976 may be the best positioned to dominate consumer and health electronics in 2036. It happened in Oil & Gas, Automotive and Industrials so it might not be a surprise. The predictive eye tracking hints at a proto-BCI, but I’m more impressed by the idea of a body area network that no one else can match. The Vision Pro, AirPods, Apple Watch and iPhone will likely create a body area network to balance processing and power consumption.
🎧AirPods: There will be no new brain-computer interface device, at least in consumers. Glasses and earbuds will collect different biosignals, enabling seamless UX for digital selection, navigation and complex commands. We already have the first BCI in the market today: the AirPods.
Lawrence: What is Wisear?
Yacine: We build the future of human-machine interface. We believe a new generation of computing platforms is coming. It will be called augmented reality or spatial computing, as Apple called it. And it requires us to reinvent the way we interact with machines. We must interact while being totally mobile with a screen in front of our eyes. And we think the only way to solve this is with neural interfaces.
So we come and place small sensors called electrodes on your everyday devices such as your wireless earphones. And with that, we can capture the bioelectrical activity of your body. That means that when you're moving your eyes in any direction, we can capture it. We can capture your facial muscle activity and, tomorrow, your brain activity and transform that into completely hands-free and silent controls for the users.
Lawrence: Is the sensor data you're collecting analog? How are you doing the analog-to-electrical conversation? How does that impact the efficiency of the system?
Alain: The signals originate from electrical activity in your muscles, eyes, or brain. Local variations in electric potential are carried throughout your whole body by ions. We place electrodes on your skin to act as sensors that convert the ionic currents into electronic currents. These electronic signals then go through an amplification stage and analog-to-digital converter, becoming numerical signals that can be processed on a chip.
Lawrence: Right, okay, so it’s a standard digital chip, as you would expect. I often hear about the need for analog or neuromorphic chips for the type of always-on biosignal collection device you need. How much of a bottleneck is the chip in terms of power consumption?
Alain: So the most significant constraint with analog processing is that you would need a lot of components to build, for example, filters that would be as accurate as what you can do on the digital side. And those components, being expensive, are big and take up a lot of room. It also tends to be high-consuming in terms of battery, which is something we're trying to limit as much as possible. So right now, we've been assessing all the possibilities, but reducing the cost is the most important thing for us. Energy efficiency is essential, but it’s not worth the trade-off of expensive, untested new types of chips. And finally, also, we are still experimenting on the algorithm side, so we need as much flexibility as possible. This isn’t possible with an analog ASIC today.
Yacine: And maybe to add to that, I think running our neural interface controls on your everyday chipset has been something that we've been really focusing on from the get-go to optimize the performance of our algorithms and ensure that we can be embedded into your everyday devices. So that's why today we can run on chipsets that are, for instance, 1MB of memory and 256KB of RAM. This enables us to be integrated into everyday earphones and not have a bigger chipset or a whole computer running our neural interface to provide value for people.
Lawrence: Talk to me then about the specific algorithms you're running. Are they proprietary or open source?
Alain: So it's all proprietary algorithms. It's basically our IP. We have a team of data scientists developing those algorithms from the get-go. There are basically two big stages. The first one is what we call pre-processing, which is all the filtering, slicing of the data, etc. The second stage is the machine learning stage, where we've tried all sorts of algorithms. We currently use deep learning and optimising these to run directly on the chip. I can’t go into the details too much, but there are some neural networks that we've adapted and optimized for the specific purpose of the signal we are collecting.
Lawrence: After reading the Meta paper from last year, I wonder how you see the development of the neural interface field. It seems there is a concerted effort from AI Labs to collect EEG, fMRI data, etc., to train models. Many of the cutting-edge LLMs are pretty general purpose, and is there a world in which we soon take an off-the-shelf LLM, fine-tune it with relatively small datasets and start to make huge leaps forward in neural interfaces?
Alain: From the technical perspective, the big challenge is the amount of data available because to build models that can accurately detect those patterns in your brain signals, you need a vast amount of data, which doesn't exist yet in brain tech, just because it's a new field compared to other fields. So that's number one. Then, in terms of the signal collection. There has been a lot of improvement regarding the sensors themselves. Looking at what's being done in research, they usually use big helmets with 256 electrodes. They put gel on the electrodes to get the best SNR (signal-to-noise ratio) possible. And so much trial-and-error allows us to pick a finer and finer signal. And then, of course, thanks to this finer picking, you can detect patterns that are not just huge amounts of neurons activated altogether but smaller parts of your brain that get activated one by one. So this is a data collection issue, first and foremost and a SNR challenge.
Yacine: To elaborate a little bit more on the data, I think this is something that Alain and the team have had in mind since the beginning of this year, and we've made sure that we are collecting our own proprietary data, which is going to be critical in the future to enable our neural interface to work. But we also think that the first company that will integrate or collect data daily from millions of people will be the company at the forefront of neural interface feature discovery. And that something we really believe in, and that's why we want to be the first one to have an everyday device equipped with sensors that you can wear multiple hours a day, whoever you are, wherever you are, and not just a medical device that would be worn once every month for your regular checkup. I think that's going to be vital for discovering these further patterns that you were mentioning. The data collection comes from having a cheap consumer device. That’s where the race is today.
Lawrence: Good point; a bifurcation between a consumer and a medical device exists. There's the brain computing interface, a medical device that is mainly invasive, and you need surgery to implant it. You get a fantastic signal, obviously, but the costs are enormous. And then we have sort of a consumer vision: a neural interface, a non-invasive headset. You get a worse signal, but it’s cheap and loads of potential users. It’s hard to imagine the same company doing a medical device and a consumer device. What do you think about that challenge?
Yacine: The mental model we have in mind here is the Apple Watch, \or the smartwatch in general. If you look at how it works and how it went, they started with an offer around heart rate monitoring when you're doing sport or when you're sleeping with limited accuracy. And as they improved the sensors and collected more data, they managed to go further and further into the medical space. But, the fact they already had a device worn by millions of people and collecting millions of data points every day led them to move from a pure consumer move to something closer to health care now. And I think that's where we see this going on our end. That's also why we're starting with control and not with health. We know that with the controls, we can provide value to users already with our control who rely on either eye activity or facial muscle activity. And we know we can bring that to a level of performance that's good enough for the user to wear daily. Our goal is to get the devices into the hands of millions of people. We collect a volume of data that no one else can match. And from there, we can improve and bring more and more use cases potentially towards health care as we move on.
Lawrence: Your strategic challenge is finding high-value, achievable use cases. And you’ve landed on control. Talk to me about what control means and what the limitations are.
Yacine: When you talk about controls, there are three distinct steps. First, how you do simple selection; second. how you navigate, and third, how you input complex commands. Which is basically the mouse click, navigation, and the keyboard. If we look at the recent Apple Vision Pro, Apple has replaced the keyboard and mouse with hand gestures for selection, eye movement for navigation, and speech for complex commands. And that's precisely the mental model we have in mind. We're replicating these commands. But we do it with bioelectrical activity captured from your ears. We use facial micro gestures for selection, electrooculography instead of eye movement for navigation, and for complex commands, we use silent speech. That’s the ability to detect what you want to say without vocalising it. And we can still do that with your earphones. So that's our way to represent this selection, navigation, and complex commands, and that's the value we're bringing to the user.
Now the question becomes, now that you've defined this human-machine interface, who is it useful for in the short term and the long term? And so, as I said, I think our target market and where we see we can bring the most value is augmented reality. In augmented reality, we see a clear and growing market around industrial and frontline workers. We know there's already a significant need for hands-free control. That's our beachhead, where we're bringing value in the short term.
Lawrence: Apple has spent all the money and all the years bringing this Vision Pro to market. And they have AirPods. Why didn’t they alight on the same solution as you? Surely someone in the Vision Pro team talks to someone in the AirPods team? Peanut butter meets chocolate; chocolate meets peanut butter style.
Alain: Apple is building the best VR headset. They called it spatial computing, but it’s VR with a little mobility. That was the brief. They have decided to build a device you will use at home or in the office in a static or constrained environment. So it has all the best sensors you can imagine. And so that makes it really expensive, and it uses loads of power. It only has a two-hour lifetime.
What we're offering is kind of the counterpart of that solution. We’ve designed a solution for a highly dynamic and mobile environment. If you have a different starting point, you end up with a totally different solution. And we know that the big challenge regarding mobility is building glasses that are as thin and light as possible. And packing all these electronics into those glasses is a real challenge. So we figured let’s make the glasses do as little processing as possible. Let’s get the glasses to look basically the same as what people wear today and do the processing on the earphones side. And have a Bluetooth connection with those glasses. So basically, by putting this into earbuds, we can free all the space in the glasses to add other sensors that do other stuff and also not harm the glasses' battery life.
Lawrence: So the glasses and the AirPods are combined in a single system, and you are offloading processing to balance power consumption?
Alain: Yeah, the Vision Pro uses like 1W of power for the eye tracking because of the infrared cameras. We are at six milliwatts, so it's just a whole different world regarding computing power.
Yacine: We imagine a distributed computing platform. There is no way we can pack the human-machine interface developed by Apple with all the sensors, just in those small glasses, at least in the short term. So it will need to rely, too, on additional devices that you will be wearing and that will be carrying part of that interaction with that new type of computer.
Lawrence: And what about heat dissipation? Or, more broadly, what is the biggest device-level challenge you’ve still overcome?
Alain: So, fortunately, heat is actually not one from what we've observed from the power consumption we have right now. And the current we need is negligible compared to what's already happening in the earbuds. So we don't really have any impact on the battery. I would say our challenges would be more on the mechanical challenge of having one or two good points of contact within the ear that are stable over time. So, our technology works because those are surface electrodes that touch the skin. And we need this contact between the electrode and the skin to be as consistent as possible. So, we've been working on designs where we ensure that even when you're doing movements or sports, our electrodes stick to your skin with mechanical resistance.
Lawrence: So it’s more of a material science challenge? Are you using new materials as the adhesive?
Alain: We always try every new material that comes to the market. But it’s not so much a new material that would solve the problem, it’s more of a design problem to fit electrodes in the ear canal at the right place with a robust mechanical constraint.
Yacine: And there is a cost issue. Yes, there probably are a bunch of novel materials that we could test, but our primary constraint is price. If you add $400 of material to your earphones, you might be able to pick up some signal, but no one will buy them because the selling price will be too high, whether it is for industrial AR or consumers. So, really, it’s not just the mechanical constraints; we need to make sure that we optimize our algorithm and our hardware to run on components that are not too expensive.
Lawrence: Okay, so more on the algorithm side, they must be very light algorithms you're running. They have to be always collecting bio-signals.
Alain: Yes. And we have been working on different strategies. We have our main algorithm and want to make this as efficient as possible. But we have also been exploring a two-step algorithm model with one lightweight wake-up algorithm that triggers a heavier algorithm. We are working on how to minimize power consumption with that strategy.
Yacine: The first years of the company were spent just trying to reduce as much as we could the actual footprint of the main algorithm. We just got one of our patents approved in the US regarding a novel signal-processing algorithm. It allows us to reduce our footprint by almost 2 to 3x. So that's what we've been spending a lot of time on. We know that the lightweight model is something that will help as well, but the company's core focus had been at first to really reduce the footprint of the primary model.
Lawrence: So you’ve done a lot of work on the software side, and yes, you’ve spoken about the challenges of the hardware, but hardware is complicated. Why not just license these lightweight algorithms to headphone OEMs? Do you need to build the device to demonstrate that it works and then eventually move to a license model?
Yacine: We think that electrodes and all the analog content required for the neural interface will be commoditized in the future. What we also know, though, is that whoever makes the first move in this market will be capturing the market from a technological standpoint. For instance, we've had the example with Ultraleap for hand tracking on VR headsets. We've had the example with Tobii on the camera-based tracking. They've been developing the first hardware, and now they are the one player that everyone uses from a software standpoint in every hand-tracking or camera-based eye-tracking solution for VR headsets. We know that we are making the hardware effort right now because we want to be the pioneer of the neural interface. Tomorrow, we want to be in every single consumer electronics device. To do that, we need to crack the hardware to show that it works. And once we've shown that everyone will pick up on this, we will be licensing the software.
Lawrence: I’m keen to understand what you think about adoption and timeline. I think neural interfaces are a decade away at least; why don’t you apply your technology to a closer market, like healthcare monitoring or something around augmented audio?
Yacine: Well, it depends on what kind of problem you're trying to solve or what problem people are facing now. We’re noticing that the audio field and all the audio capabilities are already a very crowded market with many pioneering companies. We can leave audio AR to Dolby, Sony or Apple. We think our technology is uniquely suited to control. These guys aren’t doing that. We don’t want to be another slightly differentiated provider. We want to be pioneers and solve a real problem.
Alain: Today, there are already limitations to controlling your AirPods. Touch controls are pretty limited, and it’s not exactly ergonomic. Control, play, pause your music and all that. But it’s okay. This problem will be nothing compared to controlling your headset, for which you have absolutely no control right now.
Yacine: Also, to reframe one thing, we did from the beginning to crack how we can go to market the fastest possible way with that neural interface technology. Our technology has three blocks: selection with facial micro gestures, electrooculography and eye movement for navigation, and silent speech for complex commands. The facial micro gestures are ready. We have prototypes that make it work on anyone, anywhere. We can get to market with this piece only and already add value. And then, we’ll add navigation and silent speech over the next 18 months. So, we are talking about neural interfaces in the market by 2025.
Lawrence: Why did you decide to go in-ear to get brain signals? I’ve seen startups trying the lower neck or the top of the head. Tell me where the best place to get is and what the trade-offs you've made to do in-ear.
Alain: The closer you are to the signal's origin, the better the signal will be. So if you want to record things from your occipital cortex, for instance, the part of your brain that deals with your eye signal, which is located at the back of your head, that's the best place to put electrodes. And this being said, the way we operated was to say we wanted to be integrated into a device that people already wear daily and they already charge daily. So that's the constraint we started working with from the beginning. And when you look at those devices, they are not tons of devices that most people already wear daily on their heads. There are earbuds, and that's pretty much it. In most glasses today, there are no electronics, therefore no need to charge. So, it would always need to be an additional device that people need to charge. We think it's a much better idea to start with earbuds. From there, we looked at what we could measure. We know we could capture all the signals coming from your eyes and muscles because they're stronger signals in terms of signal-to-noise ratio. And we have access to the locations and parts of the brain closer to the ears. So the auditory cortex, for instance, and signals coming from other brain parts with a stronger signal.
Lawrence: What are the orders of magnitude here? People have told me you can only get high enough accuracy going invasive. They say the signal is too weak to do anything with meaningful accuracy.
Alain: I would say they should test our tech because it already works great for selection and navigation.
Lawrence: Fair enough. One final question. I imagine a world where we interact with different devices contextually. There is a long arc towards a single interface with the mouse and keyboard towards touch, voice, gesture, etc. So maybe I do speak to my home speaker or car. And I do still use gestures when organising documents like in Minority Report. A neural interface like what you have, how does it fit into this world of heterogenous interfaces? Is a neural interface and silent speech the ultimate UX? Does it replace everything else eventually?
Yacine: There is already a lot of complexity in how we interact with devices. We have diverse inputs, and people seem generally fine learning new inputs and adapting. But there is generally a dominant UX paradigm at any one time. At the moment, it is in touch with smartphones. The next computing platform will be headsets like the Vision Pro. And we expect a new UX to come with that. Apple has some instincts there. The feedback from people using the Apple Vision Pro was that it is very intuitive. They said it became natural after a minute to navigate with your eyes and use micro registers to select. There was almost no learning curve. It’s the same with our technology, it’s much more intuitive than a keyboard or mouse because it’s all spatial and maps to how our brain actually works. We think you’re going to keep using your eyes for navigation. You're going to be using facial micro gestures as a way to control. I also imagine haptic gloves being an essential part of manipulating objects in virtual space. Obviously, we are moving into a new paradigm, so we all have to do the work to figure out the capabilities of this new UX.
Lawrence: Thanks, guys, it’s been a pleasure. Visit Wisear.io to learn more. And connect with Yacine over at Linkedin. And as always State of the Future for assessing the state of play.