Charles Spence - Experiment!

Welcome to "AigoraCast", conversations with industry experts on how new technologies are transforming sensory and consumer science!

AigoraCast is available on Apple Podcasts, Stitcher, Google Podcasts, Spotify, PodCast Republic, Pandora, and Amazon Music. Remember to subscribe, and please leave a positive review if you like what you hear!

Professor Charles Spence is a world-famous experimental psychologist with a specialization in neuroscience-inspired multisensory design. He has worked with many of the world’s largest companies across the globe since establishing the Crossmodal Research Laboratory (CRL) at the Department of Experimental Psychology, at Oxford University in 1997. Prof. Spence has published over 1,000 academic articles and edited or authored 15 books including, in 2014, the Prose prize-winning “The perfect meal”, and the international bestseller “Gastrophysics: The new science of eating” (2017; Penguin Viking) – winner of the 2019 Le Grand Prix de la Culture Gastronomique from Académie Internationale de la Gastronomie. His much-anticipated new book Sensehacking was published in 2021.

Much of Prof. Spence’s work focuses on the design of enhanced multisensory food and drink experiences, through collaborations with chefs, baristas, mixologists, chocolatiers, perfumers, and the food and beverage, and flavour and fragrance industries. Prof. Spence has worked extensively in the world of multisensory experiential wine and coffee and has also worked extensively on the question of how technology will transform our dining/drinking experiences in the future.

LinkedIn

Transcript (Semi-automated, forgive typos!)

John: Charles, welcome to the show, and thank you very much for being here.

Charles: My pleasure.

John: So, Charles, I think you are really a wonderful guest to be on the show, because right now over the next, say, five years, I think there's going to be this tremendous merger that happens between UX and sensory. There's going to be a lot of the integration of the communication technologies that have been focused on sight and sound. They're going to start to bring the chemical senses into the kind of world of augmented reality. And I think that your research is going to play a really key role when that merger happens because the research that you've done, at least for me, on cross-modal interactions has been tremendously inspiring. So on this show, I really want to make sure we get into that. But before we do that, I'd like to take a step back and have us take you through your journey into how you got to be where you are. I think it's a fascinating journey and I think that you know, maybe our listeners who are not familiar with your work as much would really benefit from hearing it.

Charles: So I've been in Oxford for 25 years now and as an experimental psychologist, always interested in the senses and always interested in the application of the latest psychological science, neuroscience to the real world. And one of the first few years that sort of started out looking out at technology in cars, talking and driving multisensory warning signals to help drivers avoid accidents. It's gone on over the years, looking at cars to make people more productive to claim support around fragrances. But for the last 15 or 20 years, it's really got into the world of food and drink. Not necessarily thought it was a good place to be, but only because the lab in Oxford was funded by Unilever for the first 13 years or so. And at some point they had a problem with some of their fruit trees and said, we come and do some experiments for us on smell and on taste. I wasn't really interested, but it turned out after doing the experiments that in fact these were two very interesting senses and they were ones that my colleagues in psychology and in multisensory science weren't really looking at. But once you got into it, they're also really interesting questions, the evolutionary history of the senses and maybe flavor being the most multisensory of all our experiences. One of the most multisensory. So it seems kind of a natural place to adopt. And of course, food and drink is something that we all think about more or less several times a day. And it's great and to try to apply some of the science from the easier to study senses of hearing vision and touch in the belief that some of the same rules of integration and interaction will play out in the chemical senses, smell, taste, trigeminal and so on. But we've got a kind of a head start because it's just much easier to study the spatial senses of hearing and vision with a computer, whereas studying taste and flavors is much harder in terms of creating your materials on adaptation effects and so on and so forth. And in our journey then and I'm sort of squarely in the world of food, drink, not designing flavors as such, but thinking about the packaging and the naming and the branding and the labeling and the context and on the glassware, on the plateware, the cutlery and all the things that have been largely ignored I think. People study to drink very often, but they rarely study the drinking vessel and try and bring it to kind of psychophysics measurement together with kind of nice food experiences, hence the gastrophysics, gastro and psychophysics and work that are funded by food and beverage industry. But having most of the fun with chefs and the chocolatiers and the mixologists and the coffee makers and the sound designers and such like.

John: Okay, and so at this point, would you say I mean, did you start off as a foodie? Do you consider yourself to be a foodie? Like most sensory scientists I would say at this point are foodies. Where are you as far as your actual interest now? And, you know, I mean, you've done I think we just had just Jozef Yossef on the show and he is the chef at Kitchen Theory, you've done a lot of work with. And of course, that would be fine dining. I mean, do you consider yourself to be a foodie at this point?

Charles: No, not really. I think other people do get worried when they either come round to dinner and why don't we bring you, I don't know. They're convinced I have an expert palate, having worked with wine so much. But I think of the other where they come from the other end of things that I imagine I'm the person without any taste buds. Could this actually be true? Is this why I'm so interested in the everything else apart from that affects the taste, the flavor or all these extraneous factors? It's not entirely true, I think, but certainly not a unified palate, nor necessarily believe there's all that much to be gained from working with specially refined palates. But I have seen through my work in wine and food with the chefs and the brands that will get to eat a lot more wonderful food than I certainly did before. I cook a lot. Cooking is separate from that one's research life. And now they certainly have come together a bit. I've been working a lot on herbs and spices, some sort of fascinating and that sort of an intersection. We are growing herbs and spices here in Colombia. The the moment but I'm also researching the history and the pairing and the chemical compatibility and the cross sensory experiential techno multisensory pairings of the future. So I enjoy, but I'm definitely not an expert and might not be able to find food and drink if it weren't for the kind of studying.

John: Right. Yeah, that's interesting, that actually reminds a little bit of my background in that. I mean, I've always been sort of interested in food, but definitely working in the area has made me more aware of some of these. Yes. I mean, it's made me more appreciative of some of these more subtle aspects of food and drink. So maybe we can have you talk a little bit about, what have been some of the big lessons that you've learned over the years? The things that really stand out when you look back at your career as maybe the things that were the most surprising to you or that were, you know, you feel like we're the kind of the most fundamental in terms of their impact, what we're kind of some of the big lessons that you would name as you think back on the research.

Charles: Well, a lot of them have been sort of accidental, I guess, sort of discoveries or insights. And one thing where I am now looking back, you know, 10-15 years working with sort of a technology interface design and as I said, a lot with car companies. And there's a very striking contrast from what happens in the world of food design, from sort of car design. Both are multisensory experiences but always like even if you found the perfect result in the lab regarding how drivers behave and the optimal combination of stuff to get them to do something, then even if you got that dream result, it might take 10, 15, 20 years for that to actually percolate through to a car that somebody might drive on the road. So innovation happens very, very slowly. Whereas, by contrast, I've been very pleasantly surprised again and again by just how much faster and more innovative the food design space feels to be. Both because I think the you know, sometimes the chefs are more creative and take the scientific ideas that they find interesting or intriguing and turn them into delicious dishes that can resonate with the general public and get talked about and emotions and memories and feelings and tears and all that. And at the same time, being their own boss if they believe in something, they can put it on the menu next week. They don't have to clear several layers of management only for there to be a restructuring and, you know. So in that sense, I think one of the easiest, refreshing and exciting, it is going to be a more rapid innovation space that I think working with the food creatives allows. And then, a part would be, I guess lesson learned is I used to think that if I got that dream result with the, you know, the graph with the significance stars highly significant effect of my factor, my lighting, the background music, the scent of people's behavior, then that was it. Problem solved. Moving to the next thing. And then I soon come to realize that too many in industry, this isn't just in the food industry, but also in retail and elsewhere. They also invite you to come along to find out the latest about the multisensory effects and influences. So not that politely, we tell them all these things that they could be doing to enhance the experience, to boost sales, of course, and then they'll go and do absolutely nothing. And I think that sort of hints at will probably be obvious because most people don't read scientific articles anyway. So once they go there, they've lost but more on that, if a finding does not fit with people's intuitions about how the world works, about what affects them, then the more mind is just to go with what you have been tried and tested before. And hence that's what led me into wanting to do more kind of experiential type events, experiences, demonstrations, because we know by having people experience some of these things, by experiencing, say, for example, the Singleton Sensory wedid in 2013 where we changed the lighting and the music and the sense and changed people's perception of the taste of whiskey, those who were there, you know, went away. Some of them changed the way they served in their restaurants, in their bars. So how do you deliver the kind of the convincing demonstration of what you found that will drive change and not just had not that exciting, I think was one of the best ways of doing that and that maybe I mean, I think to me that back in 2003 I was introduced to the chef Heston Blumenthal and he was reasonably well known, but not especially famous. And he can talk to lots of other science labs, see what was going on, and then sort of use that as inspiration for some of the dishes that appeared on his menu and through our work on sonic chips that slid into a dish, the side of the sea that got onto the menu and that was really surprising at the time. You got the experiment, you'd never have done, but we had 150 people coming for a food festival and they'd be paid some money to eat some food and be experimented on. And then two days before everything last minute, that was kind of the reason why we said, okay, let's get some oysters, we'll play the sound of the sea, see what happens. We also did bacon egg ice cream with sizzling bacon and chicken sounds and the crazy thing that you'd never do if you had to write the ethics and the grant proposal and the justification to just never work. But it did. And that led into the sound of the sea dish that became a tradition in a restaurant. And I was very surprised at the time but in hindsight, people say, well, it's kind of obvious that the seaside sounds make seafood tastes better. Well, we didn't need a psychologist or a gastrophysicists to tell us that. I mean it was obvious before, but people becomes obvious afterwards. That's why the work we did the last decade sort of came out of that in sonic seasoning and in the matching of a pairing of music and sounds and soundscapes to tastes and flavors and textures in food and drink has been so much more exciting and surprising that background music should change the taste of the food. Why should that happen when these things are unrelated? And then that's sort of surprising to people. And also it continues to surprise them. They are sort of skeptical and obviously work on everybody. But many people it does. And then to see people we have to be sort of converted, as it were when they find that their coffee or their chocolate does taste different with different music is great fun to both the challenge of trying to convert people, that's the right word and the fact that it does retain its surprise element, even 10 years later after we first started thinking about it.

John: That's fascinating. Well, you know, throughout this, it sounds like, okay, there's a little bit of the distinction between invention and innovation in that you, through scientific discoveries, have had the equivalent of inventions, but then to actually bring them to market is a different story. And that along the way, you've discovered how to market your ideas and how to get them adopted, which is really what innovation is. So I think that's fascinating. You've been so innovative and you've definitely had, in my opinion, a very large impact on our field of sensory consumer science in raising awareness. I mean, for example, some of the work that I believe you did with Betina on looking at the vessel that the color of coffee, the kind of stuff. I mean, you're right. After the fact, it sounds it seems like well, of course, it makes sense. But to have the idea to investigate it is I think is a different story. You know, I mean it's yeah. I mean, reminds me of you think back of Archimedes to have the idea that there's a relationship between an area of a circle and its radius. I mean, that's different from proving, you know, like to have idea and prove it. So maybe that would lead to a good next question, which is, you know, where do you get your ideas for your research? I mean, like what inspires you? What do you pay attention to? You're very creative. So what stimulates you for your new research?

Charles: On the one hand, I don't write grants and I have a research program, which I think obviously is not popular in my department. But at the same time, I think is a key to creative endeavor that I'm not having to write down what I'll be doing six years from today and three-month Gantt charts. So instead I'm much more open just to whatever whoever walk the door and the part of the inspiration just comes from, you know, we've had I don't think it's possible necessarily to go out there and find the people you're looking for, industry or expertize. So, we've had the Franco-Columbian chef, Charles Michel walks through the door. We get introduced. He loves cooking, Michelin-trained, sort of Kandinsky Painting playground Kandinsky salad although no one has ever sort of assessed the actual impact of food, art or willingness to pay or taste in us. I put him together with, I think it that was Betina. But as a statistician, maybe the statistician can't cook, the cook certainly can't do statistics. Together they create magic. So I'm an absolute facilitator of that link. And of course my lab would be the space in which different people come together. Than you know, typeface designer walk the door and say, wow. Everything we read is in a typeface with a font. But when I was the man was eighty four, came out and so allowed the people to actually vary the font and typeface. The first time that sort of passed by the research scientists are now in twenty, twenty one. There are maybe only twenty published studies of typeface and what it connotes, what it means and the connections between typeface, shape, weight, so on and other properties. There's a whole area of research where the typeface designer then can create some typeface or select them and we can put them together with a foodie or a statistician or all three to look at the, you know, the taste of typeface, for example. And can got with somebody who's much more tech-savvy and all of this of the online Internet based testing and then run across the globe well, across multiple countries, for example. So that's a big source of inspiration. And it also has been a lot just from being talking to being in the kitchens with the chefs like Dastan and Jozef and others and some of the things that they just sort of observe in passing what we serve food this way and people say that about it and they think, wow. So, for example, when Chef Jesse Dunford, what he serves is like a chef's table and for one of the courses on the chef's table, three mains are brought out between six people. So somebody is going to stop the main and somebody else have to wait and then you switch plates partway through. It is strange, but I'm convinced to whoever I give the cowpie to is one of the main courses. There was really love it and I think they get second. They don't seem to like so much. I'm sure it makes a difference if you get to choose first or taste first, but had not been studied, as far as I can tell over the Fat duck when we started working food to help Unilever with their fruit trees. That looked beautiful, that smelt great, but then tasted of nothing because they don't have any sugar in them. We would experiment with powdered color, with powdered taste to powdered flavor. Got some results. That's great. But then went down to the Fat Duck restaurant in the kitchens there and they were serving a beetroot and orange jelly with the colors were mixed up with the golden beetroot and a blood red orange and later in the meal are also serving a black currant pastel, the same purple color as the blood red orange. Earlier, the same color was associated with very different tastes at different parts of the meal. And they figured out that some of the fruit acids that you find in real fruits, citric acid and malic acids, if they change the balance of those, they could flip what their diners were willing to interpret that color as meaning. More fruit assets, of course. The purple looks more like a black carrots. Last night, it's more like beetroot and I thought well, for working with real stuff. They've discovered this and went way to all the food science on shelves. Has anyone looked at the impact of fruit acids on colors influence on taste? No, they haven't and that led into a couple of papers. And it's going that way from the kitchen into the lab saying things that are total findings. Let's say that through to the psychology journals are full of interesting illusions and effects and phenomena and I just think back to all those visual perception textbooks, all the amazing visual illusions, things that seem to move when they're not or by stable or alternating or gestalt, all sorts of incredible illusions. So I think just well, could we make an edible illusion so I can see lots of visual illusions or edible magic or magic visual and then together with the chef then Jozef Youssef, who you've had on the show and a psychologist come professional magician. We get to work to thinking both theoretically about is magic a purely visual art? Can you have edible magic than what people would like edible magic if it was created? And then once you figure out the parameters, then it's over to the chef Jozef to say, okay, can you make something that takes the right boxes that will then be tested in the restaurant? He talked about it a universal kind of edible illuminated light bulb. It's pretty cool. That comes out of that. And against a corporation with magician, chef, sensory scientist, and gastrophysicist and magic happens. Yeah, that's just how it goes.

John: Yeah, well, a lot of innovation is kind of recombinations or taking something from one area and putting in a new context. I mean, I think that is incredibly inspiring, though. So I would like to know I mean, it's amazing, but we are almost out of time. So I do want to make sure I get to your, I mean, yeah, there's a lot to talk about. I would be curious when you kind of think about the next two to three years. I mean, there's a lot of advances. Obviously, a lot of the advances in communications technology have been more sight and sound oriented, a little bit of touch, but mainly sight and sound. So personally, I am very interested in the kind of interface or the possibility that the chemical senses may finally get integrated in some of the communication technology and we've had just actually our last episode, we had a student in Wageningen and working on 3D printed food and I think that the idea of 3D printed food, 3D printed smell, just this kind of thing, it's all quite interesting to me. So is that something you're interested in or is there another research direction that you think is going to be more productive over the next few years?

Charles: I absolutely hate 3D printed foods. I have to say. Our technology looking for a purpose. I'm sure there is one in this case. So I'm not a fan of that and haven't seen any clear example of any sort of technology, new technologies being brought to food. And so that general space is interesting. I'm actually not really a fan of, you know, sort of robot chefs and robot cooking assistants. I am certainly curious about Sonic AI and some of the other big tech companies trying to get into recipe creation, digital recipe creation. I think that's really a large part misguided the way they're doing it currently. But still interesting to see and to follow. For me, I think, I mean, I'm also pretty negative about all the sort of the food, most of the food stuff that comes out of the HCI conferences. I just can't see any of it really delivering a great tasting experience nor being incorporated into a restaurant setting. And yet, yeah, we're looking at people like Jozef than we do. He does bring the projection mapping into the dining room, wireless headphones and so on and so forth. So clearly technology is coming to the dining table, but I think it will be, sorry to say restricted to the audio visual and a little bit of vibration. I do not see any scope for digital tasting or digital smelling and hence, it's going to be how do we take the amazing, to me how do we capitalize on the amazing technology in so many people's pockets of smartphones and suchlike and put them to good food use to enhance experience. And that might be through sonic seasoning that helps you reduce the sugar content of the other foods that you choose to eat. Maybe get Spotify to figure out your personal preferences, and it may be dining off tablet computers might have some advantages again putting, but it's the ubiquitous technology. I don't think people are ready yet to buy a new piece of technology for the dining tables, especially for eating. But there is always amazing technology just sitting right there and the question is how to reposition it. So a lot of our work is in that sort of space, sonic seasoning, bit of tablets, the sorts of ways of storytelling or we have been working on with artists on autonomous sensory meridian response. Idea that there's been something to deliver the extraordinary in food and that happens through, I think by combining with technology, sounds of the sea with the replay that brings so many people to tears when seafood never brings you to tears. Otherwise, the sound of ASMR, this whole idea of the magical and extraordinary, because I think the last 10-15 years I've been using or trying to use the science that I understand at the senses, taking it from the spatial senses, applying to the chemical senses, but all with the goal of modulating what we already experience. I can make things 10 percent sweeter, 15 percent more sour. But it's all within the bandwidth of your everyday experience. What I like to do is sort of push the boundaries and say, you know, where can we get to buy illusion, magic, other strange phenomena? Can I take taste out of the mouth? And probably in any of those ideas or hopes and aspirations, they'll probably all involve a fusion of real mixed reality, of a real chemical sensory stuff, real foods and drinks or vapors, combined with a digital audio visual element that that may seem to some a bit synesthetic as we try to match sights and sounds of tastes, of flavors. Hopefully together will create something that takes a taste out of the marathon, off the plate.

John: Right. So basically, your inclination is to lean into the technology that already works rather than waiting for some new silver bullet to come when it comes to....

Charles: I'm running a lot of this last year on smell scent in cinema, a virtual reality in a theater and shopping and everywhere else. But if you look back, we've been imagining scented cinema since the very first virtual reality simulators and television and cinema and big screen movies. And it's really come to pass. And I think it's a real fundamental problem we could develop a technological solution to delivering a range of fragrances or flavors, which is a big problem in its own right. Maybe if we could solve that problem, I think we would still be left with a problem. What I would call a fundamental misattribution error, which is that we are all visual creatures. And what I could demonstrate in the science lab or in the home, that your experience of that football game was better because I pumped out the smell of fresh-mowed grass, a player's sweat, whatever it might be, I could prove that. I've no doubt it's true. Nevertheless, you as the consumer will attribute your pleasure, your enjoyment to what you see, not to the smell. You say it's a very good game. That's why I like it so much. And so we all, I think, fundamentally misattributed our experience, our pleasure, our emotions to what we see. And for that reason, even if the technology did exist, which it doesn't, you still not be able to convince. I think, you know, the consumer to buy the refill. Why bother? I don't see the benefit and I don't know yet how we get over that. I wish we could.

John: Yeah. Now, that is definitely fascinating and, you know, we're going to have increasing sensors that are around people with wearable devices. It is true that with Internet of Things, you're going to have an idea what people are up and then you have the chance to augment their experience with sight and sound and maybe some kind feedback.

Charles: And that's they have been working with a wonderful team of scientists, computer scientists in Japan, Katsuo Okajima, over the last few years. I'm looking at augmented reality food experiences because using the technology to change, his team can make sushi seen through your mobile device, change the fish in real time without better cookies and QR codes with all the vending machine. So some amazing visual science that I got sort of interesting. I got to play more for product innovation and packaging innovation. I struggle to see the top experiential chefs out there getting their diners to put on a headset. It's going to probably disrupt the interaction between guests and so on.

John: Okay, well, that is actually very, very helpful for me so thank you for the clarification. Charles, before we wrap up here, a question we always like to ask is your advice for first students or young sensory scientists beginning their career? What would you recommend that somebody be looking at or studying over the next few years in order to really get their career off to the right start?

Charles: I mean, I do get the question quite a lot of people who say, we read Christopher love the stuff, where do we go to study this? And unfortunately, it feels like currently at that nexus of behavioral economics and design and food science and psychology that is gastrophysics, there's no other does it all catches all the bases yet. And I would say just my own research how liberating it has been to be able to work with the baristas, the mixologist, the chefs. And while that might feel like impossible, I would encourage the young scientist, you know what just go down to your local coffee shop or cafe or restaurant and you might be surprised that there are I think, a whole generation of young food chefs and cooks who are interested in wanting to learn a bit more. Learning about the sensory science behind flavor, shifting the focus just from, you know, the preparation, the presentation to the diner, the mind of the diner. And that's going to be, I think, a big area going forward and as well, for me, it's trying to get as much ecological validity as possible. So trying to take the research where one can out from the lab, from the highly trained sensory panelists into the wild, into the real world. So, I'd encourage people to try and think about testing at food festivals, book festivals, music festivals, literary festivals, you name it, where one can get access to very large numbers of people and I think that a view that was when we first did the Singleton Sensorium in 2013, the sensory scientists by the Brandos were horrified. You you're going to ask the consumer what I think about our drink. You can't do that, they don't know what they're talking about. After the fact with so many consumers responding to that shift of focus, they know maybe there is a useful sort of data there. I think it's important to try to put things in the world. Just also highlights the possibilities in terms of new research ideas and also constraints or limitations about the things that we study only in the lab. Maybe they just don't extend beyond their maybe there are so many contextual factors that are important that we try to isolate and ignore and eliminate in our lab research, but which are always there. I think, when we eat and drink in the real world, and hence maybe we always need to think about them too.

John: Okay, great. Well, I think that's excellent advice, actually, just the idea of food scientist to go down to the coffee shop and try to do something informally with your local barista, I mean, to get out into the world and get ideas, I think that's really exciting.

Charles: I mean, just a simple example, I went to Bristol to where the coffee shops stand, not so far from Oxford and with the Maxwell Davis was a proprietor there. He was sort of interested and did a study one evening, invited 20 of his regulars into the coffee shop and reserves half of them in a cafe with some latte and half was just a regular cafe latte. And that becomes an experiment comes a study probably slightly underpowered these days. But nevertheless, I think everyone wins. The customers kind to get to learn about the philosophy of the coffee shop, but also a bit about the sensory science becoming guinea pigs themselves, kind of in a way that sort of fun, and they all seem to like this with general hunger, I think, the contributing to or learning about taking part in experiments and finding out how your taste works and then have a sense of the connectedness or surprising and then it can be just as simple as that. I have colleagues in Australia doing the same thing with beers. If I go to a local pub and it does beer from a straight sighted door or a sighted glass taste the same way you go.

John: Yeah, that's excellent advice. Well, Charles, this has been really a pleasure. If someone wanted to get in touch with you, what would be some of the best ways for them to reach out? Maybe they want to apply to study with you in Oxford?

Charles: The only way is to email me because I'm not on any social media.

John: Okay, so if they were to go to your website or is there? We don't try not to put the emails on the shownotes because we don't want you to get spammed by bots. I suppose if somebody wants to they could get...

Charles: To find it, though it hasn't changed for 25 years so it should be one click and it'll be there to get in touch.

John: Okay, that sounds great. We'll, Charles, this has been a pleasure. Thank you very much.

Charles: Thank you.