Updated: Aug 22, 2023
Welcome to "AigoraCast", conversations with industry experts on how new technologies are transforming sensory and consumer science!
AigoraCast is available on Apple Podcasts, Stitcher, Google Podcasts, Spotify, PodCast Republic, Pandora, and Amazon Music. Remember to subscribe, and please leave a positive review if you like what you hear!
Professor Hort is an internationally renowned sensory scientist and is Fonterra-Riddet Chair in Consumer and Sensory Science at Massey University (NZ). She is also a Principal Investigator in the Riddet Centre of Research Excellence. Her research focuses on a multidisciplinary approach to understanding the factors affecting consumer perception of food and consequent choice behaviours, and developing effective methods to measure them. She is director of FEAST, the Food Experience and Sensory Testing Lab, and has recently secured NZ$2.25M to develop a future facing Consumer and Sensory Science Centre at Massey’s Manawatu Campus.
Transcript (Semi-automated, forgive typos!)
John: So Joanne, thank you for being here.
Joanne: It's my pleasure. And thanks to Lauren Rogers for recommending me.
John: Oh yeah. Yeah, definitely. Lauren is great. She'll be on the show actually before too much longer. So, alright, so Joanne, you and I have been talking about this new lab that you're setting up. And I think that for really anyone who is passionate about their field, which you certainly are. The idea that you have a blank slate or close to one that kind of paint your masterpiece on is something that I think I guess is blank canvas, not mixed metaphors, is something that appeals to anyone who's really expert. So I think a good way for us to start this conversation. Would it be for you to tell our listeners what are the things that you see as you're setting up your lab, the kind of new technologies or the new approaches that you're wanting to make sure or kind of set up in the DNA of this laboratory so that it set up the way you'd like it to be set up from the very beginning.
Joanne: Yeah. Thanks, John. It's been a really exciting time and I do feel very fortunate that Massey have giving me this bouquets of money to put this center together. I think it's really timely because when most people think about sensory science, they might think of people sitting in sensory booth tasting foods. But really, the discipline has evolved so much over the last few decades that the type of facilities and technologies that we need now to understand the relationship between the consumer and the products have really moved on. So when I was given the opportunity to design this center, of course, we will still have the sensory booths that we need in there for running with expert panels to profile products. But really, what I've tried to do is really investigate the technologies that will help us understand that relationship between the product and the consumer. We used to just assess products and maybe the consumer, but now it's the link between the two. So really, in addition to those sensory booths, internationals standard kitchen, etcetera. We've really tried to look at what technology we can use to focus on consumer behaviours. And so one of the things that we've done is we invest in mixed reality technology. So we've been using the Microsoft holoLens as suppose to virtual reality. Mainly because it helps with the problem of the participant actually being able to see the samples so they can be immersed in this wonderful environment, but then still not have to rely on somebody to spoon something into their mouth or find a glass, which they can't see on the table in front of them. And so we've been using the holoLens to record videos of real contacts, of cafes, of people's kitchen setups so we can test foods in those particular arenas and get the consumer response. In the new facilities what I'll also have is a room, which is another blank canvas, which we can project videos onto all four walls to immerse people in an environment that's not mixed reality because some of the issues with mixed reality is that they know they're in a mixed reality environment. Some people get motion sickness. And so what we can do in that environment is bring in real props. It's a cafe, we can set it up as a cafe. If it's a hospital ward, we could set it up as a hospital ward to look at patient engagement with medical foods, for example. And we can even bring in the smells and the sounds that are important in those environments to really understand how the consumer interacts with those products. So, yes, that was key on my list was for this new facility that I was going to be able to set up more ecologically valid contexts. Carry out our consumer research, as well as links with cafes in town and areas that the actual real contexts.
John: Right. And what are some of the measurements that you're making in these environments? I mean, like to me, that's really fascinating. Also, they have this controlled environment. So now you're in a much better position to measure all sorts of things. You know, a timing data. You know, you've got reaction time. You, of course, eye tracking, but just a little amount of time, it takes people to do things. You know, you have a level of control that would be very hard to get in someone's house or in a cafe.
Joanne: Indeed. So what we've done is we also have a couple of what I would call the more like psychology cubicles, which are protected in terms of the when we plug things into the electricity, we have to have a room that is protected, body protected, so we can then hook up people to, I've got a bio pack system which we can attach to facial muscles so that we can record how that facial muscles are responding to different stimuli that we present to them. So we're not just going to be asking questions that require cognitive processing about the participants response to a photo. How much do you like this or would you buy this? Because we know those measures don't really protect food choice behaviors or purchase behaviors. So what we're trying to do is invest in technology that will allow us to measure more implicit responses. So we've been looking at muscle movement to say frowning and smiling and even the wrinkling of the nose. To a whole host of food products. One of my research has has been feeding people Dutch licorice, which is hot. Some of them have not been able to actually eat it. So the actual facial responses have been quite extreme. It's hot, but we starting to see how we can predict people's actual response, emotional responses to to these foods by measuring the muscles in the face. So we have invested in that capability. What I've purchased is a a wireless system so it's much more comfortable for the participant because often when we do this research. You know, being wired up to technology changes the consumer's behavior beyond what's ecologically valid. So what I'm trying to do is set up environments that are more comfortable and where the technology isn't as invasive as it might have been in the past.
John: And for our listeners, who maybe not as familiar with your research, could you talk about one particular study that you feel like is really representative of how you've been combining these new technologies? A study that comes to mind as sort of, you know, typifying a lot of the new ideas, new technologies that you've been talking about.
Joanne: The work that I do, I've only been in New Zealand for 2 years. So a lot of this work started when I was in Nottingham, where we did a lot of work around people's emotional response to beer, which is always a great food or beverage to investigate.
John: If I can just add a funny story very quickly. I was involved in a lot of beer testing when I was the Institute for Perception. And at one point there was this idea for ecological validity where you go into bars and, you know do all the tests in bars. And then there was this market research agency, I don't believe we were involved in this testing. We just got the data from the test. And it turned out that there were like basically someone decided having attractive women go and serve the beers in the bar would be a good idea. And it turned out that pretty much like all the variance in the data was explained by how attractive the woman was who is serving the beer. So I think you can go too far the ecological validity.
Joanne: Yes, you need to be careful. And I think that's why our immersive environment will come in. So research that we've recently published my PhD student Marit Natuman, she published that a study why we looked at contrasting perception, emotional response to be in a standard sensory booth, but also in a bar and also in any booth context. So we evoked the bar through photographs and songs on an iPod. And what we found in those context was that the choice behavior was very different across different segments of the consumers. So much so that a large group of the consumers were only really able to see significant differences in the beer choice when they were immersed in the real context. So what we've done at Massey is we've progressed that in today, the mixed reality environment. And we actually worked. We didn't work on beer, we worked on tea break snacks. So a slice of cake. We we love our coffee and cake in New Zealand. So it's a good idea look at. And what we found is that when we compared using the holoLens mixed reality in a sensory booth with the real cafe itself, that we saw very little difference between the data set. So the mixed reality environment was able to help us predict what would happen in the true environment, which helps us to have a little bit of control over the environment where we put the paper in which would solve your problem.
John: Right. Yeah. That was exactly my problem. I don't think I would design an experiment like that.
Joanne: No. So we we're looking to write that research up, we all writing it up at the moment to show that the mixed reality is a good proxy for the real environment, but it allows you to have a little bit of control over the immersive environment itself that you putting the people in. The next study, once my facilities are finished, will be to compare that with instead of using the mixed reality will be to use it in the room itself with the videos playing the smells, the sounds. And what we can then do in that environment is bring in the EMG recording of the facial muscles to look or implicit measures to the participants responses to the products that we're testing.
John: Outcome measures then or are you, I think it's interesting to me about some of these implicit measures. They can be used either as inputs or outputs into a model depending on what you're trying to understand.
Joanne: These will be output so we can look at the movement of the muscles that help you frown. And we can look at the movement of the nose wrinkle and the muscles that help you smile. And we can look at how much those muscles are moving as a measure of the emotional response to the product. And the study that I've just described, we're already seeing relationships between reported emotional response using the 25 self report model.
John: Just to make sure for the listeners. So that's kind of a standard battery of emotional. I mean, you could think of a sort of his lexicon. It's a list of emotions that are meant to span the space. Is that correct?
Joanne: Yeah. Indeed. So, you know, somebody may express disgust to licorice and somebody may express increased liking to licorice. We have a lot of Dutch students in New Zealand, so they tend to like it. And you can see that the movement of these muscles is responding according to what they think their emotional response says. So we're starting to see some useful measures so outcome measures from just measuring people's faces in terms of their emotional response, which we think is a much better predictor of food choice behaviour than the standard measure of liking.
John: Right. There was a bunch of things like ask you about that, actually. One is I've done a lot of work on emotions research, but it's self reported emotions. I haven't actually been involved really. I'd like to be involved in more of those implosive measures. But when I was doing a lot of emotions research, what I found typically was that you would have a fair amount. You'd have several dimensions in the positive space, but you typically would have no more than two dimensions in the negative space. Where you'd have is usually something like disgust and then some sort of guilt that might be, you know, may have been a function of the product categories I was working on. But do you see that there's more texture when you've got the ability to really measure what you're like, I guess with a few benefits, if you've got the high technology, you might have more reliable results where they're more consistent over time. Right? Or you could have just greater, maybe less variability. So you can see differences that you wouldn't be able to see otherwise. But then there's also the idea that maybe you're, in fact, getting even just another more dimensions of information. So, like, how can you describe what are the ways that these implicit measures are really better than self report measures in your experience?
Joanne: I think there's still quite a bit of work to do on the implicit measures to understand how discriminating they can be between products. We're still not there yet because there are a couple of studies that I've done and others have done that have shown that what the implicit measures are good at doing is separating pleasant experiences from unpleasant experiences emotionally. We're still seeing better discrimination from over products from self report. So I did a study looking at beer aroma and we looked at bear aroma and we we changed the aromas in a based beer. And we looked at the facial response when people sniffed the product. So we were just looking at aroma. And we also measured their emotional response using a consumer defined emotion lexicon for beer that we developed and we saw better discrimination still with the self report. I think where the implicit measures are going to be valuable once they're better validated and we've investigated them more is in measuring subconscious response because obviously with self reports, we're gaining data from what the participant has processed about the experience that cognitive processing of the experience. And that doesn't always measure up to what they actually felt, because we know participants will tell us often what they'd like us to hear or what they think we should hear. Whereas the implicit measures that we're measuring so facial muscles or changes in heart rate or skin conductance, all those implicit measures, they're measuring the immediate response to the stimuli which then produces the self reported emotional response. And so I think the value of the technologies that we're using to measure what we call implicit response is getting at the subconscious response, which some sort of a true response.
John: Yeah. It's interesting. It is not discuss by system one system to right now, The idea that you've got one system, you know, which is you're kind of maybe evolutionarily, older based reaction to something essentially. And then you've got the kind of the response which is mediated by the thought process. You know, I think about what you're going to do, kind of more careful answer, right? And, you know, in fact, during my postdoc back to our research, we looked at that, we looked at how one and other systems interact with each other. And it is quite interesting. But it is definitely true that sometimes people just make up stories to explain their quick reaction to something, right?
Joanne: Oh, yes, definitely. I think it set aside, but I think it relates to this conversation. So there's a lot of hype at the moment about we saw a couple of things come through at the Pangborn symposium about using social media or data out there on the cloud to understand consumers. But of course, the image that consumers portray on social media is the life perhaps that they would like to live. Not necessarily the one that we're trying to produce products for that they are really living and I think this is true of emotional research, if we're asking them questions about their emotional response, you know, what are they telling us when they answer that question? Is it what they read out? Is it what they think they should have felt? Is it the image they want to portray? Whereas these new data technologies that are coming into consumer and sensory science to measure the physiological responses or the facial coding perhaps can help us to understand the true response. But at the moment, we don't know enough about them. I think I remember guest on RS was was talking at Pangborn about, you know, there's been this plethora of new sensory methods and new technologies that are available to us and people are using them without perhaps truly understanding their worth. More data are actually getting out of them. And a good example is something we recognized in the holoLens data because we were measuring emotional response. One of the things we noticed about the holoLens was that the response to the products that they were testing in terms of feeling excited and adventurous with much higher, with significantly higher than those responses that we've collected in the cafe. So technology itself was affecting the way in which people were reporting their emotional responses. So if we go out there at the moment and use all these gamification or virtual reality and mixed reality technology, all we how much of that is just measuring the response of the participant to that game or that what they're actually doing, as opposed to the products which we think we're interested in a way which we think is about. So I think we need to understand a lot more about these technologies and what the data is telling us before we start claiming that they can do X, Y and Z.
John: Right. Yeah. I mean, this is so interesting because, okay, one aspect of this is validity. Right? The data are just, you know, they are representations of the world, but they aren't the world. They're representations. And how you collect the data has, of course, a huge impact on the interpretation. So there's that. And then I think there's also this fascination with statistical significance that I think we as a field have to kind of move past that just because you get more statistically significant results doesn't actually mean that what you've done is better. And any kind of global sense that, you know, I mean, it's interesting, the idea that you gave maybe manufacture differences that don't matter to the consumers through very sensitive testing methods. It's much better to have, I mean, I could talk about tetra versus and other methods. I love to think tetra testing is better? Because it's more it gives you more significant results. That is not true. What's better, in my opinion, about the tetra test is you have actually a better measurement of the signal to noise ratio from this tetra test. So that's I think, more interesting that if you've got noise in products that are being sampled and you're more of the signal is making it through relative to the noise or rather the noise is smaller and the measurement. So you have better use going up. All this happened now as you have work from a precise measurement. Now, what you do with that is your business. Maybe it still doesn't matter to the consumers. You know, maybe I have a highly skeptical result that doesn't matter at all to consumers. You know, I mean, we've got to think about more than just, you know, statistical outcome.
Joanne: We do, indeed. And what I try and teach all my researches is to you know, they go straight to the p value, which is partly our fault from the way we teach sensory science as well to check the size of the effect.
John: Yes. And the confidence if you can get it.
Joanne: Exactly. Yeah. And of course, I'm a lot of the work that I do is applied. I've been very fortunate through my career to work with major food industries, FMCG companies both within the UK, out of the UK and still here in New Zealand. And I think one of the things that we need to start looking at is whether these technologies are allowing industry to make better predictions about the success of products. The choices that the consumer is actually making. You know, in the data that on collecting, we can already see that's measuring emotional response, for example, can increase the product development process compared to just taking the liking data.
John: Can you give an example to help, to flesh out that point or just talk in general?
Joanne: So, for example, if you will measure if you're making these emotional measures, you can see more discrimination between products compared to liking. So the one that I've published was on blackcurrant cordial that, you know, you can have two blackcurrant cordials that have been developed to taste more or less the same. But vary in their composition in terms of the sugar content. And what you see is that they're both equally liked, that both very well optimized products. But when you look at the emotional response, you can see that for one of those products, the consumer is less trusting, is a little bit more disgusted than not disgusted. But the scores is significantly lower and it's a sizeable effect. And so what that tells you, the liking data doesn't tell you that. It just says the consumer likes this but if you look at the emotional response, you can see that product relationship is some kind of tainted in some way because of the press. In this case and the consumers didn't know in this case, it was an artificial sweetener. But that sends a signal from the artificial sweetener. They still liked it, but saying to them, oh, I'm not so trusting of this product. So it helps industry to see how that's going to pan ou when you've got all of the information about the product, is the reduced sugar aspect enough for the consumer to still repeatedly consume that product over time. And for some customer it is and for some consumers it isn't, because we know consumers are all different.
John: Right. Yeah. I mean, it doesn't make sense that especially you're talking about to make an artificial sweetener or we might evolutionarily be very well trained over, you know, a very long period of evolution and time scale to be able to pick up natural sweeteners. Right. That's something that we did expect that the older parts of the brain would be good at. And so then, you know, the kind of instant, fast response like which we called subconscious response would probably be very sensitive to deviations from natural sweeteners, whereas you might talk yourself into being interested, like especially if you know something has fewer calories. You can talk yourself in eating it, but it's not the same thing as actually liking it or not not liking. But you understand. I mean, Yeah. Yeah, exactly.
Joanne: Our emotions drive that one so unliking is a separate thing. So I think it is very important that at the moment we've got these batteries of tests for self report on emotion. But if we can optimize the implicit methods to become more discriminatory so that they can pick up small differences. If we can find which of those physiological tests can move us to that space, then I think we've got a really good tool to help researchers, whether that's for health research or for the food industry to really understand the consumers buying if you like to those products.
John: And let me ask you, like you were saying before we started, half hour's gonna fly by whenever we talked. It was always easy to figure out through half an hour. But what do you see, Joanne, as the like the next five years? I mean, obviously, there is the need to connect the work we do to actual purchase behavior. I think it's a big problem because somebody exaggerates variables when it comes to purchase behavior. But one, what do you see as the work that needs to be done in order to, like, continue on this research path that you've started to lay out?
Joanne: Yeah. So I think it's really about that focused on the relationship between the product and the consumer. Not the product, not the consumer, the bit in between. And so, in order to do that, I think we need to focus on validating these methods that do seem appropriate for measuring that. I think we need to understand a lot more about the individual differences that are driving the consumer response across food and beverage. But I think one of the key things that is going to help us understand this more is, is the area that is close to your heart. And that's how do we manage all of this data. How do we develop the machine learning? The artificial intelligence which has such an opportunity to help us trying to understand all these different types of data. The consumer data, which is notoriously has large era or standard deviations because of the individual differences. It's not because it's full of errors, because of individual differences in people. How do we match that to the more the data that were technical data that we're getting from implicit methods, that we're missing from analytical and physical tests. So how do we bring all that together and how do we do that in a more ecologically valid context? So we've talked about mixed reality and immersive contexts and collecting data in real life situations. But the other thing is collecting that data over whole product consumption. So multiple sips not just single sips. I think we need to move away from single sips of products and understand the whole consumer or, you know, a couple of bytes. We need to move on from that and understand the relationship through multiple consumption of products.
John: Yeah, it's interesting. See, like, validity really runs through all of your research trying to get actual valid measurements that will let us make more informed decisions. I mean, would you agree with that the validity is kind of a driving thing for you?
Joanne: It is for me, because whenever I stall out any research project, whether that's with a student or with an industry, I ask them I'm probe so many questions to understand about what their idea for this product is. Where is this product going to be used? Do you even know the answer to that question? Because there's no point as doing any of this research. If we don't understand the context in which the consumers are going to be using these products or at least tell them the context if they don't know if this we can't get that data initially. Then we need to, we always look at our data to try and understand if we've got segmentation. And I don't mean age groups or genders, really. We do look at that. But I mean understanding perceptual segments across the population because we know that ethnicity and not genetics, even within different ethnicities is driving different perceptions.
John: Yeah, I agree with that. It's sensory segments, I think is not another way that I've heard people describe this. Is that what you mean by perceptual? There are people segmented by their sensory experiences maybe by their preferences. And actually, machine learning is very helpful for that. My kind of protegé, Will Ross gave a talk on that at Pangborn on how you use machine learning to understand the segments better.
Joanne: Yeah, I think scenario we really need to start understanding better in the consumer and sensory science field. We had a few abstracts come in for Pangborn, but not as many as I was expecting. So I think the real opportunity for experts to come in. I think it's a little bit like statistics. You know, if you work with a statistician and sensory consumer science, you really need to work with a statistician that understands sensory and consumer data. And I think it might be similar for artificial intelligence and machine learning. We need people to understand our discipline to help educate us and show us the opportunities for applying this area to our work.
John: I completely agree with that. I think the idea that different people perceive different things. The idea that when you take something, it may actually you may actually perceive it differently from me. It's not idea that the tech people really understand. They just think through our numbers and they're going to do their things to the numbers. So, yeah, that's that. You're right. You have to have a deep feel for that. Yeah. Alright, well, Joanne, we're actually out of time. So where can people find you? For one thing, if a student, it is inspired. So you've talked about you've got the kind of extended reality and various forms. You've got the mixed reality. You've got the holoLens. You've got your immersive environments with the rooms. You've got different implicit measures, whether they're the fit you know, physical measurements of facial muscles or I suppose skin conductance. Is that something also that you're involve in?
Joanne: Yes. I'm right next to the Red Institute, which also has in vitro digestion as well for looking at the impact of oral processing and sensory perception for example, on digestion and nutrition.
John: Yeah. So you have many, so if someone is excited about combining multiple sources of data and a valid like it seems like a dream situation for a student. So where can they come in if someone's interested in joining your lab? How would they find you?
Joanne: So that's quite easy. The website is www.massey.ac.nz. Then there's a whole website there with contact details on there. And they can also be following us on LinkedIn: Massey, which we've just launched. And they can email me directly as well. I'm happy for that.
John: Okay. We'll put your e-mail and your contact in this link in the description for the episode. Well, it's been great, Joanne. Thank you so much. So I really appreciate you being on the show. And I look forward to seeing what the coming years bring for your research. I'm sure it's been very fruitful.
Joanne: That's been great. Thank you very much.
John: Okay, thanks a lot. OK. That's it for this week. I hope I enjoyed this conversation. And if you did, please remember to subscribe and to leave us a positive review. Thanks.
John Ennis is a leading expert in sensory science and consumer research, with extensive experience in statistical analysis and product development methodologies.
