Guido X Jansen: [00:00:00] The topic for this session is, basically yeah, it's personalization. and why that's important. Why should we do it? When should you do it? And that's what we're going to discuss. we have grant and we have kneels since we're on a first name basis now. You all know, guy off the risk presentation?
I think so,
Guy Yalif: [00:00:20] unfortunately for everyone.
Guido X Jansen: [00:00:21] Yep. no introduction needed. but Neal's, co owner and a data consultant that the data story, anything you want to add to that small introduction? What did you do with the personalization?
Niels Reijmer: [00:00:33] yeah, so my background, before this, before I became an independent consultant, I worked at the bank for five years for Americans and it's basically Nordstrom.
and there I worked on personalization and building it up from the ground up. So from nothing, building yourself and start doing it and learn it along the way. And after five years of fuck, nah, I can do this also as a company for myself to hit her with that.
Guido X Jansen: [00:01:01] Exactly. And any specific tools you're using for personalization?
Niels Reijmer: [00:01:05] Preferably not. So it depends a bit on the client. tobacco-free built a lot of things ourselves, so we use for tracking, we use snowplow and then when crazy and any cloud platform, because we have the advantage to have engineers on hand as well. So I could say to them, you know what, I want to test this and I have this model and I want to implement it and make it scalable.
And they fro stuff. Technological stuff at me. And I was like, yeah, sure. As long as it works and scales, I'm happy. and now it's clients, it depends, it could be any tool. So some use home, build stuff, some use, the bigger, our data platforms. Yeah. He usual. Yeah.
Guido X Jansen: [00:01:41] Okay. And, at the first question, we already had the one in the APA during your talk.
So maybe you want to comment on that. isn't AI, just machine learning.
Guy Yalif: [00:01:51] It's a great question. Can we see people using the terms interchangeably all the time? Asking a bunch of data scientists, they would often say, look, AI is the umbrella, artificial intelligence machines, doing things that seem intelligent within it.
You've got rules-based, which is what most of us typically do when we start personalization and then machine learning where the machine is learning stuff on his own. There are other branches of AI. But tactically machine learning is a subset of AI as the big umbrella.
Guido X Jansen: [00:02:18] I think I've also seen a couple of fund meme saying that, if you want to sell it's AI.
If, if you want to sell it to companies or marketing people, and if you want to hire people, if you want to hire the smart people that were one of the work with us, you call the machine.
Guy Yalif: [00:02:30] That resonates
Guido X Jansen: [00:02:30] a lot. Actually. That's basically the main difference.
Guy Yalif: [00:02:33] And if you're raising money, either one works.
Guido X Jansen: [00:02:35] Yeah. Yeah. already had a question, another one for this session, from, I know that because he's too scared to answer, to ask it himself. And he's also not here. You ran away. but the question is, I think it's a very, common one, so most companies use personalization, at specific places on their own that way.
Sorry. It's but. the algorithms can, make sure that, the, the right contents, shown to the right person. but it doesn't really scale. you have many places on your websites that you can show different kinds of topic, but you need to generate that topic, the content for that.
and that's still something people need to do right now, at least. so how do you do this? how do you scale personalization in that sense? Once again,
Niels Reijmer: [00:03:20] in my experience, it's a transition, a company goes through. So the moment you start with personalization, for example, you have one content team creating content for one email, and that's being sent out the moment you start doing personalization.
Also, as guy told in his, presentation, you start rule-based. So you basically started with some segments and you send under set of one newsletter. You send a four. And often the content team can work with that and be a little bit more scalable, but there comes a point where everything breaks basically where you have 25 segments, you have machine learning, running, and everything needs to get together.
That's the moment when basically the organization for some part needs to be slightly or depending on how the organization is restructured content needs to be created on a different way. So in my experience, what. we have done in the past is, break content. So you have an image, you have text, you have different options, it's different buttons and you let them create that and let the algorithm, let it come to get.
Are there.
Guy Yalif: [00:04:16] I have a very similar reaction that as you're looking to scale, you should personalize throughout the entire experience. We find customers typically start with landing pages or homepage or checkout page, and then they personalize the whole thing. They might do something more, end to end, like I'm going to change the nav for everyone site-wide but then when they really want to scale, we find them doing two things.
One is just what Neil said, modularizing things. trying different combinations of things. Machine learning is pretty helpful there, cause it can automatically try all the combinations. The other way, one is to do more than what was premise of the question. Premise of the question was creating content.
You can personalize a whole lot more than the content itself. You can personalize the tone, the organization of things, what you show, how simple things are, what you hide. We have customers who without creating any new content, we'll pull things from other pages, like simple example, your B2B you're selling to businesses.
Somebody shows up, goes to a particular industry page. Great. Next time they show up on your homepage. Show that industry case study on your homepage, show logos from that industry on the homepage, greet them that way, all content you already have, but now it's more personalized meeting your prospect, where they are in their funnel with you.
Guido X Jansen: [00:05:24] How fast should you do personalization? as a, as an example, I did a user study, where customers, they went through the category, a page shoulder, there are products they, they already filtered on or whatever. went to the product page. There's not a product I liked went back. everything was different because it was personalized.
it really, it didn't work for them. how fast should that. B, how fast should you work that into the customer experience?
Guy Yalif: [00:05:50] In my humble opinion, that's it? That's a test. So you will have hypothesis, is somebody going to this category of products? They went to the soda page. Is them going there once enough of a signal?
Is this a product where that signal matters? if I'm going to, car forest site and I go to the soda page, does that mean, I only want to see soda? Of course not, but if I go to a car, if I go to. Oh, hell no. Rinaldo's page. And I go to a sedan. Should I see a sedan the next time I show up maybe not considered purchase it's worth doing.
And so in my humble opinion, there is no universal truth. there is so much of this it's worth trying out.
Guido X Jansen: [00:06:24] Yeah. are burned, remaking. It much harder for ourselves. If we, create this, these never ending smaller and smaller groups that we can test them.
Guy Yalif: [00:06:34] In particular if you're doing it with rules.
Yeah. But it doesn't scale. That's why I think we typically top out at a few tens of rules. Yeah. If on the other hand you say, look, I'm going to constrain the things that I know about. Like the example earlier, right? you're an answer to him. You see the Netherlands promotion. Okay. You know that there are a handful of things like that, or, I say, Oh, they're interested in the sedan fund.
I'm gonna show the sedan stuff, but the rest I'm gonna leave it unconstrained. I'm gonna let a system go try all the possible combinations. So I actually make fewer choices, not more choices then I think it can scale nicely. Yeah. Does that resonate?
Niels Reijmer: [00:07:10] Yeah, totally.
Guy Yalif: [00:07:11] Yeah.
Guido X Jansen: [00:07:13] Done audience questions on personalization.
Go ahead.
Guy Yalif: [00:07:17] yeah, I've got one question. Do you think that, you should always try to get as personal as possible or do people experience some sort of uncanny Valley feeling at a certain level? Do you have any experience with
Guido X Jansen: [00:07:28] where's the limits?
Niels Reijmer: [00:07:29] Yeah. there's certainly a limit, and I've hated a couple of times.
So we did a test where we predicted, based on. Not the information, but the shoe size of a client and not based on the shoe size, they filtered, but on other sizes they bought. So suddenly we showed a shoe size and what's pre-filled and then you get, I think also in an interesting topic where recommenders reach UX, which is closely related, and we had the possibility to go to a store and ask customers, like, how do you experience this?
And we had a couple of different ways to deal with this because you can. make it scary by how you present it. So ID, you can explain it. So for example, we had an option where we said to a client or to a customer, we predicted your shoe size based on the other sizes of clothing you bought.
Let's say, yeah. Now I understand fine with it. another option is not to show that we know your shoe size, but just change the sorting of the products and you will never know. So we change your experience without you. Are you seeing it or knowing it that we. Got this data from you. So there's yeah.
Playing field there, but it's certainly a appointed.
Guy Yalif: [00:08:36] I agree completely and would add two additional things. I think the line for what feels creepy continues to move, 15 years ago, 10 years ago, you go to a large company's website and you say, you're going to run multiple versions of your site.
They would say, it's going to look broken. There's no way you should do this. But now we all experience Amazon and Netflix and other places. We're used to it. We expect it even in our business environments. what was creepy before may not be now. I think right now it's in like a B2B environment saying what company you're from, because you did reverse IP lookup is like on the border saying, Oh, I know your name is whatever your, your name that for most people really is creepy unless you came from an email that was already addressed.
So that's not one thought too, you can ask permission. We have some customers that say, look, I want on a personalized and I want to do it in an extreme way. I'm going to ask them either to say, Hey, personalize this experience for me, literally as a button, they press on the page or ask them for segments.
We have some customers that like serve very different segments, like a marketplace. They have buyers and sellers and you can say, look, am I buyer? Am I a seller? And then the whole experience will change made a choice themselves, which is similar to the explanation. There's a rationale for it.
Guido X Jansen: [00:09:44] How would you figure out the creepiness of a page?
Because I can imagine that, personalization that works, within the session for a user, but it also creeps them out in a way that they might not return that they don't somehow it didn't like the feeling of this that didn't have the great experience that they were looking for. they bought something because they needed it today for whatever reason, but they won't be returning for this.
So how do you balance that?
Niels Reijmer: [00:10:12] How do you measure it? Or how do you balance it? how do you measure it is with basically all the tools you do AB testing with in that sense, doing personalization, in my opinion, is nothing, this similar, none doing irregular AB test, you ask your customer, you do your research beforehand, afterwards.
you look at the metrics returning clients, you look at customer lifetime value. Yeah. You take those things into it.
Guy Yalif: [00:10:33] I agree completely. It's qualitative and quantitative, same metrics. And then I think you said it like. Just ask literally, because there is that subjective individual creepiness factor that's worth talking to some people about.
Guido X Jansen: [00:10:45] Yup. Good. One next, go ahead.
Guy Yalif: [00:10:50] And Nigeria, I once plant in your head. So philosophy. Cool. Thoughts about personalization?
Big discussions in Brussels been going on for a while. about this idea that personalization it's basically about individualization me as an individual. And there's a very specific sector that you're impacted by this heavily, which is insurance.
The reason why is because how is risk distributed within insurance? It's the fact that we're all together and for that person that has a problem now. basically all that money goes to that. Person one way or another. And so where would our societies go? And I'm not asking the question. I'm just planting that in your heads.
As a seed to say there might be
Guido X Jansen: [00:11:37] businesses
Guy Yalif: [00:11:38] where pulling together as a group die. Dear underwear individualization might not be a good idea. So how far will insurance companies go in terms of personalization is probably one thing to see and watch out for that might be like a barrier to evolution. If, and if you have any thoughts
Guido X Jansen: [00:12:00] of scripts, you're welcome.
Guy Yalif: [00:12:04] I do think none on that specifically, but in similar things have heard people say, Hey, should this be considered, can this be. contextual variable in personalization. Okay. Technically, scientifically. Yeah, of course it can do we, as a society, want it to be, I think those are two separate choices.
You have the tool and then you have, what do you want to allow it to do?
Niels Reijmer: [00:12:27] I was thinking to me, it sounds a bit similar to a question in the previous podcast I was asked about if your client says do it and someone else's do it, will you do it? And there's also. If in a company that is something that needs to be decided, and if it's possible, a company will do it and it doesn't matter, make it right.
But at the same time, there are multiple laws in place for areas like this for insurance. I'm not sure. I'm not in that area of expertise. but yeah, I can imagine that there are, you always get bounced there, and high risks in that sense where it destroys basically the product insurance.
Guy Yalif: [00:13:07] And in other cases, trust, we had a customer ask us to do something that the motivation wasn't bad, but as we looked at it, we're like, this is deceptive.
And they said, I understand, but I'd like you to do it. And we said, no, we respectfully decline. We want do this for you.
Guido X Jansen: [00:13:21] We can test that, but me, we shouldn't be implementing it.
Guy Yalif: [00:13:26] No, we said, we want to test it. We want
Guido X Jansen: [00:13:29] the next question.
Guy Yalif: [00:13:30] How do you solve the. That's you can call it the Netflix bubble.
when a customer has a, some experience on a website, it's personalized, the comebacks comes back for two weeks, but after a year it's maybe a totally different person. Maybe he or she is married or as kits or something, So the bubble changes from the person's life, but does he enter the same bubble or.
You know what I mean? Yeah.
Guido X Jansen: [00:13:58] Or do you read, or do you reset or yeah.
Guy Yalif: [00:14:00] Yeah. So we've seen several techniques. One is just that resetting from time to time, starting from scratch and learning a new, another one is to have a bunch of rules around behaviors that trigger effectively a reset or move to a different segment.
You could imagine equivalent, let's say in a B to B environment, right? Somebody exhibited behaviors. Okay. I've never been here before. Great. They're a fresh prospect. I've been here four or five times. Maybe they're mid funnel. I submitted a demo request. All right there later. Later funnel, same logical thing, right?
They're at a different bubble based on their actions. Third is to try, with machine learning, to have something that's continuously updating itself. So it's constantly taking into account new behavior and it probably has some notion of waiting, or it could have a notion of waiting of more recent behavior is more important than long time ago, behavior so that when he or she exhibits that new behavior a year from now, you said maybe they have kids.
You can see, okay, now they're buying diapers and a year ago they were buying beer
Guido X Jansen: [00:14:56] and you can look at action, but you can also look to inaction. That also is a factor in machine learning.
Guy Yalif: [00:15:01] said.
Niels Reijmer: [00:15:03] Yeah. Yeah man. Maybe one small addition. If you're looking really from an, from a recommended perspective, what you're trying to do is normally if you look at a product recommendation specifically, you're trying to recommend products that fit the user really well.
So one of the strategies that often is applied is then to do a diversification around it. So you have 20 products that really closely matched the preface of the user, but you don't recommend those 20 year recommend five out of those. And then in the circle around it, which is close, but less, you pick two in the circle around that you pick two and then you get a diversification and then you get the option of the customer showing behavior outside of its bubble.
So they have their bubble, but you show information outside of it. And then applying techniques like. Taking that behavior into account in your algorithm, you can make sure they can escape their bubble.
Guy Yalif: [00:15:49] Oh, maybe that answers my second question, because when I'm in the Netflix bubble for say a month, and I saw oldie Marvel films, for a, for example, I want to see something else, but keeps posting me to same, movies, and I think like tired of it.
So I throw it at the Apple way or something or wanting to do something else or.
Guido X Jansen: [00:16:11] So
Guy Yalif: [00:16:11] that's a tricky one. I think so. Okay. Building on what Neil said, you can do that. And, something underlying, what you said is you can make a classically, there are four ways to go. Four classes of algorithms to go make recommendations like that, either on product or on content.
And the distinction that I think is particularly useful here is are you basing it on how items are similar? okay. You watched one Marvel movie. Now you watch another Marvel movie or are you basing it on user behavior? Similarity. People who liked things that you like and therefore similar to you, they'd like these other things too.
And that second one can then help you break out of that bubble of similar
content.
Guido X Jansen: [00:16:48] And, also in that case, I think, from a company perspective, in this case, Netflix, it's probably not a big issue for them anyway, because it's a really, it's the exception probably. So their ultimate thing for the larger group.
And if it doesn't work for that 5%, that are all the exceptions, it doesn't really matter. As long as it's worked for that 95%, is making them a little money.
Any more questions? Yes, go ahead.
Guy Yalif: [00:17:16] I guess this is on the same vein, but. In an ecosystem where you're doing like machine learning, testing, and a lot of personalization wouldn't, an excessive amount of personalization, dilute the noise and like the variants and therefore your opportunities for discovering new insights it's of the testing aspect, because if you're through Excel, so personalization, if you're creating cohesive, then you're limiting yourself from.
Highly variant experiences where you might find
Guido X Jansen: [00:17:46] you inside, or maybe there's an algorithm pro promoting that.
Guy Yalif: [00:17:52] yes, I think there are a couple of parts of the answer. One is literally that you're balancing, exploring versus exploiting to the earlier discussion, right? If you're creating these highly cohesive experiences and just optimizing for those, because they work, then you're exploiting the knowledge you have and you've chosen to swing the balance away from.
Yeah. From time to time. I want some randomness. I want some diversity. Maybe I get something other than Marvel. Maybe I get a non cohesive part of the experience so that you can continue learning. he's one part. The other part is if you're using a system that has a bunch of contextual data fed into it, you can then slice the data and learn.
Maybe the machine is doing it already, but you can yourself as part of the analytics after the fact. Glean insights that maybe weren't that obvious before, right? Okay. You have this homogeneous experience for this group, but when I slice it by mobile or weekdays or people who've been here twice before people who have been a customer more than three years, you learn something new and you're able to create a new experience out of the data that's already there, even though the thing is creating cohesive experiences.
Anything else?
Guido X Jansen: [00:18:51] Yeah, I think it makes sense to always have some part of your group that you always randomize or test for, do some multi arm bended, but there's also always a group that's, doesn't get the treatment, but it's randomized and what they get and see how that works. Maybe that's something better.
Guy Yalif: [00:19:07] So we tend to use that anyway, so that we, as folks doing optimization can say, look. I made us a lot of money because cause otherwise you do before, after analysis with all the problems that come with it. So we always encourage a whole back group, even though it's not for that reason,
Guido X Jansen: [00:19:22] but it helps. Yeah.
Yeah. Next question. Anyone.
let's see. Oh yeah, we do have a question. Yes.
Guy Yalif: [00:19:34] okay. What personal personalization.
Guido X Jansen: [00:19:36] You have a lot of
Guy Yalif: [00:19:37] options to fill the website. And how to
Guido X Jansen: [00:19:41] analyze
Guy Yalif: [00:19:42] all those options.
Niels Reijmer: [00:19:47] That depends totally on them, which state you are by the way. so I think this guy was already setting this in his presentation. When you start with rule-based, it's fairly easy. you have one rule versus the other, so you test for that and. The moment you get more into machine learning, you are starting to think, at least I started to thinking, all right, this is just algorithm version a, this is algorithm version B and version C and version D.
And now I'm doing a multivariate test where the content is done by algorithm content. A and the product recommendation is done by number B. And then I crossed that and then you move from there. It asks quite a lot from your tracking, I think because you need to track a lot of more stuff that's being shown to the user, but at the same time you're doing that already because that's, what's feeding your algorithm.
So to start doing algorithms, you need to measure a lot, like you want to know what did they see, what they click, how long, that sort of thing. And the moment you have that it's adding a version to it. So this is algorithm a and that's right. At least how I approached
Guy Yalif: [00:20:50] that problem. Two additional thoughts to add, in trying to analyze the data, I would suggest to start with the strategy, what's the business goal you're trying to achieve.
Okay. Which user flow corresponds to changing that where the big drop offs are at, we're focusing our energy here then just did it produce the results you wanted? ultimately right? Because you can analyze the personalization in particular, if you have a lot of contextual data, Forever infinitely.
just see, did it ultimately produce the results one that's the biggest, I think that'll matter then when you're trying to gain additional insights, you can use analytics to slice and dice a million ways. You could also use it to just find things that are unexpected, things that over or under index, relative to what you expected.
So you can say, Oh, we tried these two things and for everyone, they were showing it performed like this. But for mobile, it was like up here or for, in Belgium, it was down here. and those outliers can be instructive if they're not tiny audience sizes. And tuning the way you're looking at it, rather than trying to boil the ocean for those outliers, when they're big enough can be also helpful.
Guido X Jansen: [00:21:51] Are there any specific, personalization that when you have a new clients, they always say, okay, we should do this. these usually work, maybe in a specific vertical, but. What are the it's, you're both nodding very,
Niels Reijmer: [00:22:05] there are multiple tricks you can apply to her. a customer never comes in empty.
there's always, they're coming from another place they're coming in direct, via for is some device at a certain location in a certain geo. So there's a lot of information you already have that you can apply to some, in some sense through your website. So saying I always love the cold user problem.
They're called to a certain extent. You don't have a full profile, but they are never without information unless they say, yeah, I don't want to be virtualized ignore or cookies then. Yeah, sure. But then that's not my problem anymore.
Guido X Jansen: [00:22:41] Okay. Yeah, sure.
Guy Yalif: [00:22:43] w I think part of why it's valuable to try a bunch of ideas is because there are no universal truths right there.
It's one thing that works in one environment may not work in another. and so it's why you want to be able to iterate rapidly that having been said, we do, from experience, as I bet, a bunch of us have points of view on, on an eCommerce cart page, here's some high probability place to run on a checkout page.
Here's some on a PDP page. Here are some, okay. That could be a long list. The one that we almost always talk about that we all know we should do, but we don't is to treat existing customers and prospects differently. We treat them the same. Often because we're just accountable for acquisition. So we don't care.
No, but if you take that existing customer and show them the latest thing you developed or give them cross seller, upsell, or ask them to recommend, so you can drive more acquisition, you typically have the content already. And could you, yes
Guido X Jansen: [00:23:32] and no, but I guess also many businesses, they don't know the customer.
If they're returning, not every customer logged in, so that might be an issue for
Guy Yalif: [00:23:38] them. And you can then use the system to remember, have they ever logged in yep. And use that accordingly.
Guido X Jansen: [00:23:45] If they're using the same browser or a
Guy Yalif: [00:23:46] hundred percent just like totally. Yeah. Unless you're using a system that connects them, but that's a whole different privacy question unrelated.
Guido X Jansen: [00:23:53] Oh yeah. We already had that. but, so there are certain things that, that's that word. What would you say is the most surprising things that you saw in, in terms of, the personalization things that's unexpectedly. Had a huge effect or the other way around you think, this should have had an effect that didn't do any,
Guy Yalif: [00:24:11] so at least for me, the answer is like all the time.
There's not one, there's so many because. There aren't universal truths, right? It's not always true that red button works. It's not always true that, some huge changes what matters. And we've had things that like intuitively make total sense. You remember that somebody's been here before you show their competitor logos, you'd be environment didn't matter.
And then we have stuff that we thought would be wouldn't matter. And did we had a e-commerce customer that tried like recently viewed it items to drive up card size. They tried changing how the products were presented. Tried more calls to action, try bigger pricing, all this stuff. The biggest impact on that page in this particular one was emotionally affirmative language at the top of the cart.
This was selling women's jewelry. This looks great on you. Good choice. We like that in there. That environment in that moment, it made a difference. I could imagine in a B to B environment, wouldn't matter at all there it did. And so it's the. You asked for the exception. I feel like that's actually the rule that happens all the time,
Guido X Jansen: [00:25:08] but that's the great thing about our job, right?
Niels Reijmer: [00:25:10] Yes. Maybe one, one example that I experienced where we created a, carousel with, products that were pretty similar basically to the one you were looking at or have been looking at. and the funny thing, the funny thing was the click through of the carousel was through the roof. It tripled.
It was immense. The only downside was in the end, people bought less. So basically what we've shown there is the classical problem of choice overload where we say, yeah, here you have a tee shirt. You've looked at this and here we have a bunch of other shirts that are really similar. Have a look.
Guido X Jansen: [00:25:41] Yeah. We have time for one final audience question
Guy Yalif: [00:25:45] I had too, but, I'll ask one.
one was that on the top of my tongue,
Guido X Jansen: [00:25:49] you can ask the other, which
Guy Yalif: [00:25:50] is a Dutch expression. ITP. Is that a problem? The tracking protocol, because Safari, as only stores the cookies for a week and I know two weeks a week, I
Guido X Jansen: [00:26:02] think now they went back to the 24 hours. Yeah, really?
Guy Yalif: [00:26:08] But is that a problem for personalization?
How do you solve this?
Niels Reijmer: [00:26:14] It is a problem and it's a solvable problem. So move to service cookies. I know someone you can talk to now, at one point in time, there was a previous conversation who can help you with that? no, but there are multiple technological, yeah, developments there. So it is solvable.
and now it feels to me like an arms race that's going on, where there's now new technology, which, ignores ITP. And then there will be something coming up that will block that new technology again. So to me, it's solvable
Guido X Jansen: [00:26:45] problem and there's a lot of money to be made here, so it will be solved.
Niels Reijmer: [00:26:50] Yeah, exactly. Yeah.
Guido X Jansen: [00:26:52] Oh, he wants to comment on that one.
Guy Yalif: [00:26:56] I just had a last question. Do you think that's personalization and profiling is the same thing and should users
Guido X Jansen: [00:27:03] have a choice? There you go.
In the next podcast, you might have the answer.
Niels Reijmer: [00:27:17] I'm really thinking hard about this. I need a couple of born beers before afford this, I think, but I can have a shot. I think crystallization, it's going to be based on a profile and you can't do it without basically building a profile by doing it on why not looking at you as an individual, but looking broader.
It can be. Done. but I always think the customer has the full right to say, I know one personalization, I don't want profile. I want out. And then you do that. And then, like I said before, I literally say then they are not as someone responsible for personalization. They are no longer my problem, those customers, because I said, we'll give him the usual.
We'll give him the same user experience with filters and sorting, but it's not based on their behavior anymore. Yes.
