6 steps to get your experimentation culture running

Guido X Jansen: [00:00:00] Welcome. And thanks for listening to another episode of Ciro cafe. And this episode, I talk with the money from the group. Mariam did a presentation last week at digital elite alive about the six steps to, in it to create an experimental culture. And I'm definitely a one to follow up with her on that. So here we are.

My name is  and welcome to Shiro cafe. The podcast where I show you the behind the scenes of optimization teams and talk with their specialists about data and human driven optimization, and implementing a culture of experimentation and validation in case you missed it. In the previous episode, I spoke to how do I Swami kickstarted the COVID-19 conversion rates AIDS package.

And you can listen to that episode. Unless you're Oak fair websites or in the bottom, just app you're listening with right now, this episode of shearography is made possible by our partners online dialogue SiteSpect content square and convert.com. Welcome to season two episode 19. Oh. And before we start this week, we're adding a special new section for you at the end of the episode.

So make sure to stay on until the end. So my own. Thanks for joining us. you have a bachelor in data and system science, so I'm very curious. One of the things from that background that are still most useful to you today.

Marianne Stjernvall: [00:01:36] hello, and thank you for, for having me. that's a good question. studying computer and system science, we.

First of all. I think they nailed the first course because it was a double, a amount of time course where we started everything from like ITP servers, to the hexadecimal system. being able to understand the binary code basically. So they get rid of it. A lot of people in the beginning of the course,

Guido X Jansen: [00:02:09] basically it was a selection course, the right students for continuing the course,

Marianne Stjernvall: [00:02:13] something like that.

might be smart though, because it's always good to understand how things actually work like from the ground. What are the building blocks? Everything digital to work the way it does. And I think that kind of understanding, he's a little bit come greater at whatever you do within digital.

If you do have an understanding of how it works from the bottom up. Yeah. With that said we did a lot of programming as well. Of course, both front end and some backend. Which of course, has been a great advantage for me working within CRO because I've been able to build like all the AB tests myself basically.

and I'm noticed with so many other companies, organizations and people that when you can have so many great ideas, but the development part of it is like always the bottle, like. but it hasn't been that way for me. So I think thanks to that, I've been able to advance and just create this, high velocity, the testing, able to push a lot of experiments live because I've been able to code them myself.

And with that understanding, I always knew how long will it take for me to actually develop this experiment and get it live? So just from that, I had the greater understanding basically.

Guido X Jansen: [00:03:38] Yeah. Yeah. at university, did you get, courses in, of course in programming, but were, is it the same programming language that you use today to, build the AB test?

And because that's a common, at least here in the Netherlands that I hear from a lot of people have we get programming at the university, but it's totally different than what we actually use in practice, to build websites or build diabetes.

Marianne Stjernvall: [00:04:00] And I get that, With all organizations and companies, you choose your framework to work with.

And I had the fortune, at university to actually study basic HTML and CSS, but also playing on JavaScript and JavaScript is basically all you need to build tests. So that was great for me. and we also did some Python, which of course I don't use that much now, but being able to understand the, how to significant calculate, your tests and what our data scientists are doing too.

be able to build like a monitoring tool through Python has helped me a lot because we can have a good conversation around

Guido X Jansen: [00:04:40] it. Yeah. In the old debates bison versus are, I think I know where your preference lies.

Marianne Stjernvall: [00:04:46] I don't have a preference that much because I'm not the one to build it, but if I can have the understanding of it, I think that helps the conversation a lot.

Guido X Jansen: [00:04:54] Nice. So I was wondering, so how did you acquire the nickname queen or Shiro?

Marianne Stjernvall: [00:05:00] Yeah, that's a good question

Guido X Jansen: [00:05:03] because you already got that at your first job, right?

Marianne Stjernvall: [00:05:06] Yeah, it's crazy. I actually, I worked at a company called I prospect, and in that company we had a small team called, We named it, the duck team, it's spelled D a C not duck acid duck, but it's pronounced exactly the same.

It stands for, dashboards, analytics, CRO. We were very small team with four people or something. because at that time, like six years ago, not that many were working within these areas, and a coworker of mine. Out of nowhere, I guess I want to say, because I don't know how he came up with it, but he just started calling me the queen of CRO, because I guess they were so amazed with what I was doing or, I was the only one working with CRO at the office at the top.

Guido X Jansen: [00:05:55] So then you automatically become the queen. Yeah. That makes sense.

Marianne Stjernvall: [00:05:58] And he spread it around. So I guess it's thanks to him. And that's how I got the nickname, basically.

Guido X Jansen: [00:06:05] Exactly. And, and today, we moved to a two in the meantime, and today we're talking about the six steps to get your experimentation culture running.

I think that's a topic that, are of interest to a lot of sheroes, especially when they work in a company by themselves. Or just a small team and have a lot of people to convince. but before we dive into those six steps to get a sense, of your role at right now. So what is your current team look like?

And, yeah. Can you share a bit of, what you're working on right now?

Marianne Stjernvall: [00:06:34] Yeah, of course. So today I have a split role at two, actually. so I work 50% as the, CRO and UX lead for two in Nordics. And 50% of my time, I work as a CRO manager for our global function within two week call to a group. so they serve of course, two different purposes and my Nordic role, I'm responsible for, the team that works with CRO and UX.

this team has been built up, through the past year, basically. We, or the organization made the decision, last fall. So about half a year ago that CRO and UX would actually be together, before that it was only CRO. but since they're so close related and within my MassArt at university, I also started in direction design.

So UX is very closely related to me and what I know from before. So for me, that made total sense, to merge those two together. And from that we build the team. So basically what we did have was, two CRO specialists, two UX, the signers, two UI designers, one analytics person, one dedicated CRO developer, who worked like cross function to the product teams, that we had.

I say that's how the team was and what that did look like because, because of the coral and a pandemic that we are in there have been changes. And I did post about this on my Instagram. or sorry, my LinkedIn some time ago, which has been really hard for all of us and especially, being in the travel industry.

There are some team changes right now, and we will need to build up again once we get out of this, but that's the structure that we have had and that we are going to have a, when we get out of this.

Guido X Jansen: [00:08:33] Yeah, I can imagine, especially since you're in tourism, a lot of changes in customer behavior and, the revenue bonds have been, favorable frigging.

I can imagine. Yeah. so let's just get into those six steps, to get your experimentation culture running. I think it makes sense that maybe a quickly mentioned a, the six steps first before we dive into them. so it's be your hero. running masterclasses, results. number five is, what I've learned this week, or that was number four, number five, zero showcase, and number six, the monthly dashboard.

So let's start with, with, be the hero.

So what's the, so what's that one about, you need to be the CRO hero of your company, maybe the queen, even depending on what you want.

Marianne Stjernvall: [00:09:21] Actually, this is a little, a little tweak to this one. It's a little foolish actually, because I do display forest, be the hero. but it's not actually what I mean.

what I'm trying to, get people to understand here is that you need to be very humble in this role, zero specialist in whatever organization or at you need to be humble about what you are doing and what you are changing on the site. And by that you need to involve a lot of people and always learning preaching as well.

But teaching. Your stakeholders and people close to you about what zero is like, why we're doing this, teaching them about the users and how it could work. and by doing that and getting people more involved, you are giving them the insight that they need in order to make the right decisions. That's basically what's, there is, when you're doing that, you feel. Maybe like the hero, but what you're actually doing is you're making the people who you work closest with to be the hero themselves, because they can reach out even further to the company and say, are web developers that this, and we know that this is the right way to go because we have this data on it.

And then they can show that what they are actually doing right now is the best possible choice. Yeah. So it's not about you being the hero. It's about making everyone else be the hero

Guido X Jansen: [00:10:52] you need to meet an invisible hero. The person that enables all of us in the company.

Marianne Stjernvall: [00:10:57] Yeah. So you can give them the insights needed, then the recommendations and the data, and basically the knowledge to be able to.

Make them make the best decision.

Guido X Jansen: [00:11:05] And you said in your presentation, not everyone will like you when you started with the CRO. So I'm very interested in the personal story behind that one.

Marianne Stjernvall: [00:11:15] How did you know there was a first, there is definitely, I can, there are several I could say. and, That's very common.

I think you have to understand coming in as a specialist, that you will not be a very liked person to start with a, so let's take the example that happened to me. I started there about three years ago and you know how it is when you come in like your first days and you're walking around with like your bus or someone and getting introduced to a lot of new people, some UX designers, some product owners, and.

Then we walked around to say hello to the developers, coming onto one of the teams that worked with the CMS or the site. I said, hello to a guy called Amy and I got it and producing. This is Maryanne. She's our new CRO manager here. And at the time we had this AB testing and personalization tool called

Which had, had some issues, maybe not the tool, but the people behind it, before me. So I was introduced with a word saying this is Maryanne. She is starting to use CRO manager and ambulance response was, Oh, you're the new frost Mo

Guido X Jansen: [00:12:37] sure. You were already a many points behind,

Marianne Stjernvall: [00:12:41] definitely. Oh yes, definitely.

And I'm grateful that he said that in one way, because it made me, reflect a lot upon okay, so this is not an issue with me. Of course, not personally, but it is an issue that has been here before. So I needed to understand, what are those issues? How come you don't trust? The tool or what is it that you don't trust and why are you calling me a tool's name?

Is it really the tool or something else?

Guido X Jansen: [00:13:12] good to know.

testing professionals, Nazi and product recommendation or blusher. , this is on the tax office clips for doing optimal performance.  SiteSpect pushing 80 minutes for tagging. I think guns will flicker effect Dave and

in impact have opened our beta tester. Impressional Jayda, med SiteSpect

Yeah. And let's see, I think a lot of 'em. Also with a, with developers when you start out, they see it as an extra thing on top of their work, right? you're creating extra work for them, which they not necessarily like a while after, hopefully I'll throw a couple of months, they might realize, Hey.

but the sheroes are actually, trying to validate and test stuff. Before we build them or we build like an MVP and not the whole thing, but just an MVP. And that actually saves money.

Marianne Stjernvall: [00:14:19] Yeah, definitely. I think like I totally get the developer side of things. I think it's okay. A big fortune that I come from that side as well, because a lot of people working with CRO, are creating AB tests on top of a code that is actually built by developers first place.

And they are the ones who are responsible for, if something breaks like everyone would call them. And they might think of, Oh, maybe this is this test. I need to call our CRO specialist about it, but they are responsible for it, but maybe they're not interlinked all the time as well as they actually shouldn't be.

So you really do need to think about that relationship. And making it AB test is just manipulating the code that is already existing on the website. learning how to code dynamically for AB testing so that you can safe fit so that you don't actually ruin the code on the site.

If they would make like an update or a release or something that's essential for having, I think, a secure or more risk-free CRL program

Guido X Jansen: [00:15:24] and, of, sorry, your second point, is running a master class.

Marianne Stjernvall: [00:15:28] Yeah. Exactly. as to we, we have masterclasses in the house, everyone has them like whatever kind of area you're working within.

When I had created, some numbers, the closest stakeholders basically knew what zero was. Ah, then I asked if I could set up a master class just for Sarah to spread like the knowledge, because that's one of the most beautiful things would CRO is that it's so easy to show.

It's so easy to show inspirational cases from like Google or Netflix or whatever, great companies that are really driving this experimental culture. so just taking that,

Guido X Jansen: [00:16:10] and so do people get, can they follow that from the whole company? your company is not in one office, so is it like an online masterclass that people can follow is like on demand or, periodically or has it worked?

Marianne Stjernvall: [00:16:22] we do it physically and also we do it on, online. so it's live. So because I want to have that kind of interactive class, whenever I'm basically doing stuff. and also we have so much material now that we have tested all over to it. So basically in that master class, I can also ask them. So what do you want to see?

Because whatever area you're working in, you're interested in some part of the site. So then they can say, okay, I'm interested. And the checkout. Then I can just like that show them five tests that we did in the checkout so that they can get a greater understanding about what they actually care about.

And I think that will make the learning more fun as well.

Guido X Jansen: [00:17:05] Yeah, exactly. And, and how do you get people then, to sign up for the masterclass?

Marianne Stjernvall: [00:17:11] we have an intranet where all the most classes are and add, we, have 40. Learning hours every year. So it's they really promote you going to these master classes to learn about all areas within the organization, which is great.

so you can sign up, for those just there or. I also tell people because some people come to me and they're like, Oh, I never see when they are on our intranet can just give me a note when you have them there. So you have to help out,

Guido X Jansen: [00:17:40] it's already in your business culture to go to masterclass, that's already a thing.

So it's not just the zero mosque classes. There are master classes for all kinds of things.

Marianne Stjernvall: [00:17:48] Yeah, definitely. But if you don't have that within your organization, a masterclass is basically an email invitation. A room or a digital setup and a PowerPoint. And you, you can do it.

you can just say I'm having a learning session for CRO, for our company sign up for it. It doesn't need to be that structured.

Guido X Jansen: [00:18:09] Yeah. And how often do you do those? So mass classes,

Marianne Stjernvall: [00:18:12] I do them like four to a three to four times a year. Okay. So quite often, but. So in the beginning, like more people obviously come and then it will slow down.

But as I also want to say, it's keep that up and don't like, don't let them go. Of course you can do fewer as time goes by, but always keep them going because new people will come to the company and you want them on the same page too. So don't care that less people come because You wouldn't go to the same master class twice either.

Guido X Jansen: [00:18:44] Exactly. No. So it's a similar Moscow clause, each time, but especially for a new ones is really interesting to invite those rights, to get them into the mindset of experimenting.

Marianne Stjernvall: [00:18:54] Yeah. Because there, you really have the chance, right? Because they're new coming into the company to show them like, this is how we really want one to be and how we want to act and the culture we want to set.

Those are easier than the ones that have been with the company for 20 years.

Guido X Jansen: [00:19:09] And then, the third one, the results and sharing those. So how do you do that?

Marianne Stjernvall: [00:19:15] It's a kind of a tricky one that, I set into several steps. Just that one actually, because the results is everything when it comes to the work of CRO, but mainly what I mean is that.

When you first do your, like the first AB tests, just gather like all the closest stakeholders, after every AB test is executed and sit down with them to actually show them the depth of the data and the insights, and also hear again, teach them about it. don't only go through like the top findings.

Teach them about what happened? What did we change in behavior? And also all of these stakeholders have like their own darlings. Everyone has. So if someone, I think it has changed, they probably, we will ask you like, okay, so more people signed up to the newsletter, but how many logged in? Because that's my darling.

I want to know. and then you can actually show them that you're so good at what you do and go into that analysis tool and show them the numbers while you're in the meeting, because then it also shows them the depth of what you can see. And you can also open up the floor and say, do you have any other questions?

Look at all this data, we can find out anything and you will get better ideas as well because you learn from the organization what's important. so that's one of the most important things I would say. About results. just taking that time with the people. 50% of the year or work is basically people who work so really engaged with them.

and also then later on that, open up the results to everyone and be transparent about them, so that you also get to share everything that you've been doing, but when your stakeholders have that understanding. Of the results, then let them see everything else you've done because then they can deep dive themselves, and grow away with even more.

Guido X Jansen: [00:21:16] Yeah. And about opening up those, those results for everyone. how do you do that? How do you give access to everyone, to those results or instruction tools you use for that internally or.

Marianne Stjernvall: [00:21:26] so we have a documentation online, which is basically a link that anyone within two we can access. Yeah.

So we have it in Atlassian, which is really convenient, for the documentation and some have it, in just a one drive folder or anything that you can share with the organization basically works. I would say then we also have a bigger like monitoring tool. That is also just with that link.

And the two of VPN, you can view everything that's online and how all the experiments are going right now. But we actually, didn't we didn't release that to everyone until maybe two years after I began at Stewie because. Seeing experiments when they are running, ask the stakeholder and the product owner, seeing one experiment, you're like, Oh look, it's up 20% after two days.

They will take actions on it directly. And we know that you should not do that.

Guido X Jansen: [00:22:25] Becca uninformed ends our beta test and happy Oak lost from the beginning of flickering. A very artist did come through the key test. Tessa was not to be included and imposed if Tesco's that. No Tyler's year. be a convert, comes our beta testing software, the smart insert heat about any flickering of  fifth gear Snelder support fee, a 24 seven jets.

The health would go up and, but dive is the last minute. If it was 15 key, carbon positive, you do this yourself. You have a knife and the fork and the  during the

Yeah. So start there are, if the culture isn't there yet, there are risks and risks in sharing all the data. And showing if they don't have the context on what an experiment is, how it should be run and how that works, it's difficult. It might be, working against you if you share everything.

Marianne Stjernvall: [00:23:19] Yeah, no, absolutely. yeah. I think you should be very careful and understanding what they are. What is their level of knowledge compared to what are you showing them? Yeah, in the beginning, I was really like, not sharing anything basically. So you say, for example, let's say you have executed an AB test.

You have it finished and you do the analysis. Then you invite the stakeholders to that, like first meeting or you are going to go through the analysis. In that like meeting notes or invitation, I will not, I would not share the PowerPoint presentation with them at first at all, or any results I want to be there care to explain the results.

Do them not just send them on the paper because everyone one will interpret that in their own ways. Yeah. Especially when you are not used to seeing these numbers. So at the beginning, I was very like holding back with what I was showing. And then as time goes by, and again, when you have been teaching them and when they have learned more and more, and their knowledge is getting bigger, then you can open up more and more as well.

So it goes to bed hand in hand. You have to be careful about that. I think

Guido X Jansen: [00:24:30] be aware of the level of knowledge of your direct colleagues and the people that you're showing this too. Okay. And, so you do use it lessons. So is it, does it make it searchable? Is it searchable for people and that's gender, the results from the test?

how do you handle that? that's an issue that I see a little shooter. I was struggling with. I have all these maybe power points. but how do I make it searchable so that in a year I can still find that test that I did.

Marianne Stjernvall: [00:24:55] Yeah, no, I think that's very hard. I think a lot of that knowledge lies within the people who actually made the AB test.

Which is dangerous for the organization. And I think that's really tricky where we don't have a great solution right now. We're looking into some of them. we're doing that on. On the group level, which is the more like strategic, zero road. And, Yeah, I will take any tips that anyone possibly have because we're looking into several different possible solutions.

I have a big vision. I wanted searchable what labels, you want the winners and losers. You want everything to be searchable. I want all the tests from the past three months on the stock page, which were winners only on Facebook traffic,

Guido X Jansen: [00:25:36] Yeah, exactly. Yeah. We all will. Not one.

Yeah. on to number four, Four, it's a weekly meeting that you do a, and then sharing what you learned this week, Or not necessarily you, but your team. Exactly.

Marianne Stjernvall: [00:25:49] And that's the point that you said, it's not me. I have nothing to do with this meeting. The only thing I did was to set it up and I didn't even do that because I stole it from Luke  Google.

I heard he was doing this with his team and I really loved the idea. So I thought, why not try it out? And everyone loves the meeting. So the meeting is basically all about what you learned this week, about the user. So it's a Friday meeting that a little fun thing to close the week, basically. And it's the people meeting.

This is the chance for everyone within my team or anyone else for that matter, who works with the user attitude to be able to. Get in there and show, what did you learn this week about our user? And that's a place where we share, of course, all the results that we have from the AB tests, that closed that week.

But it could also be like new user tests or heat maps or some other insights you have about like the whole user journey, basically. and it's also the important thing here is that this is a place for the people to actually shine. And lift up, what did I do? what did I do this week that I want to show off?

and not be as humble anymore, just like share it and brag a bit about it and get applause. And we're all happy for you. And, yeah, just want to bring that kind of fun and exciting feeling to it. Yeah.

Guido X Jansen: [00:27:18] And as I understood those weekly meetings also lead to a weekly newsletter.

Marianne Stjernvall: [00:27:23] Yeah, exactly.

So after the meeting, I send out the CRO weekly, which is basically containing like what new tests are live, what tests are still alive since last week and what tests have been finished and which have been analyzed. And that's just a real, it's a one page, one page, your per experiment, basically.

so it's short. But it should be just an overview of, what the website is looking at right now, because everyone is so dependent on what the website is. And when you have a see that a bit in your experimentation program, and you have a lot of experiments live, a lot of them like taken down and then pushed back up and so on.

It's a lot of changes that's going on and I've noticed I'm getting more and more questions from From developers or from customer service or from other parts of the organization saying is this a test? Is this a test? Yes, my friend Amy tests, because everyone is getting different of the site.

Yeah. this is a way for everyone to be able to see, okay, what's the current setup of this app

Guido X Jansen: [00:28:33] this week, and also things that are not a test. that's usually my experience. When you started with a company that didn't do any AB testing before and you start AB testing. Then after that is when everyone knows there's AB testing on everything.

That's going wrong. Going wrong on the website. It's every little mistake it's Oh, let's ask the AB testing team or person.

Marianne Stjernvall: [00:28:55] Exactly. no, it's totally like that. so this has proven to be really helpful actually.

Guido X Jansen: [00:29:01] I think a struggle, at least I have an a, I know from other sheroes that when you weigh in trying to engage those people.

So you have a CRO newsletter that you send out to, to the company. how do you balance what you, if you overshare. They won't be reading it after a while anymore. They don't know what look. it's not the only newsletter people will get that. how do you balance that? how do you, what do you share and are there specific things that you don't share in those newsletters to not overwhelm them and to have them keep reading what they need to read?

Marianne Stjernvall: [00:29:31] Yeah. So it should be very clear. I'm very sure this newsletter or, if they're interested in more, they have a link in the newsletter, which takes them to the documentation pages for kind of these tests. So this is basically very simple bill overview. this is the start page. These are the such tests.

This is the checkout and that's it. And even I don't. I want to make it clear, even from the start of kind of opening, opening it up, what are the highlights? So that people who know that, Oh, there's some highlight within my area. I should check on that. But if you're not highlighted that week, basically you don't have to check it if you don't have time.

So you should make it like really quick for people to understand

Guido X Jansen: [00:30:17] on to number five, zero showcase.

Marianne Stjernvall: [00:30:20] Oh, yeah, the showcase. I like that one. I like all of them, of course, but the showcase is fun. the showcase is something I started with and have down net many workplaces. Now, it's a place at you. We started with it about two years ago.

And the showcase is a monthly meeting, basically where everyone's invited it's, both online and offline. and yeah, who do there is that we show off the greatest things we have learned by AB tests at jewelry during this month. And it's just a great place for, as you said, like these newsletters and weekly meetings, like you don't want to overload people with information, But to have something once a month where everyone can come, that's enough for a lot of people, the ones who are not on the newsletter or don't come to the weekly meetings, the ones who are a bit further from the CRO subject, but are still interested in what's going on with the site. those people are the ones who come to the CRO showcase, even our board members and very high management.

And, people also come to these meetings. And, so we had that within to Nordic for awhile. And then I had a conversation with, with a colleague from Belgium and he asked me like, Can I join your showcase next month? And I'm like, yeah, sure. By the way, do you want to show like a test then since you were like joining you're like a guest they're like, yeah.

Okay. Let's do that. And the next month, I think it wasn't the Netherlands who also wanted to join. And I said, do you want to share a test to then maybe? And then it was, it felt like you were a vision show or something like calling out to different markets and like welcome Germany. But then we just decided to rebrand it basically and make it into a global CRO showcase.

and to not sit there for five hours to talk about all the tests that we did. We said that each market can show like two to three tests that they have learned the most from, good or bad, but learned the most from during this month that they want to show.

Guido X Jansen: [00:32:26] Yeah, exactly. that was what I was wondering.

So it's not only the winning dust at your show, right?

Marianne Stjernvall: [00:32:31] no. It's both the good, bad and ugly. Yeah, of course the ones you learned from the most are often the losers, right? Yup.

Guido X Jansen: [00:32:41] Exactly. Online dialogue.

does ANet specialist

At team

and ultimately this year salmon Mecho

sales funnels and customer journeys. Mia info. Hi, now online dialogue, fentanyl. And then we have the, the sixth one, steps to get your experimentation, cultural running gloves. One is the monthly dashboard. I assume that's connected to the monthly meeting.

Marianne Stjernvall: [00:33:24] Yeah, it is absolutely. They're both done on a monthly basis.

The monthly dashboard is something that we have worked a lot with during the past year. and I would say it has come from the global CRO role that I've had for a while. Now it's a way of trying to show higher top management. What is the actual value of, and would CRO. Like the million dollar question, and we have agreed to some primary KPIs that we are always looking at.

So in this monthly dashboard, we show how many tests were executed this month divided by market because we still work and divided by each market. So that's the one thing we're looking at. and then, sorry, we're also looking at something called a margin attribution value. So we're trying to look at if we would, implement this test a year ago.

Okay. How much margin would we have earned? So it's like a prediction modeling that we're working with a test that are the best and winners. So that's a margin revenue number we can actually bring forward. But we're also, as I said, like the losers are often the ones who learned the most from, so we also do, on the opposite side of it.

We also do the margin loss prevented. So we calculate the losing tests and how much we've actually prevented to if I'm losing, because we didn't implement that. Yeah,

Guido X Jansen: [00:34:58] exactly.

Marianne Stjernvall: [00:34:59] Yeah. So that's the monthly dashboard that we're displaying, but in there, we're actually also doing a bit of a fun twist to it.

So we're playing the monthly winner. So that's one AB test from all the markets that is announced as the winner for that month. It's a, it is the one with the most increased conversion rate or margin attribution, some of our top goals. Yeah, and this is just like adding a little bit spice and a small, like fun competition in there, like a friendly competition thing.

but it's also like a good, attitude. we have different websites today for Germany, Belgium, Sweden, et cetera. And this is also a very good, like learning thing for the other markets to see, Oh, you did this in like the search result then probably all the markets should prioritize that one for next month.

Guido X Jansen: [00:35:50] Yeah. Yeah, exactly. Yeah. And, and the copy and revalidate those, those experiments from other countries.

Marianne Stjernvall: [00:35:56] Yeah, exactly. And then we have a, from that, now we have a great prioritization list with winning tests through months that we can follow up on. So it just gets bigger. I'd say, and you get a better view also around like different markets, what works and what doesn't work between different markets, but also very good and stable prioritization lists.

And the best data you can have, there is this is actually the same company that you're in, but in different markets you should probably test it.

Guido X Jansen: [00:36:26] Yeah. The do already. Do you remember, any examples of that things that worked in one market and didn't work in another market?

Or

Marianne Stjernvall: [00:36:34] there are, I would say everything.

Guido X Jansen: [00:36:39] That's also a good learning, right? So to revalidate everything in different markets. Yeah.

Marianne Stjernvall: [00:36:44] I'd say so. I mean test that we've been doing the same once in UK and Nordics or in Belgium and UK and so on. They always have slightly differ in results. Yeah. Me having worked with like the Nordics also for three years, we're saying the Nordics, but it's actually for individual markets, it's, Finland, Norway, Sweden, and Denmark, and we always have different results.

But saying that if you would, for example, go to the market then of UK, you could also say, okay, so UK has one result. That's not really true either, because if you drill down into demographics of UK, you will then get up the results. And within those groups, you have other results. Just a matter of how far do you want to drill it down?

And what results will you look at? That's why we have personalization.

Guido X Jansen: [00:37:35] Yeah. So w we earliest spoke about leaving things out of, out of the things that you report, to stakeholders, if they're not educated enough, depending on the level that they are at, is it similar with the monthly dashboards or other things that are specifically not in there?

Marianne Stjernvall: [00:37:50] So the things we, show in the Munsell dashboard, it's very narrow. It's actually only these like main KPIs that we do have. And then it's the link to the presentations that we now I'll have shown them because everyone can join it. And we want to create that person awareness and engagement from people, but it doesn't have to be that detailed.

So these presentations that we show are more like. Those have a storyline, which is like easier to understand for sure, for anyone. And then just more of an inspiration, basically. not that digging deep into the data.

Guido X Jansen: [00:38:27] Yeah. Yeah, exactly. So my, thank you so much for sharing these six tips with us.

some of my final question for you will be, if you have any, do you have any books that you'd like to tip to our audience?

Marianne Stjernvall: [00:38:40] Yeah. I have one favorite book of all times. it's called the design of everyday things. Have you read it? Don Moore, Norman? Yeah, exactly. Yeah. I think I read it at university. And since then I've read it like five times, I think.

it's also one of these books where you understand how stuff works from the beginning. Why do I understand to reach for a handle and how do I understand to push a door? And some of that book gave me clarity to understand and put words on what doesn't work. And I could actually understand what the problem is instead of only feeling frustration.

And I think that kind of. That is exactly the same, what CRO was, but with digital devices.

Guido X Jansen: [00:39:27] Yeah. Yeah. I think, the book, we will definitely link to the book, in the show notes of the podcast. And I think the book has already, it's quite old already, right? It's it has nothing to do with the internet.

No,

Marianne Stjernvall: [00:39:38] nothing. They're talking about doors and lamps and everything, but you can figure out yourself and understand the connection between them. And that has, for me been a very good exercise that I did from the book, actually taking these words like affordance, for example, taking that from the physical world, a world and putting it into the digital one to understand.

Yeah. How are those gonna map together?

Guido X Jansen: [00:40:03] Exactly. Yeah. A quick search, told me that the publication date is, 1988. So yeah, that's great. So it's the same principles, Then they're also trying to explain that to people. when I explained, so I have a background in psychology, how do you apply that in online?

And indeed the study had nothing to do with align, but it's still human behavior. it's still a lot of the same principles apply. And the nuances are different here and there. And of course we have a lot more data online, which is great. but it's still a human behavior that we're looking at and trying to optimize.

Marianne Stjernvall: [00:40:36] Yeah. I think you do go back to where we started with this talk. Like you have to understand how this stuff works from the lowest building blocks to really understand why we are behaving as we are right now. Yeah,

Guido X Jansen: [00:40:51] I think that's a great summary, Marianna. Thanks. You so much. Good luck with what you do and, good luck with, yeah, the whole situation with, within travel and tourism.

I hope it's a, we'll recover fast, fast enough to, for you to at least start rebuilding.

Marianne Stjernvall: [00:41:05] Yeah. Thank you so much. And thank you for having me. Thanks,

Guido X Jansen: [00:41:08] bye. Bye. And that brings us to the special new section I mentioned in the intro, our sponsor contents, crass, quite some clients, and quite some data on things like traffic sessions, dime transactions and conversion rates split over 20 industries and a couple of other dimensions.

And the amazing thing is that they have. But all of this in a public dashboards and they are closely monitoring the effects of the dynamic only commerce giving us some very detailed insights, how these different industries are effective. So from this week on worth, we'll have a small section in the podcast in which they will update us on the latest developments.

Say hello to  from content square

Tjeerd van der Putten: [00:41:49] to provide understanding. During this uncertain time, we are monitoring the impact of Corona virus from online consumer behaviors. Every week we analyze billions of online sessions across 900 websites to understand online behavior and to see what trends are unfolding across all industries this week.

To understand the impact of the crisis on traffic transactions, they choose and more we've compared data from the past week to the period immediately preceding the global reporting of the outbreak or the first six weeks of the year, which we call the reference period here is what we service the past week.

Generally speaking, it seems to the story this week is stabilization. We've seen a steady growth as digital continues to be the only store which has started to level off this week. Here are three key stats. Traffic overall continues to grow steadily it's slowly with a two and a half percent increase compared to 2% the previous week.

This steady growth over the past two months has resulted in a 29% increase in the number of site visits since before lockdown transactions globally. And cross industries has grown by a massive 48% since the first quarantine orders. And after an initial period of rapid growth, the conversion rate has now stabilized at around 15 to 20% above normal pre COVID-19 levels.

So we're seeing growth that has significantly at seeding the numbers we are seeing pre COVID and this week stabilization can start to give us an idea of what the new normal will look like for online brands. Now, we will take a quick, deep dive into three industries that have had some interesting trends this week.

And examples from brands who are getting it right. The luxury industry has had some interesting peaks this week. Luxury rents were one of the first to be effected when COVID first spread to ship to China, the country who citizens accounted for 90% of global luxury market growth in 2019. However, the industry has started to take a turn luxury brands have seen a 5% increase in visits to their sites in the past week and the 90% boost.

In the volume of transactions, we have seen many of our luxury clients, launch corporate social responsibility messaging with brands, such as Tiffany and co burglary and bell staff implementing strong homepage campaigns. Belstaff a luxury British clothing brands are offering to donate a jacket or every jacket bought on their side to the homeless.

They ran an homepage campaign, which has resulted in a huge increase of interactions with the content and users who interacted with the content were 40% more likely to purchase. It's been great to see all the different social initiatives, luxury brands have been launching, and it seems to be paying off a win for both customer and the community.

Second off is travel. It's been the hardest effected industry by far, but this week we have seen signs of growth and recovery. The travel and tourism companies are starting to see an increase in time, spent by users on their website this week, although still 25% below normal. But this is a great sign for companies that users are starting to consider transacting.

Again, this industry has taken the hardest hit in terms of transactions where the steep drop of 88% of the whole covert period. However, the last three weeks we've seen an improvement. With an increase of transactions of 14%. This week as locked down, rules are loosened and users are more hopeful. This is a great time to start encouraging your customers to start booking for future dates.

Many of our clients have launched summer 2021 campaigns with great engagement rates. Lastly, moving on to an ever popular industry. During this time, supermarkets e-commerce has had the most dramatic story of all from a spike pre quarantine of three times. Normal traffic. To massive conversion rate declined due to delivery and supply chain issues with brands.

Now, responding to these changes this week, supermarket websites continue to see a traffic search with 133% boost, but more interesting transactions are seeing a big significantly. For home and do it yourself. DEC and sport products. Conversion rates suffered a cut of 50% back in March, but has recovered since then.

And this week we are at just minus 13% below the precollege levels. We've seen some really interesting innovations by supermarket brands, for example, Tesco in the UK. Effort back to shelf button, which place when products are out of stock, we assuring users that similar items are available. This is a common theme we are seen in the, in this industry with COVID delivered three and product reassurance, playing a huge part in reducing user frustration.

So that's all from us this week. If you want to see KPIs for your industry, be sure to check out Colton squares COVID-19 e-commerce impact data hub. It's updated every Monday with the freshest behavior data across all sales.

Guido X Jansen: [00:47:50] Thanks so much Jared for sharing this very interesting stuff. And yeah, if you want to explore the data yourself, the link to the dashboard can be found in the show notes.

This was the season two episode 19 of Shiraz. Shearography with my, on the Campbell from the two week group. And in case you're interested in my own little also be speaking at digital elite day in London on September 29th. More info that can be found on digitally day.go.uk. And as always, this show notes of this episode can be found on our websites $0 it's cafe.

Although we started out as a ditch podcast, putting out more and more English content, and if you want to skip all the Dutch gobbledygook, please go through Sierra that cafe slash English to see an overview of our English episodes and to subscribe, to get notified about new English episodes. If you're interested in promoting your products or service to the best Ciro specialist in the world, please take a look at ciro.com/partners.

To see how we can collaborate, talk to you next week and always be optimizing.

‍