Decentralized versus Centralized CRO

Guido: [00:00:00] thanks for, joining us on the CRO.CAFE podcast So my first question, of course, what's your background? Can you introduce yourself a bit? And of course, how did you end up working in CROS

Andrea: [00:00:08] of course.

So I'm Italian and I started working in Paddy Power Betfair in the gaming team. Basically what we were doing was like bonuses, slots. and I was managing a lot of the database queries, promotions calculation, this kind of stuff. but I always had a passion for UI, UX. Coding. I developed some apps on Android as well, so I had this opportunity to move to Dublin, and joining their, CRO team.

Basically, it was like a small team. but it was definitely quite interesting for me. So I decided to take the leap and I moved to Dublin from, from Rome. I worked there for like one year, and then I moved to London in the Bedford office. And I kept doing CRO basically. and then I basically decided to move to ITEC and I've been there for like three years, managing their end to end process, basically for optimization.

And now I manage a team of five that are doing basically optimization the whole day. Okay.

Guido: [00:01:11] And so what did you learn from the, from the gambling industry? I think that's quite interesting. also socially, the social interaction there and, and the, how, the, how people interact

Andrea: [00:01:20] I definitely, that's a lot.

it was actually. A very good qexperience because the Italian business was set up as a startup. So I get basically to learn from insights team. I'm an engineer. I study engineering, so there was a lot of nice like insights based on like you can imagine how much data you can collect in like a bookmaker, all the spins, all the interaction with the games, all the promotions, marketing.

There was like multiple channels. There was like. Full of data and for a person like me that was like insane. I really loved it. but I always wanted to make changes to the product. You know, when you, when you're there, you see, Oh, actually we could do this differently. And I had no power to have an impact on that.

This is why then moving into a zero position, it was like a dream come true because then I could start influencing roadmaps or stuff to prove that actually we might have. I bet they use that experience and that will lead to better results or better conversion rates or more revenue. Basic-ally.

I

Guido: [00:02:21] also, always found it interesting to visit Magento conferences in Vegas and being in those kinds of areas where there's a little of gambling. it's always really interesting how they also pay a lot of attention to are the building is set up, that are really hard to find an exit, for example, are they arrange all the gambling machines

Andrea: [00:02:44] to maximize, and I think actually you're right there.

The biggest learning that was very important. The start of my career was to start thinking about the psychology of the customer. and you need to really think with, with the brain or see things with their eyes because sometimes you might want to, make a change that actually just looks better, but it's not functional for the way they interpret gambling.

And you have no idea how many tests. In the beginning I was actually failing just because of that. But then when you start tweaking your mindset, then you start actually becoming closer to the user. Is there, when you start seeing, like when the saw you start seeing, like shifting the needle in terms of metrics. Yeah.

Guido: [00:03:24] So I know at the iTech you have a team of five, so how does CRO at iTech look like?

Andrea: [00:03:30] Yes. So, basically I decided to move to iTech because I think he's a heavily CRO focused company. basically when I joined iTech, there was a team, a smaller team. the company was smaller because it's, it's a young company and it's still growing.

And we were working as, I would say, like a typical setup. I'm not gonna get lost too much in the jargon, but basically we had a team that was responsible for AB testing and every request or every test that. What's supposed to be run in the company, was our remit basically. and then as the company grows, as the products grow, then you start feeling the need for change or structural structural change.

and this is why we moved from, I would say, like a centralized center of excellence. just because we used to be the go to people in the business for every change in the product, or we used to act more like as a consultant sometimes, with product managers, sort of the people in the business just asking for feedback or they wanted to get an idea implemented.

And, we were basically just helping them. so we moved from this type of setup to, an agile type of setup. so we inspired ourselves a bit to the Spotify model, even though it's not a copy and paste, because that will definitely not work. and now we have experimentation chapters. Embedded in each of the squads that we have within the company.

So it's a big change and it's still, in ongoing, the idea is to get probably by the next year into an Abrey model, where are we going to have center of excellence? So people with high expertise in conversion rate optimization and the entire company, so each other shop, they're doing their experiments and we just work support them as consultants.

So, more or less the the idea. So at the moment, we, I would say halfway through.

Guido: [00:05:24] A lot of companies you say work with, a centralized,

Structure

team of experts,

they would go to for everything, surrounding the experiments. So why do most companies end up doing that?

Andrea: [00:05:39] You think. well, the centralized structure is very effective, especially when you are like small company. you basically have everything you need within the same team. Communication is great by default, because it's a small team and you always basically are next to each other the whole day, which is great from a zero point of view.

But then as soon as you start growing, you lose contact with product because then teams start to grow. So this shift up, I would say it's almost necessary most of the times from my experience, just because you need to be closer to the product, which means to be closer to the user, which means to be closer to the end goal, which is improving metrics.

so yeah, that's, that's, that's my take. There are pros and cons in everything. what I can tell you is that. You lose a bit of space of base, when, when you migrate to a decentralized structure because, an idea that we will build or design in, I dunno, say a week and it might take two or three weeks now because you start having sprints because of the agile framework, because you start having four or five different chapters involved in the process and every step you are into the, into the workplace, obviously slowing you down.

on the other end, quality usually goes up and that's a bit more structure in what you do. So you end up with, I would say less what, what happens when you have like, centralized structure. Sometimes it's that you start pushing stuff to the side. Then each time you use maybe different designers. So the design is not consistent or the coding standards or guidelines are different because he used several contractors.

when you're moving from a decentralized structure, it's way more consistent. So it's more future proof in that sense.

Guido: [00:07:20] Okay. So, but, but that, I guess that depends also on the, how you're, how you are decentralized. If you're a centralized model and you have that designer in house , in that small centralized team, then , that can be fine.

Andrea: [00:07:34] Yes, but it's true. But the only, the only, obviously it depends on the company where you are and the setup. but I guess if you have a product team and then you have a decent realized CRO team, inconsistency is, might still happen. Because if you, in a decentralized team, even if you have a design that there might be other two or three designers working.

In say an agile way and the product team. And there might be, you know, a lack of communication in that sense. So this might still lead to differences in other know, like the, the brand guidelines that they use or assets or in my actually do pick up the work as well. It's not effective in that sense or efficient.

Guido: [00:08:16] Yeah. But then if you decentralize it more and more people get involved, I would expect the communication to become harder. It is consistency to become a harder ,

Andrea: [00:08:24] it is. It is , I mean, consistency to me in my experience, obviously this all depends on, on the, on the company. I think consistency for us went better.

way better. I will say even like the QA in general is, is, is a bit easier. but the process itself gets slower because the communication gets slower because there's more people involved.

Guido: [00:08:43] Okay. And so if, if you go to a new company, a new company asks you guys to come in and work with them, do you always advise them to go for decentralized model or the Spotify model or does it also depend on the company?

Andrea: [00:08:58] The best model. It depends on the level of maturity for servo and these are all program. it depends on the type of expertise that you have in the business. Usually if you want to get low hanging fruits and you want to be as quick as you can to push changes, live and test stuff, probably the centralized model is the quickest way to achieve the result.

If you then start. Working a bit more structure way on, there's a lot of stakeholders involved, probably with the decentralized, type of structure. It might be helpful to achieve by the results.

Guido: [00:09:34] Okay. And so, and, and will tell us something about the type of company that does, you said the maturity of the company.

So how do you guys. Measure that's , that that maturity. How do you go in and sit and decide if you go for centralized or decentralized?

Andrea: [00:09:49] well, iThink media, even though it seams like an agency, it works with just on. client. So we basically manage the, end to end process for those products, which means that it's not like a typical agency type of work.

So we don't, we don't get to make this choice basically, if that makes sense. so imagine like you all, I don't know, like hundred sites and you optimize them, and to end. So in that case, it's not like a choice of, okay, which model do I apply in general? If I was working for a typical standard agency.

I will definitely, look at the structure, have a chat with their teams and try to understand which mother will do the best job. Just to give an understanding, say Paddy power. Betfair we used to work in a centralized way. but then it shifted normally towards a decentralized one. And this, just because when you have.

mature teams, it will just get national because there's too many other stakeholders have projects involved that you need to embed the experimentation within the squads within the team. Otherwise, this is going to be like clashing between the CRR activity on the products. Unless you are very good at keeping obviously communication and everybody aligned .

Guido: [00:11:02] so what are the biggest challenges that you see when going from that, a centralized to decentralized on the, on the client side, what, what are their biggest challenges on doing this?

Andrea: [00:11:11] well, first of all, you need to start, with the imaginization of  CRO, and you need to get everybody. In squads.

I'm basically happy and excited about experimentation, and this is sometimes a long process until you're work in a centralized way. Everybody's aware of what experimentation is, why you need to run experiments and how to do it, but then when you start doing this at the squad level or a tribe level or a company wide level, then you need people to understand experimentation.

And it's a process that takes time to get everybody. to the same level, a lot of very good level of experimentation. Knowledge.

Guido: [00:11:50] do you guys have like, exercises for people involved or, or how do you approach this.

Andrea: [00:11:57] First of all, we, we now try to involve everybody since day one. When there is a project that involves a B testing, being in squads, that's actually, a very, I would say, easier than when you work in a centralized model.

so we try to get everybody involved since, since the first steps. And this helps in getting their buy in or getting their, their ideas or the feedback on stuff. And we try to work as much as we can as a one single entity though. What's every project experimentation included? so by that, it's a continuous stream of activities that we try to put in place to make sure that everybody knows about experimentation.

And my take is that you need to, take a different approach depending on the chapter, say developers, if you have to talk to frontend developers about experimentation, you need to tailor. The communication to that you're talking to because they're all interested in different aspects of the experimentation activity.

So you need to to be more effective, I would say. You need to treat them as obviously separate entities and then get everybody up to speed with the experimentation program.

Guido: [00:13:08] Okay. So basically you need to do personalization, but internally for the people that you work with.

Andrea: [00:13:13] Exactly.

Guido: [00:13:14] Okay. So are there. Any things that you see that continuously need effort in this?

I mean you go and do this transition from a centralized, decentralized, hopefully you get everyone trained and used to working like that, but are there things that continuously needs efforts, to keep this rolling?

Andrea: [00:13:34] Yes. well, first of all, the sharing learnings is something that helps in keeping everybody hooked to the experimentation.

train, because when you see results of your job, you get purpose of, of what you do and why. AB testing is, is very important. also, being, I would say cutting edge in as much as you can, obviously in the activities that you do is important as well, because it excites people, and what they do.

So for instance, we. We are now migrating most of the products to server side testing. And this is something that for developers was very important. It improved a lot. Their workflow. it's way more interesting from a coding perspective gives them more flexibility. and this is something that helps same way the sinus.

Like when you start working on projects that are more interesting or they involve like the latest trends and stuff, you will obviously get them. More interested in the experimentation activity. It's, it's, that's, that's a lot of things that you need to keep up and put in place. But at the end of the day, like what we really want to share is I, everybody thinks experimentation and, and this is, this is all.

Guido: [00:14:49] Okay, nice. And, looking forward, what are the things you guys are working on for the next 12 months? What are you looking for to, to change or optimize? What are you working on right now?

Andrea: [00:15:00] so yeah, as I mentioned, we are migrating, most of the sites, at least the most important ones though, are server site testing, which is an interesting project.

and then I will say for

Guido: [00:15:10] us as an

Andrea: [00:15:12] interesting project because, you know, when you go service side, there's a lot of backend, involved and we have multiple. West side websites are multiple CDN. So everything, every site basically has a different setup. So there's a lot of people involved. It's, send interesting project.

There's a lot to learn from that. and then I will say the usage of data is something that we need to improve. because again, when you go to into a decentralized model, then you need more or less everybody to think in an experimentation way. So using the data, both qualitative and quantitative. The type of research that you have to come up with ideas and analyze test.

So this is something that I really hope that in the next 12 months will improve. So it will get us closer to the hybrid model that we want to achieve. And that's, I think that's pretty much the top two points that. And obviously more tests,

Guido: [00:16:09] but how do you do this? how do you unlock the data? What kind of tools do you, do you use that everyone in the company has insights in customer behavior.

Andrea: [00:16:18] Yes. So first of all, you need to make the tools accessible to everybody. And this is something that we already did so everybody can access the data.

But then you need to train people and you need to, try to explain to them what's the most effective way to use the data. Because if you are a data savvy person, you, you might even want to spend one hour in front of your, of your laptop or more to just drill, drilling down to the data and trying to fund what you, what you're looking for by, if you are.

A chapter that is a bit more, let's say, less data savvy or you're not that interested into digging into the data, then you need to provide them with the right infrastructure to get what they want in a quick way. And this is why building the right dashboards, so telling them exactly what to find and where to find it.

It's very, it's very important because they will learn this way and they will get more hungry for data. the, the more they use it basically.

Guido: [00:17:14] Yeah. Are there any tools you use for this?

Andrea: [00:17:17] Yes. Yeah, we, we, we use session, come for visitor recordings. We use Adobe analytics, for the analytics. But then we got a lot of like other data that comes from our database and the rest of it.

So we are trying to join basically all the data sources and to manipulate the data so that it will be easily accessible for

Guido: [00:17:35] everybody. thanks so much for sharing all of this with us, Andrea. so my final question for you, do you have any, any tips on books that you'd like to tip to our audience

Andrea: [00:17:44] so there's a couple of books that I, one, actually it's almost done, I've read it, but that's a new one that I got recently is called experimentation works from Stefan Thomke.

Guido: [00:17:55] Let me see Experimentation Works. This one.

Andrea: [00:18:00] That's it.

Guido: [00:18:02] There we go. Fanboys. Yes.

Andrea: [00:18:03] And then there's another one that I almost read it all, which is from Georgi Georgiev, it's called statistical methods for online AB testing.

And it's quite  interesting because I. Like when I started, there was not much about the online, about the, the way you can analyze a B test. So I got really interested in that and doing a lot of research and stuff. But then this book came out and it's a very good book to start learning more about the statistics behind the AB testing.

Guido: [00:18:34] So it probably has a chapter of Frequentist versus Bayesian.

Andrea: [00:18:38] yes, that's a lot of stuff. Very interesting.

Guido: [00:18:44] Cool. Well, thanks so much for sharing. We will, we'll definitely add those books in the show notes. So anyone can find them. thanks so much.  Good luck with, well you're doing and helping your clients with, the, the tradition and staying, staying the course.

talk to you soon. Bye. Bye.

‍