Data Science Hangout | Matthias Mueller, Campaign Monitor | Understanding Customer Actions

All right, welcome everybody. So Rachel is taking a well-deserved break from work this week, and I am stepping in her place to host the Data Science Hangout. If you are a repeat attendee, welcome back. If this is your first Data Science Hangout, welcome. So glad that you're here.

This session is really meant to be for you. So I'll ask a few canned questions to start, but we want this to be audience-led, so please don't be shy and ask questions. You can participate in the Hangout through a multiple of ways. You can have your video and audio on and jump in. You can raise your hand. You can post a question in the Zoom chat, and we'll read it out loud.

With that said, just in the interest of time, I figure I'll hand it over to you, Matthias. Maybe let everyone know who you are and what you're doing and why you're here.

Yeah, absolutely. Well, thank you for hosting as the interim host. I appreciate it. A little bit about me, my name is Matthias. I head the marketing analytics function at a company called CM Group. We're a marketing technology company. We're at this point, I think, the largest family of products and brands available in the email and personalization space.

We're actually closing on another acquisition, I think, yes, as of Tuesday, folding Cheetah Digital into our fold as well. We have roughly 70,000 customers across the globe and essentially utilize or we facilitate several billions of customer connections every year.

My team specifically sits on the marketing analytics side. So there's groups that reported to me specifically tasked with identifying opportunities of how we can increase the revenue streams or reduce costs of acquisition. It's basically split into two groups. So there is an insights and reporting function that deals with a lot of the questions of like, you know, what did we do and why did we do it and how is it performing and those types of questions. And then there's a more advanced analytics practice where we build anything from lifetime value, lifetime value calculation models to multi-touch attribution, marketing mixed models and those kinds of things, all with the purpose of supporting our roughly 100 percent marketing team in order to make good decisions.

Marketing analytics overview

Very good. I mean, maybe for those who are not familiar with marketing analytics, give a little more insight with that or some of the data products that you can share that you're producing and the stakeholders.

Yeah, absolutely. Great question. So I kind of touched on this a little bit. So ultimately, the idea of marketing analytics is, can we try to understand how our prospects are acting on different properties? So whether that's a website, whether that's interacting with ad properties that we have, you know, that could be anything from, you know, simple Google paid search ads that could be OTT properties. So out of, you know, wherever we're buying ads, we're trying to figure out where do we get the best marginal return on our ad spend?

So with several brands and products that we obviously oversee, it always becomes a bit of a prioritization of like, you know, where is the best investment? Where do we feel like we can put the next dollar in order to get the best return in order to get more customers for less cost of acquisition?

Privacy, regulation, and the future of marketing

Yeah, so again, so I presume GDPR affects America with UK legislation, how global everything is. So there's more going on to that. So recently, of course, we saw we've seen the whole Apple versus Facebook, Apple with their privacy changes, making apps have to explicitly ask for to track your data and everything. And of course, and everyone's anti-Facebook and all that data tracking. How do you see that changing the future of marketing?

Because, of course, the expectation is that legislation will be in place globally, the US will likely have one, the EU probably have another one that's probably more strict with everything that they do. And do you think that's going to impact how people not only go to market, but how you focus on different aspects of marketing?

Yeah, fantastic question. And you're actually spot on. So for those of you that aren't in marketing or have data protection laws to deal with, GDPR is a law that was written in 2018 in Europe, essentially governing who can access your data, who can process it, where it's processed and a lot of other things, really putting the ownership of data back into where it honestly truly should belong, right? Like the data should belong to each individual and giving a bit more options to how you can allow companies or disallow companies to utilize your data.

There's also US-based regulations. So there's the CCPA, so the California Agreement, essentially, which isn't as stringent as the GDPR, but it's kind of going into that direction. And I would certainly expect that there's more regulation coming.

But to actually get to your actual question, so I feel like marketing companies or companies as a whole shouldn't necessarily try to figure out how to skirt those different things. But there's a lot of things that can be done to actually test your marketing initiatives without being sort of on the creepy marketing side.

There is probabilistic marketing attribution, for example. A lot of the things that we've been doing internally is relying a lot more on testing and experimentation. So, you know, really interestingly, more traditional initiatives like building marketing mix models, for example, is probably going to become a little bit more invoking and because, you know, you aren't going to have that definitive click attribution that you are used to.

But yeah, ultimately, from a company perspective, it's about offering the right thing to someone. And I think sometimes, especially on marketing, we've gone to the point where everything and I say this as a data person, everything is so measured that people have forgotten that really, truly marketing should be about just offering someone or offering a prospect the best possible customer experience. And that's really what it's about.

But yeah, ultimately, from a company perspective, it's about offering the right thing to someone. And I think sometimes, especially on marketing, we've gone to the point where everything and I say this as a data person, everything is so measured that people have forgotten that really, truly marketing should be about just offering someone or offering a prospect the best possible customer experience.

Yeah, maybe adding to that, what do you think the future of marketing analytics looks like?

Yeah, so I think we're going to be going into a I mean, everyone kind of in marketing talks about the sort of no cookie future. Right. I certainly think it's going to move in that direction where you're going to get less and less personally identifiable information on people. From a marketing analytics perspective, I mentioned this a bit. I think it's going to come down to being more in the like testing and experimentation space. I think that that would be the one to still prove that value exists in marketing campaigns. And then, you know, there's different models, I think that would still apply on the probabilistic side to understand how marketing initiatives are going to be used.

Anomaly detection and surfacing insights

There's an anonymous question about the size of the data that you are analyzing or the type and the real question is, are you dealing with big amounts of data, digital touch points? Question mark. Yeah, so the quick answer is absolutely. We have a treasure trove of data. In fact, sometimes it feels a little bit like there's too much information and it kind of becomes more of a question of what is actually interesting rather than, you know, can we look at everything?

So each of those different brands that we own, so, you know, CM Group, the company I work for, owns the namesake company Campaign Monitor, but also Sailthrough, Seligent, Emma, Cheetah and a few more. There is different levels of sophistication that we have in terms of data models for all of these different products.

But to generalize, I think on the marketing side, we are probably the most measured function in our company. We tend to track and over track everything we can track. And it's become the idea is really how can we find like those needles in the haystack to actually try to tell us what we should be looking into?

And one of the and I mentioned that my team is kind of split into two groups. There's like a insights and reporting team and then there's advanced analytics team. One function inside of the advanced analytics team is like an automation practice where essentially we write a lot of code that deals with anomaly detection. And the idea behind that is if we have, I find a really bad metaphor, say like our data is the size of New York City, right? There is no way for us to actually look into that and kind of investigate the whole New York City. So through the automation anomaly detections that we write, we try to eliminate sort of like most of New York City and just tell us, OK, these are the specific houses that you should look into that have something unique going on.

Hi, Matias. I watched your lead anomaly detection talk and I implemented something on my own Slack bot. So first of all, thank you for that. I love it. Yeah, I'm just wondering how the insight finding leads to business decisions. So I'm kind of trying to balance out the granularity of the information that I share and I'm wondering how that conversation goes within your team.

OK, first of all, I love that. I love hearing everything about this. And we could probably talk another couple of hours about, you know, where I feel like business intelligence has a place in the business and where it does not. And what we found, again, we have this specific use case that my team is sitting on the marketing team. We're integrated into marketing and we have a lot of people that are the end users of or the stakeholders of our analyses that do not have like a quantitative background, right? Like you might be able to or you might have to do an analysis that has a regression in it and then explain it to a team of copywriters. That is a very unique thing that happens here all the time.

And what we found and we have a, I would say, decent stack. So we have a Looker instance sitting on top of a Redshift instance that flows almost all of our data from a reporting perspective flows into that. But what we found is that building more and more and more business intelligence didn't really increase the knowledge or data data aptitude for our for the teams, because there is a hesitancy of utilizing those BI tools.

So instead, this is what what you're referencing there. So instead, we've actually kind of flipped it around. So we wrote this Slack bot that basically takes information, you know, in this case, like, you know, some anomaly with detection and then surfaces that information right to when it happens to people on Slack, just pinging them like, hey, something's going on in your area that you should be looking into. So with that process, we've been able to really kind of change of how people digest analytics in that, you know, it doesn't it doesn't it's not like an active, oh, I'm going to go and look at data, but it's basically you're being furnished with data in the moment that it's interesting to you.

It definitely is like a definitive or distinct sort of area that you have to walk the fine line between not, you know, shooting people and spamming people with information. But what we found based on like the iterations that we went through with this thing. Is that as long as you're very precise with what someone wants and like understand what someone's exact requirements are for specific pieces of information. So, for example, our director of performance marketing, she wants to know every time when our cost per click falls below a certain threshold based off of where we would expect it to be or if any region performs out of the ordinary in terms of impressions or click volume. So when that happens, that's when she gets like a Slack notification that we we wrote, you know, hooking into a data warehouse and shooting it off with R and Slack.

What we found is that the success of this ultimately comes down to how precise you are in understanding what someone actually wants to know and then giving that person that information. What we found less success at was to hook into like generic channels and just kind of having more broad insights pushed into these channels, because as soon as you do that and you do that on a recurring cadence, if it's not pertinent to someone, then they will stop reading it after a little while.

What we found is that the success of this ultimately comes down to how precise you are in understanding what someone actually wants to know and then giving that person that information.

Understanding stakeholder requirements

How do you go about getting those initial requirements or getting feedback from people that this is what I want? This is kind of what I want, but it's not really what I want. How do you encourage people to be honest?

Yeah. Oh, great. I think it's more, first of all, I think of data products as living and breathing things. I think data products are not something that you publish and then let go. They're kind of like you're forever tied to it, especially if it's something that pushes information to people. So you're going to have to be open to feedback from people.

Ultimately, we are, as analytics, we are a shared service that is a service provider in some shape to the greater marketing organization. So our success is measured by how well people adapt the data that we give them. Because if we didn't, then we can have all this information if they don't actually use it to change their behavior, if they feel like they need to change the behavior based on a data point that they see, then why even give someone the information?

I think the big thing that I found here is that it helps to try to kind of try to get to like the core of what someone's actually trying to answer rather than getting into like a solutioning process too early. So sometimes we have people that will come to us and say, like, hey, I would like to see I would like to get something sent to me that tells me how I'm tracking against the end of the month in terms of pipeline, just making this up.

And what we've actually found over time is that doing that without actually understanding what their true question is, because they wouldn't come to you if they didn't have a question that they're trying to answer. Right. In this case, maybe they're trying to find out, you know, maybe, you know, in this case, you know, end of month pipeline could be something that they're trying to see how they're tracking against the target.

So understanding more of like why they're actually interested in this kind of information and then trying to see whether that is truly the best way of giving that to them, like this specific already defined report. I think a lot of it happens there where you really speak through, you know, let's let's put away what you have in mind for a second and let's understand what you're actually trying to answer. Like, what's your question? And then can we try to come up with a process that would give you, you know, or get you very close to getting all the information that you need in order to make a decision on, you know, whether you can answer it or not?

Serving the needs of end users

Because really, I enjoyed your answer, especially to the anomaly detection and delivering the results with the Slack bot. I didn't think it was off topic. I think it was really spot on because it really struck me that it's kind of ironic because you're working in the marketing department and actually there was a marketing problem. You're trying to sell something to the people and you want to see that they're buying it basically. And I mean, the people that you're talking to would tell you the exact same advice that you're giving me too much information and it's not the right information and I'm going to ignore it.

And I feel that oftentimes we do this as data scientists. And I wonder if you agree. As scientists in general, I think we tend to service our needs because I spend a lot of time making a really fantastic pipeline or algorithm or look at this, how beautiful this model is. And they don't care. They just want to know the answer. Right. Like I don't I don't need to know why Google put this email in the spam folder. I just want to know that the junk is getting into spam folder, although they have some wonderful algorithms and back when it does that. Do you think that we oftentimes service our needs instead of the needs of our end users?

I completely agree with that. I mean, I think it's sort of a self-made problem on the data science end too, right? Like, I mean, if you're if you're trying to digest any kind of content that's written in the data science space, it's always content about like the newest new shiny thing that came out. Well, not sure that misnomer here, like not shiny, but like the new sort of, you know, the newest thing that came out, you know, a while back, everyone's talking about like GPT-3, like all of those things are like things that I think we've done a little bit to ourselves where we want to do like the cool stuff.

But, you know, data science or like let's say like machine learning might be a good example in that even where, you know, I think a lot of companies go out there and they now understand like, hey, we should probably hire data scientists because that's what people do now without, you know, ever paying attention to like, well, maybe there should be like a data engineering function first to actually build the house before we can hire like an interior decorator that would be the data scientist, you know, coming in and making everything pretty, like I actually have to build the walls of the house first.

I think it's sort of a self-made problem on the data science end too, right? I certainly do it. I try to be mindful of it.

So one thing that strikes me is that I feel that oftentimes they don't see how much work went into doing something when you do show them that result. And I feel that we constantly want to kind of up our game and show how much we're worth and that, OK, we're really adding something to the bottom line and we're really proving our worth because I think there's still some like, all right, like was it worth having this team and hiring these guys? So do you feel that there's ways to really show the value of your work without just I think we make it sound more complicated than it is to do that? Are there better ways than just overcomplicating our analyses and our presentations?

Yeah, I don't want to take over your talk, but like I think about this all the time, like like I've been part of my life forever and and I love that this has come up straight up because this is this is make or break for data science like it really is. We will only get so high until we solve this problem. Right. So it's it's it's a great topic.

I think part of the piece that is part like that causes this problem are kind of two elements. One, if we're talking about the two different parties, the data science party and then the business party, we see different things as cool. Right. Like we just do like different. We have different values of what is cool. And you talk spot on about like what we think is cool, the shiny objects, the how that's not cool to them. If you use sales as an example, what's cool for them is making their commission. Like that's what's cool and that's what speaks volume. Right. So understanding the language to talk about is part of it.

But how do you get there is what we're talking about. And it's probably one of the most under discussed, in my opinion, boring. More challenging and not really start up a bowl so you don't get a lot of the financial attention is data literacy. Right. That's just straight up literacy. And I'm not talking about like, OK, how what's the syntax of writing something and how to build? I mean, what can I do for you just at that core level? What can data and analytics do for you other than increase your shareholder value for VCs? Like, like, really, what can it do?

Was the first thing and this is why I actually like Robert's question so much. The first thing, truly, it comes down to understanding what the requirements are on a core level more than, you know, this is the data I'm looking for. But like, what are you actually trying to answer and how does it impact? You know, like you said, for a salesperson, ultimately, you know, how can this data product that you're building for me help me actually make more money? Right. Like that's what's what it's after.

But then, Rick, the second part is that I think one thing that we sometimes do on the data science and is sort of disappear in the process. Right. You you do something. You have you have your eyes on like, oh, this I know now how to do this. I know now what I want to build. And then you data science does take a long time. And I think it's really, really important, like Ryan said, to take someone on the journey with you and kind of like let them know what is happening throughout.

We build things internally. It takes several months of building like a marketing mix model, for example, is something that is probably like a six to eight months project. If you don't take someone on that journey with you, they think you disappeared for six months. Right. So it becomes incredibly important to to speak through and have a very clear understanding of like, look, this is like all the things that have to happen. These are the things that we're going to need your input for and kind of like make someone feel that they have ownership over that project. Even though you're the one or your team is the one writing code. You're producing that project or data product for someone else, not yourself. And I think that's what this comes down to.

Intuitive data products and dashboards

One of the things that I feel like I did wrong in my last job was make the end deliverable. I mean, in my case, it was a shiny app, but is make it way too like geared for someone that has analytics training. I mean, I had like a FAQ page that explained a lot of the different visuals or numbers that you're seeing. I thought it was great because, you know, my perspective was, hey, like I have all the instructions here, like someone that is a non-technical you know, salesperson or account executive. They can simply look at this FAQ and understand what's happening. But it became clear that like the data I was trying to communicate was not intuitive.

And so through a few iterations of like that app, as well as a few new apps, it like the way I tried to start building them was to have no instructions. And that really kind of helped communicate all of our data points or visuals a lot better. Like someone who's non-technical should be able to go into this dashboard and kind of intuitively understand what it is that you're trying to analyze or, you know, the actionable items. You shouldn't need to give them training over training or a long FAQ in order for them to like come out with actionable insights. So anyway, just wanted to chime in with that.

I love that. You know, I mean, what's ultimately our purpose being in analytics, right? It's it's not to produce like dashboards and give people data, but it's to make people understand what the data means. And I think there is a very definitive difference between the two, you know, which is also one of the drives why we actually went away from just churning out dashboard after dashboard, because you don't actually know how someone receives that information if they're on their own.

So, you know, I think over the like the last few months, years, sort of like the concept of data translation has been like a little bit more sort of up and front center. I don't know if anyone has like heard of that, but like, you know, Google is now hiring like data translators, which, you know, feels like that's kind of what we should be doing, I guess.

And I think you're on the right path where, you know, if you're building a data product, I think the idea is that you have to build it in a way that it's intuitive enough for someone to use it, because if it's not, then it can become it can start feeling like work. And you don't I don't think that's the purpose of us, right? We're as as people in analytics or data science. It's our job to, you know, give people more information and help them make decisions based off of information that we can give them, rather than kind of like adding another thing onto like that plate that they have to check every day.

Outlier detection and when outliers matter

Yeah, so for me, like I find it quite interesting looking at outliers becoming patterns, especially as you get more and more data. My general idea at the moment is if it happens once and it's an outlier, if it happens twice and I research a bit more and it happens a third time, then I go, actually, this isn't an outlier. This is becoming an emerging pattern. I was wondering how you do that with the size of data that you have. I'm assuming it's all automated.

It is incredibly interesting to talk about this question, because if you had asked me, you know, I think I've been with CM now for like roughly three and a half, four years or so. If you'd asked me before my time here, I would have given you a completely different answer, because I think from a data perspective, the first thing that you generally do, you know, like the first thing that they teach you is, well, just, you know, the outliers probably should be normalized out. Or, you know, like if you build a model like, you know, like you kind of have to make sure that it's not affected by outliers.

I've actually completely like changed my thinking on that working here, because some of the products that we have are like the outliers are the things that matter. Like the specifically for a brand like Sailthru, for example, which is sort of on our enterprise and, you know, the marketing there is on the ABM side. So what does this mean? It's not a volume business, but it's, you know, very much, we essentially already kind of know the opportunities, the accounts that we'd like to target, and we target that way. And, you know, it really makes a big difference if a singular account like closes or don't, or doesn't.

And, you know, from that perspective, I've changed my stance on outliers, because it feels like for some brands, those are the things that are actually interesting. And when it comes to outlier detection, that is something that you have to keep in mind that is it for something where it can be like a something that you're actually looking for? Or is it something that you're trying to figure out how do I build a good model so, you know, it still is statistically valid.

There is a lot of startups now that deal with this type of issue as well. Like one big one is like outlier AI. I don't know if anyone's heard of them. But they do similar processes where you essentially just push all the data that you have to them and they build models for you on the outlier detection. I felt like I want to be the owner of that. So we do, we write all this internally, because of this reason that I don't want someone else to make the decision of this specific outlier. Is this just, is it a true outlier that, you know, a couple years ago, I'd be like, okay, let me just ignore this. Or is this something that we actually should look into further, because there's a business opportunity here. And in many cases, we found that that is actually what's happening.

Democratizing data and building data literacy

Santiago has a great question that he put into the chat. So outside of a specific project, and educating folks on that, have you done anything successfully or failed even to try to educate the broader marketing team? For example, have considered Lunch and Learns, or like a newsletter or something?

Oh, goodness. I think it's, if you figure out how to do it, it's the Holy Grail. In a sense, it is a bit about setting expectations, right? Like, what does democratizing data actually mean? And if you set out for, you know, like I said, we're, you know, a bit of 100 people in marketing, setting out with the expectation that everyone's going to be like a closet data analyst, that's not going to happen. However, what you can do is make people understand the power that data has. And the way that you do that is to give them the understanding, or have them understand how well it can help them make decisions.

I think in this process, one of the things that I try to emphasize is that I'm not the decision maker on things, nor is my team. So when we do analytics, and we provide like a data product to someone, it is still that someone's decision to make in the end of what they want to do. I don't take that power away from my team or from the marketing organization. And I'm very protective over the fact that I feel like data should be a neutral party to someone, like a completely, essentially just giving someone, here's everything that we feel like from a data perspective, you need in order to make a decision.

But then the way that you empower someone, and actually have them understand what the value of information is, is if you still let them come to their own conclusion about how this decision should be made. I think some organizations have the approach that, you know, there's like a, you know, maybe a business, a biz ops team that does all the analysis. And they, you know, push essentially their thoughts into like an organization. And then that's the thing that everyone kind of has to follow. And I think that's a really, really quick way of having people sort of not participate in this idea of, hey, data is actually important. Because you can give people data, but the only way if that people will actually start using it and that you truly democratize it, has to come from that person. It's not something that you can force someone to do.

So that, I guess, is my longer answer to this. We've certainly done a lot. In fact, I, next week, I'm teaching the entire marketing organization, Looker, yet again. So I'm still doing sort of lunch and darns. Absolutely, I don't think we're all going to get around that. But yeah, the idea is people are going to, to me, people are going to adopt it if you make them understand that what they can do with it can actually make their life easier. And it can help them sort of put more behind their own job, rather than, you know, kind of mandating, hey, this is the data that you need to use.

Balancing new technology with business value

His question was, how do you balance steering clear of flashy new things when keeping your skills current? I don't know if I can.

So it definitely is a balance. I think that's where it comes down to because I think technology is moving so fast that if you don't look for those shiny things and at least know about them and kind of have an understanding of what they do, then it can be rough, right? Because who knows what's going to be around in five years.

But technology is just moving super quickly. And if you don't have an aptitude to kind of like follow things and kind of be open to learning new things, then I'm not sure if analytics is going to be a great fit for you because there's just so many things you're going to have to keep up with.

On the other hand, and I forget who said it, but ultimately, what we talked about earlier, you have to keep in mind that what you're making is only as good as, you know, people understanding and people utilizing it. So sometimes a more simple model that's more easily read and more adoptable outperforms the shiny thing.

So I think in that sense, it's really important to think about the business requirements. So you cannot sort of let yourself make that mistake of, okay, I just really want to do natural language processing because I feel like it would be really cool if it has no application to what you're doing. So you can't let your sort of personality try to dictate what the business needs.

Sometimes a more simple model that's more easily read and more adoptable outperforms the shiny thing.

To wrap things up, I thought maybe one, how can people find you? And then two, any sort of imparting wisdom to people who are interested in getting into data science leadership, moving up in their career.

Yeah, I'm a huge fan of these talks. It's one of the recurring meetings that I don't dread. You have these days with everyone working from home, it seems like there's just meetings and meetings and meetings. And in many cases, dedicating like a whole hour every week is something that can be challenging. But every time that I've logged into one of these, I've just found incredible value. So thank you.

How can you find me? I'm on LinkedIn. That's probably the easiest way. I don't like, I'm not very selective. So if you send me an invitation, I'm sure I'm going to invite or I'm sure I'm going to accept it. Other than that, yeah, I guess email me or find me through a pigeon. But I'm happy to connect with anyone that has additional questions about this or just wants to chat, data science.

And the last question that you had was about like how to actually get into data science and data science leadership. I don't know if I feel like a bit pretentious talking about this and answering this because I don't, you know, I certainly still feel like I have a lot to learn to move into that realm also. But in a sense, I think to me, it's understanding how you can provide business value to someone and really kind of trying to be or trying to use your skill sets in order to make someone else's life easier. And once you show that value, there's natural buy-in that generally people will help you along the way.

Thanks everyone for participating. As always, next week, same time, same place, forever and ever until we run out of data scientists.