Hey, everyone. Thank you for joining us. Yeah, joining us on a new platform, so appreciate that. Hopefully, it's, a little bit slicker than Google Meet. Got a nice chat. Feel free to to say hello in the chat, where you're calling in from. I thought that's always a fun thing to do as as we wait for people to kind of join in. So feel free to say hello. I'm Taylor. If you missed the last session, I am, head of data and product at Kount. Using account a lot today, so get get a bit familiar with it. But today, we are gonna be building a MetroTree live. So I could start sharing my screen. Hopefully, everybody can see that. Just do this. Okay. So I'm just giving people a few minutes to to join in. Okay. Great. So, yes, like I said, today, we're gonna be building a metric tree live. We're joined by Sara Savzakari, who is an analyst at Lockbox. She's somebody also who, you probably have seen on LinkedIn. She's building some some great stuff in count. She's a great blog as well. I really recommend, just a good person to follow and very skilled analyst, as we'll come to come to see in action today. And by Matthew Brandt, he's a decision engineer, and Andrews. And, yeah, if you guys wanna know anything about decision science, decision engineering, Matthew is, great about that. Honestly, it's probably worth a whole webinar series on its own, how interesting that field is. He's also a, kind of analytics livestream expert, so he'll be bringing the, professionalism to this, today. No pressure. And, yeah, it's just me the three of us. We're gonna be working this through live. So, stakes are high. There might be some errors. There might be some mistakes. But, hopefully, it gives a good overview of of how to do this. So, particularly, we kinda want to cover a few common questions that, we can we can get you asked. So things like, how do you decide what your metric tree should look like? When you're just starting out, how do you decide what you should and shouldn't include? When does it stop? How should we use it, how should you work with stakeholders to build it out. Obviously, we kinda prefer an iterative approach. So what does that look like? And, yeah, just kind of getting a feel for what this actually looks like to to work. As we go, please feel free to drop any questions in in the chat here, and I'll be keeping an eye on it. We also have Lewis on board as well who'll be helping out with some questions and things like that. So, yeah, don't be afraid to to chat if you've got questions at any point, and I'll pause and we can cover them as we go. Cool. Before we kind of dive into the situation, because we're using count so much in this session, I wanted to kind of give a quick overview so you're familiar with what was going on so we can kinda focus on on the actual building process. And I should say, a lot of the stuff that we're covering applies to to other tools. So it's not like all this is just, stuff that you have to do in count. These are principles and concepts that you can do anywhere. But just so that this all makes a bit more sense, I want I want to cover what this what count is. So like I said before, it's kind of like a a Miro, but you have a SQL IDE, a BI tool, and a notebook combined in them. So you could see we're in an infinite canvas here, so it looks kinda like a Miro board. But we have data over here on the side. So I can pull in some some data here, and these are SQL cells that are running like this. Off of this, we can create visuals. I do, whatever we want here. Just build something sensible. This, we can connect these cells. We can add Python cells as well in here, and you can connect these cells and build out DAGs as you would in a notebook, for example. And then you can, yeah, you can kind of combine this to make different visualizations. You can add control cells to parameterize your data. And with that, you have all of these other types of objects that you'd have in in in a whiteboard, for example. Now this combination means, yeah, you can kind of combine this really flexible way of working that you would in a whiteboard, but with all the power and things like that that you have in the data tools that you're used to using. So it'll become more clear kind of how this works as we go through this example, but I just wanted to give a high level overview of of what we're working with here in Cal. Cool. So, this scenario, I thought it was helpful to kinda give us some context for this metric tree that we're building instead of just diving in and building something kinda random. So in this in this sense, so Soren has been asked, to look into DAU, daily active users. And we should contextualize this a bit more and say, we are a kind of an elearning company, maybe like Duolingo, for example. Maybe maybe exactly Duolingo. And we're looking at, we've noticed our daily active users, metric is dropping, and we don't really know where to look. So there's been constant disagreement on the leadership team about where we should focus our effort in terms of acquisition, retention, reactivation, etcetera. So we've been asked to break this down and figure out what is causing this drop in DAU, which is kind of one of the first requirements, and therefore, where should we focus our efforts? Now the goal for this, I think it's important to to set this stage when you're doing this yourself so you just don't end up kind of deciding to do everything. Set yourself an initial point that you wanna stop to instead of saying like, oh, we'll just keep going and going and going. So we're just trying to get to an MVP or minimum viable product of our tree where we get a common understanding of the metrics we're including, the definitions, some summary metrics, and also some row level details so that we can kind of validate and trust the numbers that we come up with. And who do we have on the team? Let's say we've got some business stakeholders, CEO, head of product, head of marketing, and data team members. We have an analyst and data engineer helping out. Now, Sarah, Matthew, and I are gonna play the role of of the data team in general today. So, we're all gonna be kind of floating around doing a bit of everything, but we're gonna gonna be doing what the data team is up to. And then, also, I think it's helpful to kind of build a metric tree completely from from end start to finish. It can be quite a big project. These are kinda some some steps that I see commonly done. This can vary depending on how your team likes to operate, what type of team that you have. You know, data engineering is separate from data analytics, as well as the scope of the metric tree that you're working on. But I think this is a good kind of starting point for mapping out how you wanna do this work. So let's say, you know, we've already planned and scoped our projects. We've drafted the metric definitions, which I'll we'll go through in a moment. And we're on this stage here, which is just adding in some data to get a first draft of these numbers and figuring out, how our initial metric definitions kinda map out. After this, we could look into automation and scaling, rolling this out to the wider business, expanding it beyond, you know, our kind of limited starting point that we had to start with. Okay. Cool. Any questions so far? Keep an eye on the questions. Not yet. If you do, please please ask. Okay. Cool. So like I said before this, we had looked at starting to define our tree. So we've just done this with sticky notes. I I advise doing this pretty lightweight way to start with because you wanna get approval of these high level definitions and structure before you start writing any code, or else you're not gonna be building, say, productionalizing the wrong things. So before you start writing any code, please make sure you have something like this mapped out. So we have our DAU daily active users North Star Metric and this kind of breaks down into these four different user types: we have reactivated users who are learners who are active today, who are also active in the last month but not the last week, We have new users, so these are people who've just joined. We have resurrected users who were last active more than thirty days ago but have come back, and we have current users who were active today and were active in the last week. Resurrected users breaks down more into reactivation rate which is a percent of users active in the last month and the at risk monthly active users. Similarly resurrected users breaks down into resurrection rate and this idea of dormant users who are learners who've been inactive for at least thirty days. So we've got the definitions at this point. We've already got these kind of approved by the, the stakeholders. They agree this is a good place to start. So I've done enough talking now. I will quickly go through the data that we have available, and then we can we can start diving in, to actually building this live. So, we've just got a couple of tables here. So we've got sign ups. We've got daily activity. We have these activity states, which helps us work out, yeah, on a certain day, what is each person's status. And then that's all been aggregated up into this nice kind of summary table that we have here, which we're gonna be using for a lot of this, which has for each day, just a total aggregate of all these numbers. Now I will say in, in kind of the the real life, you'd have a lot of SQL that went into this. We really wanted to focus our time on not being a SQL workshop and being a Metrature workshop, so we've kind of jumped ahead and built this summary table for you. Like, you would have to spend this time kind of getting the data as you want. But there'll be some analytics that we come across as well, but we're gonna focus our time on on actually building this tree out. Okay. So I think from here, Sara, I think so you wanna kind of take over and and as the kind of head analyst, how would you how would you go from here? We've got some definitions here. We've got some data. And and, yeah, what would you do? It would be a good question. Actually do it. Yeah. I think well, the first thing, obviously, getting the definitions done is a big load off the shoulders, And that takes a lot of work, and I completely acknowledge that as you're building out the tree, you know, you'll probably have to refine and, like you said, Taylor, iterate over the the process. So maybe you'll go back to the definitions and say, we're not identifying the correct leaders here for this North Star metric. So once the definitions are done, the next thing I would do is think about what do we actually need in our metric tree. What I would do is just kind of prototype it a little bit. So, you know, as you would do with the sticky notes, let's say let's make a tile. So let's get a frame together. I would say, okay, we obviously need metric title. So let's say at risk monthly active users. So I would go ahead and I would start formatting my, proposed tile for my metric. So we're looking at one metric at a time here. And then I would probably briefly have a description as well. Hopefully, this isn't my watching paint dry. And now I'm really conscious of what my typing speed is. You get to see it. And then what we would do is I would think, yeah, we want we want to know where we are as a as a business. So I would put in a sticky note, and this would probably be the aggregate current period. So I would do this, and we also probably need to see a trend line. We need to know what you know, how are things going. So I would probably include a visual at the bottom showing us what, the trend has been for this particular metric. And then I would if I wanted to spice things up a little bit, I would put percentage difference or percentage comparison to the previous period. So this is this is a prototype that I would probably take to, you know, the the stakeholders that we spoke about, so the CEO, head of product, head of marketing. And if this is descriptive enough of the key metrics and definitions that we pulled out, then I would go ahead to building, which we can do now. Nice. Yeah. So When you're starting to dig into the to the data itself, yeah, how would you how would you go from there? Yeah. So I would let's take a copy of this. Bring it down. I would pick a starting point, and normally, the way I like to do it is I would start from the bottom. I'd start from, I guess, the most granular point or the the last node of the metric tree or leaf, I guess, you can call it a leaf leaf node. Or you can obviously start at the top. I just prefer to work my way from the bottom to the top, and then I I'm able to see quite intuitively if if the numbers are marrying up and if there's a data, you know, if we need to think about the data model, if there's a data quality issue. So, yeah, I have this I have this plan now. I have my template, and I will start off with putting a visual together for my aggregate. So we want to look at let's start off with that risk MUA, MAU even. And, yeah, let's get a scorecard. Let's get an aggregate of how many out at risk monthly active users do we have. I'm mindful that you can't see me pull the, fields together. I can, I can build this one? Awesome. So Yep. So and then we would add a date filter as well. So let's say we're looking at, users from thirty days ago. So let's say the let's say tenth of June. So on or on or after the tenth of June. Or you could do a custom date. So you look at thirty days ago as Taylor's done there. Awesome. So this is daily active users. I'm gonna quickly change it to at risk monthly active users. So this is our bottom node. We're working off of this here. And so this is an aggregate. I'm gonna pull the template that I created, copy it, and then what I will do is just format it slightly so that it fits what I like. And once I've built this, then it means I can simply just copy this template and replace the fields really quickly with all the other metrics that I'm interested in. So here we go. We've got an aggregate of the current period. And let's say, we can put a title current. Now let's put our trend line together. So you take the same, essentially, the same, visual. We're going to change it into a line plot. There we go. Put in the date, and I want to see what the at risk monthly active user trend has been from the beginning of our data collection. So let me pull up the date filter. We can see there's a drop at the end here. That's because we haven't got a full week of data and I'm aggregating by week, so we can filter that out. So we're looking at what's the last day? It's the thirtieth of June, and so we wanna include all data from thirtieth June before, before that point. So on or before thirtieth of June. Oh, there we go. Now we have I'm sure Taylor and and and Matthew can make this look really pretty, as I build it out. And, so once we have, like, a final style that we've we've we're all happy with and we've decided on, then it's just a case of copying the tile and replacing the fields with the metrics that we're interested in. So I'm just gonna quickly update this. Yeah. That looks good. Now like I said, we can spice it up a little bit by including a percentage difference, from the previous period. So we're comparing the current, aggregate to the aggregate from the previous period that we're interested in, and we're looking at the last thirty days. So what we could do is then I'm gonna move this to the side, and we don't need this anymore because we've got a template that we can work off of. I am going to create a Python cell where I will analyze our data or I will, I'll melt the data. Oh. This is just because it's, you're basically unpivoting it, right, essentially, for those of you who know, with pandas. Yeah. So melting would just take all those columns that she has at top, kind of move up a bit, and turn them, into into rows, if that makes sense. Yeah. Exactly. And now I'm trying my best to remember. Remembering Pandas syntax on the fly with people watching is, a special skill, so don't don't feel any any pressure. Also one of those situations where you could do it in SQL, but it's much faster in in Python in this case. Yeah. That's true. I think, Ginger's Ginger's good for that as well for kind of pivoting. But, Pandas and Times is Yeah. The issue with the issue with SQL is, of course, because you need to when you're unpivoting or pivoting, you always need to select all the columns individually. It's you can't, like, iterate through them in loop or something, so it's really good to have Python support for that. Or you could just call the, all the columns, which in hindsight could have done and just copy in that way. But last for here. And then, also, let's put in, the at risk as well. Let's not forget those. Does this work? Spelling mistake? Resurrect it. There we go. This should work. Right. So we've we've melted and we've unpivoted our table, which means and you'll soon see what I'm trying to do here. I want to query this using a SQL cell, and I want a column for the type of variable or the type of segment of users we're looking at, the aggregate for the current period, the aggregate for the previous period, and do a quick calculation where I calculate the percentage difference or the, the, yeah, the change from the current versus previous period. So what I'll do now is put a SQL cell, yeah, fill out a SQL cell, and our cell is f. We want the variable. I'm going to group by the variable, and I'm gonna do this now because, I hate running into that error where, you know, you've you've written out your SQL code, you've forgotten the group by, and then you just have to go in and do it again. So let's look. So let's do some case when date between, what is the we said we said eleventh of June, didn't we? Yes. Yes. We did. Yeah. Yeah. In June. Yeah. Tenth of June greater than or equal to cent of June. K. Let's just see this works. If you guys see me making a mistake, just jump in. Yeah. Looks good to me. This is our current period. First try. It's pretty good. Oh, it's yeah. This is this is a oh, no. I I was gonna say we had a streak, but I ran into that error with the, resurrected. So that's true. Well, I must say what what we can do just so we're not all kinda watching you write sequels. I can I can take a stab at kind of prettifying this template here, and then you'll come back when you've got the the percentages ratio? And then, actually, I think, Matthew, one thing it might be good to do. So this filtering that Sarah's done here, I think we're gonna have to do it with all the data to make sure that one week doesn't cut off. So can you maybe add a control cell to this this top cell that filters for that date? Yeah. We just maybe don't show the last week, the current week. Yeah. Yeah. I guess you have to add a Easier for another cell downstream. Yeah. Oh, we won't we won't watch. We won't watch. Why don't you spot the error? I mean, it's a question. I'll actually take this moment to to address Josh's question. So Josh asked, I think he's referring to this tile here, this at risk MAU. So he's asking, since this tile is one of the several you're going to build, how would you normalize this trend considering it's highly dependent on the number of new visitors during that period in cohort? Is it better in this format to bring in additional metrics to show a comparison, e g number of total users in each day's cohort, even though that number will show up elsewhere or show it as percentage instead? Yeah. It's a good question. I think it's my my advice would be to keep it as simple as possible. So, like, in this case, we have that risk here, and then we have our rate is gonna be right next to it. So as much as possible, you kinda wanna just show each of these elements as a trade off. So maybe you can see or you can, like, break this down further in a tree and show how it is new visitors and cohort size or something like that. And just so you're not trying to conflate too many things at once, that would be my advice. I think the one caveat to that is if if there is a normalized metric that the business already readily understands and uses, then I think it could be okay to add that in if it's something that's just like, we all understand this is a normalized metric already, add that in. But as much as possible, I try to break it down into, like, you'll see this pattern a lot of this total, and then there's a rate and another kind of total number. And this pattern will repeat itself a lot, and this at risk MAU could break down further into, yeah, new users and, conversion rate, something like that. And so that's where you kind of start to piece together those other levers. I hope that hope that makes sense. And any other questions, yeah, please, in the chat. Okay. Cool. She's checking in back over here. Yeah. Sarah's got percentage change. We've got our nice little formatted bit here. I'm now behind in building UAM in a little, frame here. Yeah. I think, interesting thing just now. So we've got, obviously, we've got, users. We've got metrics that we want to see drop, and we've got metrics that we want to see increase. And I think the default with creating any sort of, visuals where we have new percentage changes is in negative is red, positive is green. And so I wanna sense check you guys. So we've got with the at risk monthly active users. Do we want this number to go up or down? You're asking us you're asking the the wider me. Yeah. Imagine I could. Yeah. Or let's ask the Generally, I mean, generally, in in the context of what I've seen, we, as humans, want good things to go up and we want bad things to go down. So we want, like, number of errors to go down and, like, number of successful onboardings to go up. So based on that logic, you would expect something negative, like an at risk number, like, the good version of that would be zero. Would be positive. Mhmm. So if I'm if I want to visualize the percentage difference, do I want to display the pause the negative as a as a green? Oh, I see. Yeah. Or is it one that we just you know, we're we're thinking, like, the users will probably jump in, and they'll interpret the the numbers, you know, straight away from the colors and and infer whether it's a positive or negative. Or maybe just a bit keep it. I think it's one that feels like one, one one area of this Canvas actually that we have that we haven't talked about is I've kind of got these, frames set aside for questions for the business as we go. So that feels like one to add in there, of how people want that. But I think to start with, yeah, maybe just keep it consistent to start, and then we can we can flip. People people do react quite strongly to colors, I think. So should Mhmm. Awesome. Although, also curious to hear from the audience how people think about that in terms of color shading because I've seen canvases where they have the the negative trend at the top of the y axis, and I've seen the I've seen the opposite as well. It's, like, basically going to reducing to zero. Right? Okay. So I've put the percentage difference together. I can look. Nice. Let's make it a slightly smaller size. Let's see if we can do sixteen. Yeah. I think that's a decent sign. So, Taylor, I added this, control cell to the chart. So I've connected that up because you're you're sharing. It would be good to to show how that's connected up to the chart. Oh, yeah. So that you have in the in the filters, you have it connected to the control cell, and the control cell is just sitting above the, the chart at the moment. At the moment, it's it's kind of interesting because no matter how I cut it, even if I put, like, six or seven or eight days ago, there's always this drop off. Is that because of the weeks, how they're cut? I think that was the original thought, but if it's still it could also be that that that week is also dropping naturally. But, yeah. Sorry. I was just just checking Josh's questions. I think that's why it's good to have this as a control cell to kinda play around with it a bit. And, yeah, this is something, basically, we can when we have the rest of the charts built out, we can add this to all the charts so we can see how this affects not just at risk MAU, but with all the metrics. And maybe this is something weird that's happening with this particular metric. We can look into it. Mhmm. I think for the moment, we'll just put it as, we'll just fix it to what is it? Eleventh today. We'll fix it to the third of July. I think that we can leave that there. Okay. Obviously, if we if we if we continue copying this and we see that it misses one of the other ones, we can always change it afterwards. Yep. Yes. Basically, control cells are just a nice way to add variables, basically. Awesome. At risk MAU is done. I've got a little card for you here if you'd like. That's great. So what we can do with this is, basically, I can take this and make a copy of it, and then I can drop it in here. And then we can be sure that every actually, I should copy that first. So this is kind of acts like a template. And we can just drop it in these different places. And then yeah. Awesome. Nice. Move this around. Okay. Nice. Okay. Okay. Alright. So now we know how many users are at risk of slipping away. Let's should we do the next metric, which will be so you can do two things. You can look at the number of reactivated users. So we're looking at the metric that sits above this one, and that's next in the hierarchy. And then we can back calculate what the reactivation rate is, but I actually already calculated the reaction activation rate in the data. So you'll see in the, in in this table at some point in the data model we calculated the reactivation rate. So you can you can do it either way, but for ease, let's do it this way. So should we move on to the next one? Yeah. Yeah. We'll go ahead and follow you. Sure. Mhmm. At some point as well, we can also kind of, divide this divide and conquer on this. You can assign some of these to each of us, and we could take a take a stab at it. Sounds good. So I'm gonna quickly change I do the, like, the the last bits that, I always forget to do first. Otherwise, things look funky. Right. So let's look at well, let's actually let's calculate what the reactivation rate is, because we can't we can't aggregate them. So let's calculate it. So let's do so how do we calculate reactivation rate? So we're looking at risk users over reactivated users. So I'm just entering in a calculated field. Yeah. It's happened to watch my screen, and and so I was doing all the, all that exciting stuff. Okay. This is a conversion rate. At risk over reactivated. Does that make sense? No. That should be under one. Otherwise, we have a hundred and two conversion rate. Just really good. Can someone become active twice in the same time period? Like, if they supposed to be. If they if they fell out because the definition for reactivation was percent of users active in the last month, but not in the last week that returned today. So that technically means they could have been active in the first week and then not active again and then active again, and they would have been resurrected twice, which is normally, like, mythical wizardry stuff, but maybe it's not possible. I believe it's not possible, and the reason why is because I got the, metrics the wrong way around. So it's a better outcome, I think. There we go. Still really positive, though. Ninety eight percent. Oh, yeah. Nice. That's very high. I mean, this this fictional not Duolingo is actually is very persistent, so that's so Here we go. So there's a slight positive story. It seems like the reactivation rate is increasing slightly, which is nice. It's good to see. Nice. I have, yeah, slightly changed the, template there. Yeah. Good to see. Okay. Should should Matthew and I take different portions of this and and kinda start building them out? So, Matthew, do you wanna start you start from the right? Do you current users I can start at this You wanna do the resurrected block? Yeah. Yeah. Yeah. Sure. Mhmm. I'll I'll I'll grab current and new then. I can do those too. Okay. Nice. So, yeah, this is kinda how you can quickly scale this stuff out. It's usually nice when you've got people to help. We see it kinda common. It doesn't have to be synchronized like this at all. One of the things we see is, like, you could just I could put this and say, you know, sorry. When you next come back, I got a comment and ask her to look at this section later, and it can kinda be up to date when at a later time. But doing it at the same time is actually quite fun. So Yeah. I also find that, if you've if you're working on a bigger metric tree, then you probably have, you know, you analysts in the team that have different agreements or scopes, especially if you're an embedded team, that they understand better. So I imagine even, you know, within Lockbox, what we would do is we've got, you know, one an analyst that knows one product really, really well. We've got marketing, you know, an analyst that's more focused on marketing, and so we'd probably divide and conquer that way. And it's also just a a really good way for all of us to have visibility of areas that we're not familiar with or we we don't actively work with on a day to day. Yep. And I think that's the the powerful thing about Metrc Tree is you you do have that full visibility as a a business stakeholder regardless of, you know, where you are. So resurrection rate, what's my denominator inactive here? Resurrection rate, you're looking at dormant users. Dormant. Okay. Cool. Thank you. So resurrected over dormant. Okay. Right. I've completed one part of the metric tree. Yes. I'm gonna move on to new users, and then we're almost done. I was just about to do new. Oh. That's fine. Guys, I've got I've got plenty over here in my notes, so feel free to I'll I'll come I'll help you. I'll I'll come see you soon. So I need to fight over new users. It's, plenty. We might might have to fight over new users. I don't know. Depending on how we're are we compensated in any way for bringing new users to the app? I'll do the the DAU. That feels important. Yeah. Get that going. I might, need to move this slightly. See if there's any questions. Not yet. Yeah. We are planning on, you know, the last ten minutes or so for questions. So now is a good time to to add them in, so we have time to to answer them. And I guess, Matthew, we wanna add your list to all the visuals as well. So the already in there. Oh, it's already Control's already in there. It's already it's already linked up. Yeah. Okay. Cool. Because you copied the the template. So Yeah. Yeah. Okay. I believe we've got one left, and then we have our metric tree. Yes. Mhmm. Quick work, guys. I can see that the last thirty days cell is has a manual date filter in it. Oh, yeah. Remove the cell. Yeah. Good spot. But we would have we would we'd have to link that up to, another control cell. We could do that easily. Oh, I found a big problem. What's the problem? We wanna hear about twenty minutes ago. New new new users minus two hundred and forty two percent difference the last period. Oh, that sucks. Yeah. Right? I guess we've we've answered the question to some degree, Taylor. I mean, look at this look at this graph as Nickelback says. I mean, that's what he sings. Right? I can sing it if you want, but I think everyone knows. It took me a moment to be honest. Yeah. Good question from from Eddie. Yeah. Let's go ahead and connect these connectors cells. So, yeah, I just asked, hey, folks. Great workshop so far. Thank you. I think the most interesting part of this is how you would bring this info together and explain the narrative to the suite the c suite. Good question. I think one of the nice things about MetruTrees, if I'm honest, is is really not much explaining to do. I think if anything, I would, you know, maybe add a big arrow to this, what we found here. Like, sorry. You have a steady hand to add arrows. And just point out that's what we found. The metric tree, I mean, breaking things down like this, it's a pretty kind of, natural way to to read it. So I think it should become quite clear. I think this is something I was just talking to the, the three that are gonna be on on Tuesday, who are kind of sharing their experience doing this, it like, in their in their jobs and stuff. And that was one of the things they said as well is you'll be surprised how kind of natural this is for someone to pick up and read. Like, once they kind of get it, they'll they should be kind of taking it out of your hands. So there won't be too much I think you need to explain. The explaining, I think, is this part is the most difficult part. Like, if you agree to this, then all you've done is just add numbers to this explanation of, like, do you agree that daily active users is composed of these four things? And if you do, then it's kind of straightforward to understand. Yeah. It feels like quite kind of a non answer, but I think it should be relatively easy to explain on its own. I'll show you. And the other thing I would add is you have oh, go ahead, Matthew. Sorry. No. You got it. No. No. I didn't wanna interrupt you. What I was what I was gonna say about the explainability is, this should have been discussed. Like, the the overall tree design or, like, why why you're making the tree in the first place is probably something that would have come from them initially. So they would have given you an input like, oh, please investigate why DAO is decreasing. So, naturally, they're already curious about what the answer to that is. So you can basically just hook into that initial question and say, okay. We found out why this is happening now. Now we need to research why this happened. We go deeper than that. And then in this case where we found that new users is obviously declining, very rapidly, we would need to investigate further why that's happening. And we can maybe it's not valuable yet in a traditional organizational sense to go to c suite and say, oh, the reason for DAO decreasing is decreasing new users because you don't have an answer to that yet. But already sharing that step with them, I think, builds a lot of trust with with data. Yeah. I agree. I think there's a there's a story that we shared on on Tuesday, that kind of exemplifies this. And it's it sound like someone, let's say, someone separately found that there was a problem with with DAU or something. But having this to already kind of troubleshoot where to spend your time looking is a really good first step. Like, how many kind of meaningless not meaningless, but, like, you might get a lot of questions from a lot of different people that are trying to resolve this question, but they might be kind of scattered all over the place. But because you're at least able to target what you should be looking at next, you're spending a lot of good time there. And the only other thing I'd add to help explain it or maybe to help trust in the numbers is adding a table of data underneath it, which we haven't done here today. But I think if you're trying to kind of bring a a stakeholder that's you know, maybe they're kind of not trusting of these numbers or something like that, having a way for them to spot check these numbers and check it for themselves and see, you know, okay. Last week here are the ones from last week. You know, here are the number of users last week that we had from these different sources, whatever it is, so they can kind of see this broken down in a way that they understand and trust, I think, goes a long way as well. Yeah. I've just added one in so you can see what it looks like. Yeah. So you can see what the numbers look like on a week to week basis. And then, like, I guess it has there's there's more transparency here, So you can easily answer answer questions that come through like you said. Yeah. Or even just have, like I'm guessing when you go through this, there'll be, you know, I wanna see this by whatever the slicing slicing in a different way and being able to just quickly pull that up and go, okay. Let's look at new users by different sources or, you know, whatever it is. You can start to to dig into that. But I think what's important in, like, looking back at this flow is that this is just the first draft. So you've got you've gone one step, and you're gonna look at this, and it's gonna get bigger, and it's gonna get more complicated. And now you've got a place to start digging, and then you're gonna have another meeting to be like, crap. Let's figure out what's going on with new users, and that could lead into a whole another subset of this tree that gets built out because you need to figure out what's happening. Actually, I think it's kinda related to the next question actually from Steve. So in practice, would a small metric tree like this be connected to a much larger or much bigger tree that includes a company wide North Star metric like net profit, or would you wind up with many small metric trees that are focused on particular areas of the business? Good question, Steve. I've seen both, to be honest. I think it's up to your organization, the size of your organization, how you operate. I think there would be it's quite ambitious, I think, to have one mega metric tree. I think it's it is technically possible, but it's quite difficult to comprehend. So, usually, we kinda see it broken down into subtrees, but, the the three guys on Tuesday have each done it differently. So one kind of went from the top to the, NorthStar metrics. One went to each individual department and worked on theirs. It, yeah, could be done a variety of different ways. I think it just depends on what's needed rather than kind of one general solution, if that makes sense. Any other questions coming through? So I've just gone and done a little small thing, which I think is a a nice quality of life thing. So, basically, the last thirty days thing, you might like, a stakeholder might have told you, oh, I wanna see last thirty days in this metric tree, and then they decide differently once they see the data. And this would be a lot of effort to go in and change seventeen different tiles. So what I do is I use a combination of both text inputs and, date field. So the date field is coded to, to the actual, filter. Like, this is the thing that actually filters the data. And then I just type the actual filter out of what I want it to say in the text. So, for example, if I'm saying last x days as an explainer, then I just write thirty in there. So if I wanted to change this to the last forty five days, I could just do like this and then change this to to forty five, these two filters, and it will auto update all of these values and all of the text descriptions in all of those cells as well or frames. Nodes? What do we call them? Leaves? Yeah. Leaves. Leaves. Leaf leaf nodes. No. The tree leaves. Wait. So hold on a second. I have a little small thing now. So there's reading tea leaves. So you're saying that this is reading metric tree leaves as, like, a science form. I I mean, I'm you can say that. I'm not saying that, but it it's it's not not being said, I suppose. Yeah. That's a good, someone someone can own that, I think. That's good. We got a question from India. Yeah. Yeah. I noticed that the breakdown of DAU is pretty usually exhaustive, the subset of reactivated plus yeah. We'll read the whole formula. What if the breakdowns are causal in nature? Example, what would a metric tree to describe a stock price be? Yeah. Good question. This is, our decision to make this, basically, break down the function for DAU was intentional. I think it's a really good place to start. If you're wondering where to begin and match your tree, looking at the actual function and breaking that that down is a really good, solid way to start. You can't always do that, as you say. In which case, yeah, I think it's about choosing nodes that, one, makes sense to the business, and two, kind of can be broken down and and decomposed into other things that make more sense. So I think if you think about, oh, how quickly am I gonna get into a dead end here? Then that might be a way to to avoid that. And for things like that, for the the the vague ones, you probably are gonna get stuck. Like, you're gonna go down a path and be like, oh, no. I've I've screwed up, and that's okay. And you just kind of regroup and come back up and and try something else out. And, again, there's a few suggestions coming on Tuesday from the guys about, from their experience, what are good ways to to do that and avoid those mistakes. But, yeah, I think it's just something make sure it's meaningful to the business and make sure you can continue to decompose it. You don't try to do too much. That's why I'm kind of reluctant to to do any kind of normalization and stuff within these within each node because I think it's difficult to break it down further. Yeah. Good point. I think, with MetricTrees, what what's so powerful about them is they're kind of a diagnostic tool. And so they're super they're super descriptive, and they describe what your growth levers could be as a business. So, you know, for example, with new users, what we could do is we could look at, different marketing channels. What how many users are coming coming through our paid marketing, you know, organic channels, referrals. Has there been you know, did we change our marketing strategies so so that we don't invest as much in paid marketing? And is that the reason why we're seeing a drop here? So what I'd probably do is do an area stacked area chart showing, you know, number of users from each marketing channel. And maybe that would, you know, quickly answer our question or maybe, you know, warrants further exploration. But these, yeah, these are metrics that are are things that we can influence. We can change them, hopefully. And using this, we could probably also identify which one is more realistic to change as well depending on how many how long the hierarchy is, you know, how many iterative changes we'd probably have to do as a business in order to see improvements at the very top. And also what you mentioned, Taylor, about vague about vague ones, like, if you have a metric tree for something like brand awareness, it's it's really difficult because you you have something that you're observing and you can you can measure a lot of things. You can measure how many blue cars go past your house, but you can't really affect it in any way. This is what what Sara just said about it's really difficult to have a metric metric tree that offers explainability and diagnostics for stuff that you can't do anything about. That's really disappointing and very, very struggling for a business then. So you you'd wanna avoid diagnostically discovering things that you can't change, like where your physical store is situated. Like, if you discover that your physical store is responsible for like, that location is responsible for, you know, thirty percent decrease in foot traffic because of some major construction going on in the area. Like, don't think that you can try and shift the construction work to, like you know what I mean? Like, that's that's something that a metric tree is just gonna tell you what you probably already knew anyway beforehand. Yep. I think that's a good point. So I guess, yeah, one of the other things I'm just gonna cover is what what would we do next? So, sorry, Matthew, if you have any thoughts on, you know, from here we get to this point, what would what would be the next steps? Oh, I guess now we know, what to to drill into. We know the new users. Now now that we have this metric tree and we can actually spot we can do the root cause analysis, which I think I just went into and probably others have a lot of experience in. Like I said, I would probably look into the marketing channels to see if anything has any obvious changes have occurred. Maybe we're not investing as much in paid marketing. And if that's the case, I would definitely have a conversation with the marketing team and say, whatever strategy we implemented thirty days ago, sixty days ago, I think there's a consequence that looks negative for the business. And so then we would probably have would it would probably warrant a further, you know, more exploratory analysis, but at least we have something tangible to work with, and we can start thinking about actions being more actionable. Yeah. I would probably look at in particular to new, I would look at something like the onboarding. Some apps, like in a learning app, the onboarding is pretty short and very self explanatory. You basically just start doing language exercises, or learning exercises. But in in other apps, the onboarding process could be more more convoluted, more complex. Maybe there's KYC, the whole, like, know your customer thing. They need to do identity check of some kind. And and so I would I would try and understand this classifier as new. Right? Maybe there's a a very broad range of what is a new user. Like, are these just users that hit the app and then there's nothing else happening with them, or are these users that have already passed all of those identity checks and things like that? So I would maybe break this down in addition to doing things like by marketing channel as as Sara mentioned. I would definitely break that down into a better classifier of new and better understanding where if there's any hurdles there as well. Yeah. Nice. I think there's a both. Yeah. Very good places to go net go to next. The only other thing I was gonna touch on before we sign off is, just on kind of a more technical front, I suppose. So as this tree gets bigger and bigger and kind of more closer to production, there's a few things, you could do. So, one is I can put this in a frame, and I can share it as a report version. So if I have, some stakeholders here or, obviously, you maybe don't wanna show all the working to other people, you can do this. And then as as you're getting ready, any of this code can be deployed, or, like, exported, and it can be saved. It can go back into DBT. It can be exported into, you know, just whatever your tool is. So you can start to kind of scale this as it was. You can the nice part about it, I guess, is you can build this here really quickly as we've done in an hour, but you can start to set it up since it's actually kind of scalable and fits in with the rest of your your stack and system there. So, yeah, I just kinda wanted to touch on the the technical bits as well. Very cool there. I think we're getting ready to to sign off then. Or hang around for another few minutes if you have any other questions. Otherwise, admin wise, you'll get the next invite, hopefully later today or early tomorrow for the session on Tuesday. And you'll get a re recording for this as well around the same time. If you've got any questions for us, feel free to message us. You can email me. I think you'll have my email. Maybe find these guys on LinkedIn so they're willing to answer your questions as well. And, Sarah, Matthew, thank you both for coming. Really appreciate your time and the effort you put into this. And thank you guys for tuning in live. Really appreciate it and asking lots of questions. It's really great to see. I think this is definitely better than Google Meets as well. So thanks for bearing with us in a new on a new app. And cool. Thanks. Have a good rest of the day.