This is an edited transcript. For the blog post and video, see A Guide to Estimating Migrations - How Much Will My Drupal Migration Cost? Part 2/3.
[00:00:00] Janez Urevc: Hello and welcome to Tag1 Team Talks. The podcast brought to you by Tag1 Consulting. Today's talk is next in our guide to estimating Drupal migrations. You're going to want to stick around because we'll answer the one question we hear more than any other, which is, how much will my platform migration cost?
[00:00:21] Janez Urevc: We're getting more outreach than ever before from organizations that need help migrating to Drupal or that want to support upgrading their existing Drupal sites. So we're doing a series of talks covering everything you need to know about Drupal migrations from planning and estimating through to execution and delivery.
[00:00:40] Janez Urevc: The Tag1 team, the people who actually built and maintain Drupal's migration tools, and the leaders behind some of the most complex and challenging Drupal migrations are going to share their experiences, tips, and tricks to help ensure your Drupal migration or upgrade is a success. In the first two parts of our guide to estimating Drupal migrations, we talked about defining the scope of your project and the many business and technical decisions you have to make to narrow in on the best approach for your needs.
[00:01:13] Janez Urevc: That includes some decisions that are counterintuitive, like the fact that lift and shift may cost as much, if not more, than the greenfield re platform of your application. A lift and shift might, at the same time, reduce risk, making the extra cost worthwhile. If you haven't already checked out parts one and two, I highly recommend that you do.
[00:01:36] Janez Urevc: Check out the links in the show notes. Cost is often a determining factor in scope, affects how you approach your project. And decide what you do or don't do. And that's what we are focusing on today. How you go about estimating the costs of your project. I'm Janez Urevc, Senior Engineering Lead at Tag1, and I'm joined today by really well known top contributors to Drupal, including Lucas Hedding, Senior Engineer at Tag1, who is one of the five current Drupal Migrate Core subsystem maintainers.
[00:02:10] Janez Urevc: And Michael Meyers, Managing Director at Tag1, who produces Estimates While Sleeping, and he's a co host for this series. Welcome, both of you. Thank you for joining me.
[00:02:22] Michael Meyers: Thanks, Janez.
[00:02:25] Janez Urevc: Um, so, first, I would like to, uh, talk about estimating from a higher level. Like, there are different ways of approaching an estimation.
[00:02:41] Janez Urevc: Uh, some methodologies can produce estimates very fast, but might not be that accurate. And some methodologies require a lot of effort, but could result in, um, You know, fairly high quality estimations that we can really rely on. And Michael, you have a lot of experience with all sorts of estimations for many different projects.
[00:03:06] Janez Urevc: Um, so what, what are, what are the methodologies that you usually use when you're faced with an estimate?
[00:03:16] Michael Meyers: So for us as a consulting company, it might be a little different than say an end user of Drupal and where they might start, but a lot of organizations come to us and you know, the first thing they ask is, you know, how much is my migration going to cost me?
[00:03:32] Michael Meyers: Right. We need to figure out if we're a good fit for each other, right? Is this the type of project that we're looking for? Are we operating the same kind of budget range? And the way that we try and achieve that is twofold. You know, first, uh, you know, I ask questions like what's your budget and your timeframe, and then after everybody erupts in laughter, we get to the non answer, you know, nine times out of 10, that's something along the lines of, you know, we're still figuring our budget out.
[00:04:01] Michael Meyers: Um, or we're looking for you to tell us what the budget is. It's really rare that an organization is going to share their budget, unfortunately. And so where we end up starting is, um, what I would call a, uh, a comparable estimate, right, in order to try and narrow things down as quickly as possible and try and figure out if we're in the same universe, um, we look at other projects that we've done and we try and say this project is somewhat similar enough.
[00:04:27] Michael Meyers: That we can look to it and say, we think it's roughly around here, maybe more, maybe less, you know, if that sounds okay to you, we're not like wildly off your expectations. Let's continue the conversation and dig deeper.
[00:04:47] Janez Urevc: That's awesome. But I assume that, um, you know, not, not all projects are the same. And I guess it's really hard. There are always [00:05:00] unique, unique aspects of each project. Uh, so to do that, you need. A lot of data, I guess. Um, and even if you have a lot of data, even if you have a lot of past projects that you know how much, how long they took and, uh, you know, are similar to what you're trying to estimate, there is always some something that is different, some detail.
[00:05:25] Janez Urevc: Um, how does that affect the whole process?
[00:05:30] Michael Meyers: That's a really good point. Um, one of the reasons it works well for us is that we've done a lot of migrations. If you're an end user for Drupal, uh, or you're an agency that hasn't done a lot of migrations, you don't have the data to make comparisons. Um, and then, like you said, it's, you know, uh, especially when you get into large projects, there's, you know, there, there are certain fundamental similarities, but ultimately they're very different beasts.
[00:05:56] Michael Meyers: And so it's very hard to say your project is like this project, there just aren't those kinds of direct parallels. Um, so what we try and do is, uh, look at multiple projects and say, this aspect is similar, that aspect is similar, and we try and cobble together, uh, a comparable as best we can, um, you know, uh, ultimately there are times where we just can't come up with the comparable where we have enough confidence in it, even as a ballpark.
[00:06:25] Michael Meyers: Um, you know, uh, we did an estimate together, the three of us recently with a really large, complex project that took a very novel approach, you know, in their architecture, um, and, you know, it just didn't map to anything that we had done before from a migration standpoint. And then you add on, you know, the approach that they wanted to take, you know, more of an incremental swapping out different components or a different time.
[00:06:49] Michael Meyers: Um, we, you know, we just couldn't look to another project and say, Oh, this fits. We couldn't look to 10 other projects and say, these components fit. It just, it just didn't, you know, it broke the mold and that's not uncommon, you know, that it breaks the mold entirely or enough that we just say, you know, what, um, we can't give you a, you know, uh, a comparable that's really dead on.
[00:07:10] Michael Meyers: Um, you know, we can try and give you a range. We did this with another organization where we said, Hey, look for D9 to D10 upgrades, uh, you know, the lowest, the least expensive one we did was over here. The highest one we did is over here. We think you're somewhere like three quarters of the way based on what you've told us.
[00:07:28] Michael Meyers: So, you know, a pretty broad range and an estimate, but again, for me, comparables are not like, I wouldn't use a comparable estimate in a, in a proposal. But I also, you know, a lot of times wouldn't want to do a proposal without having the sanity check of a comparable estimate and make sure that everybody is somewhat aligned on where we're at.
[00:07:50] Janez Urevc: Yeah. The problem is if, uh, the actual estimate is an order of magnitudes different than, you know, what the client expects. And [00:08:00] I think that, uh, even not so good comparable estimates. Uh, help you prevent that from happening. Like it, it brings everybody on the same page, at least somewhat, which then helps steer the discussions that follow.
[00:08:21] Janez Urevc: Um, and what about like, let's say we, we did a comparable estimate and the prospect client, uh, was happy with the range that we provided. And then they, they would like to get a more detailed estimate. How would we go about doing that?
[00:08:47] Michael Meyers: So we combine a couple of different approaches to do, uh, our actual estimates that we'll include in their proposal.
[00:08:54] Michael Meyers: And I should say, you know, we only work on a time and materials basis. We don't fix bid things. There are times [00:09:00] where we'll time box things. It's not really appropriate for a massive project like a migration. Um, but, um. Typically what we'll do is we'll take, um, a, a bottom up approach, a three point, uh, and a, and a parametric methodology.
[00:09:16] Michael Meyers: Those are three different, uh, similar and sort of compatible ways of approaching an estimate. And it sounds fancy, but it's pretty basic and logical. Um, and what we found is that we get very accurate estimates from this approach. So when we look at our actuals at the end of projects and we, and we compare it to what we originally put together, uh, with a bottom up three point parametric methodology, uh, it's, it can be astounding how close we get, uh, to what we originally planned.
[00:09:47] Michael Meyers: Um, so a, a bottom up, um, uh, basically we, uh, we break a project down into tremendous detail. And then we work with engineers to really dig into what it's going to take to build that at a granular level. Um, and that's because it's really hard, you know, if, if I say, you know, how much is it going to cost to build this, you know, it's, it's, you know, you can't answer that, but if you start breaking it down into smaller and smaller and smaller pieces.
[00:10:17] Michael Meyers: You know, I can't tell you how much it's going to do this little thing and that little thing and what kind of resources we need to build that smaller component. So you create this, you know, really fine grained, uh, approach. You know, it's sort of like, you know, uh, scrum and story points. You wouldn't put a, you know, a massive story point item in a two week sprint.
[00:10:37] Michael Meyers: You know, you want to break it down into a, a smaller item that you can identify. Um, and the other key thing is, you know, you have to do the estimation with people who know what they're doing. Um, now breaking things down into smaller components definitely helps you make guesstimates, even if you don't have a clear sense, but the accuracy that we get comes from, you know, folks like [00:11:00] yourselves, you know, Lucas, you know, talking through and saying, here's, you know, what, you know, based on my experience and knowledge, it's going to take this amount of hours from these different resources to achieve this result.
[00:11:13] Michael Meyers: Um, and so. Um, it definitely, um, is significantly better with expertise, but I think it's, uh, it's still a good approach. Um, even if you can't break it down to the level of detail, uh, even if you can't have the expertise, you'll, you'll, you know, it's sort of like the comparables, you'll get better, you know, than you would otherwise, you know, uh, The other thing that we do is, you know, the, the three point, and that's just a, again, a way of saying, um, three data points, uh, we create a low estimate, a high estimate, and then we create some sort of, um, third point, which is either the average of the low and the high, or the average of the average and the high.
[00:11:56] Michael Meyers: Um, and, you know, you know, some people want to call that the likely estimate, you know, it's, uh, I don't like to use the word likely estimate because these are pretty large, you know, massive projects that we do with a lot of variables, you know, um, I like to give the, the ranges and say, here's where we think it's going to be at, and we're going to track our time and, you know, keep an eye on things and, you know, do our best to stay within that, that high end estimate, um, Um, And then the last component of it, uh, the parametric component, um, this is just a way of using existing historical data to refine your estimate process.
[00:12:33] Michael Meyers: So when, when I talked about the comparables before and I said, Oh, we pick and choose from different things, that's kind of a parametric approach, um, you know, bundled into the, the comparable. Um, another way that we do this, uh, is. You know, oftentimes, um, a project can be so massive. We can't estimate everything in the timeframe that we have, and we have to make certain assumptions.
[00:12:54] Michael Meyers: Uh, an example that we typically do, uh, component based architectures are very common these days. It's not, you know, uncommon for us to get sites with over a hundred components. And so we'll have an engineer go through them very quickly and sort them into buckets. This is a simple, a moderately complex, a complex, a very complex, you know, three or four buckets, and we just sort them into those buckets.
[00:13:19] Michael Meyers: And then we look at our historical data on what it would take to implement components in each of the buckets. And we just take those lows and highs. So rather than going through and, you know, estimating each individual component. To a hundred, 150 components, which would take a ton of time. We distill that down into a much smaller exercise based on existing data.
[00:13:39] Michael Meyers: Um, you don't, you know, and again, you can do that without having as much data, you know, you, you can estimate, you know, uh, one of each, you know, two of each and then collapse everything into the model that you have. Um, it's just a way of, you know, trying to be efficient with the time that you have, because you never, you know, it's like anything or a project, you never have enough time, money, or resources to do what you want.
[00:14:01] Michael Meyers: So you have to find ways to make compromises.
[00:14:06] Janez Urevc: Yeah. Like always in life. Um, it's, I think it's also worth noting that, uh, while this approach can produce very accurate estimates, it takes a lot of effort, like for, for big projects. When you start, uh, splitting things into smaller pieces, you can end up with a lot of pieces, um, and then you need experienced people to look into these individual pieces, provide estimates that can take a lot of time.
[00:14:42] Janez Urevc: Michael, do you have any? Any guess, like how much, how many hours an estimate would take us?
[00:14:53] Michael Meyers: We've talked about tracking this data and I don't, I don't want to know.
[00:14:57] Janez Urevc: But I would say like, yeah, go ahead. Sorry.
[00:15:00] Michael Meyers: Yeah, no, no, it takes, you know, this isn't something that we take lightly. You know, an organization comes to us.
[00:15:05] Michael Meyers: It's not like we just say, oh, great. Thanks for calling, you know, let's. You know, invest all this time in creating an estimate. This is something that we do very far down the line in the process where they've either already decided on us a lot of the case or it's down to us and one other organization and we're willing to make the investment to close the deal and demonstrate our expertise and other factors of that nature.
[00:15:30] Michael Meyers: Um, you know, we, we worked on a, the three of us worked on an estimate recently and it took over two weeks. Um, and it was, I mean, um, I was putting in nights and weekends, right? They, they came to us and they said, you know, you have a week to do this. And we were like, oh, wow, a week. And we started on it and we realized, oh my gosh, there's no way that we can do a justifiable estimate on this in a week.
[00:15:53] Michael Meyers: And before we could say anything, they came back and said, you know what? Well, you guys can take two weeks. Um, and it was a brutal two weeks. And I, you know, um, I don't know about you guys, but I, you know, I have a sense that we all put in more than 40 hours each week. Um, and, uh, I think that the difference shows, right?
[00:16:11] Michael Meyers: I think that the kind of estimates that we deliver, we consistently get the feedback from people saying, this is amongst the most thoughtful, you know, detailed estimate that I've seen in my career. You know, um, you know, a lot of people end up hiring us because the estimates that we provide, um, give them a tremendous amount of confidence, not just in our expertise, but in the thought that we put into it, our commitment to the project.
[00:16:33] Michael Meyers: So it's a way, you know, I don't know this would work for, for all agencies, um, Tag1 because of our, our reputation, a lot of people come to us already having made a decision to work with us, um, so, you know, we have the ability to, to do this, um, you know, otherwise, you know, if, if you're doing this earlier in the process, it's, it's, it's a much bigger risk, you would have to cut corners, you know, you, you have to rely a lot more on, on parametrics and historicals, it would be less of a, Um, you know, estimate for your specific project.
[00:17:05] Michael Meyers: Um, and you know, uh, there are times where we have to do that as well. Um, but you know, that's what I love about it is we get, you know, we can really dig into this with an organization. Say, Hey, we really want to work with you on this. We're going to make this commitment. Um, and it gives us a real sense of what that project is going to cost.
[00:17:22] Michael Meyers: And it also, uh, you know, leads to a lot of really important conversations. Um, I think we talked about this in part one. A bit, Lucas, where we started realizing, well, what about this? And what about that? Did you think about this? And, you know, I know you guys said you didn't want to do any major changes in this area, but if we did this change, it would actually be more efficient and save us time and money, would you be open to that?
[00:17:44] Michael Meyers: And, you know, it, it, it's an entirely different conversation at that point. Um, that gives us more confidence about who's the organization we're going to be working with. Are they going to be a good partner for us? They, you know, do they know what they're talking about? Can they make decisions? Um, you know, which is just as important, you know, where each, each side is sort of evaluating the partnership to determine how well this is going to work.
[00:18:07] Michael Meyers: So it, it really, you know, uh, we've walked away from projects where it's clear that it's not going to go well, right. Um, because of what we've learned during the estimate process. And so. Um, I think it's well worth the, the time if you can do it, but there's definitely ways that you can pare it back, uh, if you can.
[00:18:28] Janez Urevc: Yeah. Sometimes, uh, uh, an estimate for a fairly large project can take as much time as it would take you to build a smaller project as, as we've seen from our personal experience.
[00:18:44] Michael Meyers: Yeah. These are to put things into context. We're talking about multimillion dollar. Uh, you know, six month to 12 plus month type projects, um, you know, definitely worth the investment [00:19:00] at a quarter of a million dollars, you know, half a million
[00:19:02] Michael Meyers: You'd have to start making, you know, uh, broader estimates with different approaches, like the parametric comparables and narrowing it
[00:19:09] Michael Meyers: down from there.
[00:19:12] Janez Urevc: Yeah, that's that's great.
[00:19:14] Lucas Hedding: Um, and I've been able to do that, Michael. I've I've been able to take and use use that on smaller projects here and there where they're not half half a million, but they're 50 50, 000 or.
[00:19:29] Lucas Hedding: Let's, let's turn it into hours. Let's say, you know, the final estimate was three or 400 hours. Um, you know, that it can still work. You just have to be much more cognizant of where you're spending your time on those estimates and, and, and how confident you are you're going to get, get the work at the end of it.
[00:19:51] Michael Meyers: Exactly. Like you have to make that, that judgment call on the investment. Um, and your, your point is, is great, Lucas. Like, um, when we do D9 to 10 upgrades right now, those are typically much smaller projects, right? You're talking, I think on the low end, you know, 30 to 50, 000 on the high end, you know, low six figures are typically not big projects.
[00:20:11] Michael Meyers: Um, and. we're able to take a similar fairly accurate approach at a lower dollar level. And because it's a smaller project, it takes less time. Um, so it definitely maps well, especially when you're doing the same kinds of things, D9 to D10 upgrade, you know, there's a lot of, you know, there are more comparison points in some cases, but you know, still every organization is unique and different.
[00:20:35] Michael Meyers: The number of modules they're using. So it's, you know. Always challenging.
[00:20:42] Janez Urevc: Always challenging, yeah. But in general, these updates are smaller than a migration from a legacy platform or a migration from Drupal 7 to Drupal 10. And usually there are. Less unknowns and less moving parts. Okay, so we've come to a point where we broke down, uh, the project into smaller pieces.
[00:21:07] Janez Urevc: And now we have to go about estimating those smaller pieces, uh, from the bottom up. Uh, so Lucas, when you are presented with... Uh, tasks like that, um, where, where do you start? How, how do you approach this? Uh, well, let's
[00:21:26] Lucas Hedding: go into what are those pieces. So the pieces of a migration include the configuration and then the data, the configuration for a legacy site.
[00:21:37] Lucas Hedding: If we're coming from a previous Drupal site tends to be somewhat minimal. Like there's only one site name. And once you've pulled over the fact that your site name is Acme Incorporated, you don't have to pull that over day every day, uh, as in the case of like a more complex incremental data migration.
[00:21:57] Lucas Hedding: So, uh, put some time and effort into that. that, uh, figuring out what you're, uh, splitting out your configuration. But, uh, let's, let's talk about data. So the data is all that field stuff that has your first name and the body field and all of these, um, it's custom blocks and all the, the rich content that makes up the site.
[00:22:21] Lucas Hedding: And that's where you split that out by Content type or by, uh, paragraph type or field collection type, or maybe you have some custom entities. You start to split it up into these smaller pieces. And just like, uh, when you're trying to estimate for anything, if The estimation for the page content type is looking like, you know, we've got like a hundred fields on this page, content type, and this is going to, and you just like, like mind starts to blow up.
[00:22:59] Lucas Hedding: We'll split that into smaller pieces. Well, can we take the simple trivial data, the string data, the body field and all, and then there's maybe 50 fields that all have string data. And then you can say, well, that's really quick. That won't take long because we're just mapping the data straight over, but then you have complex data and you really just try to break it down into the smallest pieces that are manageable.
[00:23:22] Lucas Hedding: Just because it's a migration doesn't mean that all the other things that you've ever learned about estimating. You should throw out the door, uh, if your wife or your, yeah, she comes to you and says, Hey, we need to, um, we want to remodel the kitchen. Well, you can, you can start looking at, well, what are we going to do?
[00:23:45] Lucas Hedding: New cabinets? What are we doing here? And so you start to figure out what, what you're actually doing. You split that up into little pieces, same, same exact thing. You're just splitting it up into the smallest pieces and trying to do your best guess. So, uh, if you've got custom fields, those are more difficult than the ones that, um, address field has been, uh, has had a migration path and contrived for years.
[00:24:12] Lucas Hedding: But if you've got a custom field that doesn't have an upgrade path, you've got to figure out how you're going to map that complex 3, uh, nested 3 field combination data into a new data type in the new site. So what I'm saying is break it into the smallest pieces possible. And and then put that high and low estimate on it.
[00:24:34] Lucas Hedding: So if you are looking at the page content type and you think, you know, I've only got 5 fields on there. There's 10 pages total. Well, no, there's 100. There's 100 pages. I miscounted initially, and that happens regularly where you miscount and you have to go back and revisit. So there's 100 pages. There's five fields.
[00:24:59] Lucas Hedding: That [00:25:00] doesn't seem hard. So then you put down 10 or 20 hours for that, because you're going to try to incorporate. Are you going to incorporate QA, Quality Assurance, testing time into that 10 or 20 hours? Or do you need to separate that out, make that decision? So let's say it's 10 to 20 hours, uh, if you're like me, you could be a lot very, very confident because I've done a lot of migration.
[00:25:23] Lucas Hedding: So that's a solid 10 to 20 hours. But if you've never done a migration before, and you're not sure, and you're just kind of guessing, put down that 10 to 20 hours, and then we'll come and revisit it at the end of the, of filling out our spreadsheet. Because what we'll then do is we'll say, well, how many of these things are we really confident about?
[00:25:42] Lucas Hedding: And you might add a little fudge factor on, on the ones that you're on, um, not very confident. Uh, tag one, we use like a t shirt size of around confidence levels. Like this is. Oh my goodness. Really, really hard Or no, this is really, really well known to us. You can do that yourself. And so if you say that page content type of 10 to 20 hours, but I'm not sure , so then maybe you add 50% overhead on that just, and now it's 15 to 15 to 25 or 28%, uh, 15 to 28 hours.
[00:26:20] Lucas Hedding: Add those all up. Um, use your experience from like things that you've done in the past. Uh, and then don't forget to add any scrum hours. Like if you're going to be meeting as the team, or if it's just you, well, there's no scrum hours, so you don't have to. But if it's if you and two other people, you've got a, a, a themer that's going to be doing some things, make sure that you have time in there to meet and discuss things in there.
[00:26:47] Lucas Hedding: Um, So let's say now you've got an estimate. Um, maybe it's an easy estimate. So we'll say 400 and maybe it's a massive estimate. It's 4, 000 hours. Um, at this point, we're still working with hours. Uh, we're not attaching hours or dollars to these hours. You might want to figure out, uh, if you're doing it in house, like you're the agency yourself, because you work for the company that's doing the migration, um, you have to figure out.
[00:27:15] Lucas Hedding: Okay, I can do this part, but I need to make sure that my theme is available to do this. Or if you're working at an agency, you've got to work with your, your, your project management office to figure out resourcing on this and to make sure, well, we can give those things to a junior themer, or we can give those to an advanced themer.
[00:27:33] Lucas Hedding: You know, these are very gnarly migration concepts and, um, data here. So we're going to make sure that our, our best migration experts on that. And we can give all the rest of these simple migration things. To the guy we hired six months ago, and so you sort of map some of that out, and then you can start fixing dollar amounts to these hours, and it's a very involved process, but it comes out with some really high levels, uh, very [00:28:00] high degree of confidence on what this is going to cost, because you break it down to the smallest pieces, and because you are breaking it down to the smallest pieces, even if you miss large sections of things in that estimate, you now have assumptions, okay?
[00:28:17] Lucas Hedding: You've assumed that you're just migrating the page content type, and when someone in the organ, you know, they come up and say, Hey, we've got, and then they bring out this whole new piece of thing that they want to migrate that they didn't tell you about, well, that wasn't included in what you're assuming that you're going to migrate.
[00:28:35] Lucas Hedding: And so you can start to go back and have that conversation and say, well, let's make sure we estimate that now. Um, time and materials is a godsend. Trying to do an estimate with a migration on a fixed bid, you're going to lose your shirt, uh, almost every time. So, so make sure that if you're not, if you, if you are forced to, uh, a fixed bid.
[00:29:02] Lucas Hedding: That you have a ton of contingency added or, uh, really, really solid estimate such that you can go back and do change control because you've got it down to that level of fine grain. Hey, we're going to migrate the page content type when they pull out the blog content type or a WordPress blog or are a medium blog or wherever else we want to do this too.
[00:29:26] Lucas Hedding: That should be easy, right? Well, sure. It'll be easy, but it's not included in the original scope here. So we can then have a conversation about what that might
[00:29:36] Lucas Hedding: come in as.
[00:29:38] Michael Meyers: It's a really great pro tip in there. Lucas, like if our template didn't include this, uh, I swear there are times where I would miss it.
[00:29:46] Michael Meyers: Like when you, when you're doing a, a granular estimate. It's easy to get myopic and, you know, that tunnel vision of focusing on what you're doing and you have to remember, what about project management? What about the team's participation in Scrum? What about testing in QA and, you know, uh, revisions and reviews and feedback.
[00:30:05] Michael Meyers: And there's all sorts of, um, things that happen in a project. That aren't necessarily part of an individual item. And, you know, it, again, it, it depends upon how you, you, you know, you approach estimating and what you do or don't include in items, but it's really critical to, you know, that can be 50 percent of a project, you know, the things that I just mentioned, um, for, you know, and it's crazy to think about that, but it's true.
[00:30:30] Michael Meyers: Uh, and we're very transparent in our estimates. And a lot of times we get a little bit of shell shock from organizations. And, you know, there are organizations that we work with, where it's clear that there's going to be a tremendous amount of overhead in working with them, and it can be more than 50 percent of the project.
[00:30:45] Michael Meyers: Um, you know, uh, we have a client, you know, um, I've seen meetings with upwards of 30 people in it, right. You know, where they want all hands on deck type scenarios on a weekly basis, like that kind of stuff burns through, uh, you know, a tremendous amount of hours very quickly. Um, you know, so it's really important to factor in the non technical components, all these supporting aspects, uh, not just, you know, here's what it's going to take me to, to build these different components.
[00:31:14] Janez Urevc: It adds up very quickly and it's very easy to forget about those other aspects, especially for us technical people, because we are always focused on achieving the goal and forget about everything besides that.
[00:31:29] Lucas Hedding: One thing I've been adding more and more frequently on my migrations. On my estimates for migrations is big data.
[00:31:37] Lucas Hedding: If you've got more than a thousand field, uh, um, pieces of content of a certain content type, or we've got 2 million users, or, uh, we've got a hundred gigabytes of files on some remote file server or something that we need to migrate, all of those things just, uh, are a complexity multiplier. So while it didn't take long to migrate the pages, uh, to set up the migration for the pages.
[00:32:07] Lucas Hedding: There's just so many of them, so many things can go wrong and it'll die in the middle of the night. Now you run into sort of the man month issue where it's taking longer to migrate than what you actually have time. It takes two weeks to migrate over all these users and then you got to go back and think, well, how can we make this faster?
[00:32:24] Lucas Hedding: And I think we've got a, uh, discussion around how we do performance tuning for a migration. Or, or, but maybe you can't make it any faster so that you've got to figure out mitigating ways to, to make these things work. And it's all because of the volume and the complexity of the data.
[00:32:42] Janez Urevc: Yeah, my past experience is exactly for this reason, Lucas.
[00:32:47] Janez Urevc: Um, migrations were during, throughout my career, always the most time consuming projects for exactly these reasons. Um, [00:33:00] the time just starts flying when you're dealing with huge amounts of data.
[00:33:06] Michael Meyers: The deep migration aspect of a platform migration is often the most expensive, uh, section of a budget. If not among the most expensive, it depends upon the amount of features and functionality you have and how complex your site is.
[00:33:22] Michael Meyers: But typically speaking, the data migration component is, you know, the most expensive or, or very close. Um, in any platform migration is substantial. Um, it's, it's a big project. Uh, and then Lucas, you made me think of something else, which is. Uh, the QA, and, you know, I don't, I don't want to get into it because we have a lot to cover, but, you know, just, you know, something to keep in mind, um, who's going to look at the data after it's been migrated, right?
[00:33:49] Michael Meyers: You know, if you have millions of rows, millions of pieces of content, millions of images, there's things that you can automate, but you can't even, you know, run that level of automation, you know, [00:34:00] frequently enough, you know, necessarily, right? You know, you really need to think through how you're going to invest in.
[00:34:07] Michael Meyers: You know, and, and, and account for that, um, because that, you know, itself could be a big effort, you know, and often, you know, it's something that we have to do collaboratively. With the organizations we work with, you know, no one organization can take it on independently, even with automation.
[00:34:24] Lucas Hedding: Well, the thing to keep in mind with, with QA around a data migration is the data is still there.
[00:34:30] Lucas Hedding: So if, uh, if you get to the point where it's like, where did all these and fill in the blank with what is missing. Go? Well, it's on the old site. Hopefully you've, well, obviously you've mothballed it. Right? Right? You've mothballed the old site somewhere. So six months later, they say, we'll pull out the data from this old conference that we, uh, conference content type.
[00:34:53] Lucas Hedding: And we're like, well, what conference content type? Oh, that one on the old site. And so you pull it over then. And, and, and it's
[00:35:02] Lucas Hedding: not hard.
[00:35:03] Michael Meyers: Don't delete your old system for a year.
[00:35:06] Lucas Hedding: No, yeah, leave it. It's, it's cheap. Throw it on an S3 file system if you need to, you know, the database and the, the files from the Drupal 7 site, but keep it around.
[00:35:19] Janez Urevc: My, my rule of thumb nowadays is that I never delete anything because storage is just so cheap that it makes sense to just keep everything around. And that's my wife. It drives her nuts because she wants to have everything clean and organized. And she hates me for that. But I think that, yeah, keeping stuff around, um, proves to be useful at some point.
[00:35:51] Janez Urevc: There is another thing that, um, could affect the estimate quite a lot. And that is custom code, like. Um, usually, especially on larger projects, there is some custom business logic that is implemented as, you know, custom modules, if we're talking about Drupal, and, um, usually we have to move that custom logic over to the new site.
[00:36:21] Janez Urevc: Um, how, how would we approach estimating that?
[00:36:27] Lucas Hedding: Well, first a note on that, between data and code, I've never had a client say, I can't upgrade because my data is too hard to migrate. It's always because of the code. I got too much complex logic in my code. That's the, that's the toughest nut here. So let, you know, let's go into with our eyes open.
[00:36:50] Lucas Hedding: Our, our code is, encapsulates our business logic. It's the, the meat and potatoes, the lifeblood of our, of our, of our system, our website. And you can't make it, uh, you can't minimize that at all. However, however, there are so many simpler ways in Drupal 10, Drupal 9, much more simple ways to do things than what there were in Drupal 7.
[00:37:22] Lucas Hedding: It's just.
[00:37:26] Lucas Hedding: It applies to custom code too. So like when we went from Drupal 7 to Drupal 10, so many of these core, uh, contrib modules disappeared off the marketplace, like, um, a lot of token got pulled out of the token module into core token, or I don't know, dozens of modules, same applies here with core or with, with our custom code, we have permissions that were very, um, complex in our old site, in our new site.
[00:38:02] Lucas Hedding: It's going to be much easier because entity access is so much more powerful. It's, it's not node, node access anymore. It's not this whole node grant system that was super complicated. It's, it's so much easier. Um, and I could just keep going on and on. So, yes, it's the most complicated part. But many of those things that you were doing were bolted on over time, and, uh, you can do it a lot better, a lot easier, and a lot faster, and it's going to cost you a lot less, uh, your technical debt is going to be a lot less, but how do you estimate?
[00:38:47] Lucas Hedding: Take it down by functionality. Break it down into those small, bite sized pieces. Just like you broke it down to a single page content type and then you looked at it and said, okay, that that's the right size or no, no, no, that's still too big. And then you need to break that down anymore. Break down your, your modules and you're going to have, in all likelihood, a lot of really small modules.
[00:39:08] Lucas Hedding: So those are easy. Get those out of the, out of the way. Snowball your estimation here, right? So get the easy things knocked off. Oh, that was a simple one. We don't even need that. That's a simple form alter. That's not, you know. Uh, or it was nice to have, and we don't really need it if we're, you know, we're working on a most, um, minimum viable product approach here.
[00:39:29] Lucas Hedding: Um, so you, you get rid of the simple things and then you come down to the really hard to, to crack modules and then you split up your big. Huge Mundo massive module into little pieces and say, Well, this is for malters. This was customizations for the conferences that we had these things over here. We don't even use that anymore.
[00:39:50] Lucas Hedding: I didn't realize we had that, you know, and so you break it down. And then you've got now, uh, a smaller, smaller chunk and [00:40:00] start applying that high. and low, both ends, estimate on it, and depending on your confidence of your knowledge of Drupal 10, you can, um, say, well, I think this will be easier, or maybe I don't know at all.
[00:40:16] Lucas Hedding: So then if you don't know at all, then you start having to, and it's a complex piece of your, your logic, then you have to start adding in some uncertainty factors and multiplying it, um, high and low. By 54 percent more or, you know, I'm pretty sure this is not going to change between Drupal 10 and Drupal seven here.
[00:40:34] Lucas Hedding: So let's, let's say it's just 10 to 20 hours. We don't have to add any fudge factor and get as much of it down as possible. Uh, you'll, you'll. You'll have your backlog. So that's the other benefit to doing this, this very fine grain approach is you now have a backlog. It's not waterfall by any means. We still do scrum.
[00:40:56] Lucas Hedding: We still do two week sprints, but at least you've got a nice backlog of 50, 100 things that you can pick off. The chart and say this is our most important thing and you can really prioritize right out the gate. The pieces, uh, you can bring in like your content editor team and they're like, well, we need to, we've got these, all these brand new pieces of content we need to build.
[00:41:20] Lucas Hedding: We didn't have in the old site at all. Well, okay. Let's make sure we build that new content type first for you. Uh, cause again, that's a good example. How often when we're rebuilding a new site. Do we have new requirements inserted while we're still worrying all the time about this old content, but the old content will be there at the end of the day.
[00:41:42] Lucas Hedding: So if there's some new features, the, the, the customer wants, make sure that those get weighed in with the old content too, and you can prioritize all of this when you've got the, the, the, the tasks because of your fine grain estimate and then you start working it. Go ahead.
[00:41:59] Janez Urevc: You've made a really good point that, um, a really thorough estimate, uh, helps you with project management as well.
[00:42:07] Janez Urevc: So it's not just an estimate, it's also already an investment into the project itself. Um, so it is time consuming, but there is, if an estimate is done right, there is more value to it than just knowing how much it will cost and how long it will take. Um, and yeah, speaking about cost, we will, um, we will have fourth and hopefully the last episode in this series, uh, where we will.
[00:42:40] Janez Urevc: You know, finalize this discussion, bring everything together, and really, um, finally answer the most, uh, important question that everybody is joining us to hear the answer to, uh, but before we wrap. Um, I, uh, would like to remind everyone, that besides this series and, uh, final episodes of, uh, this series about estimations, we have some great talks coming out after that.
[00:43:12] Janez Urevc: Uh, the first one will be. What we think will be a three part series on ETL, um, which, uh, where we'll going to cover the Drupal Migrate tool, uh, into a lot of detail and, uh, discuss how do we extract from various sources, which is E, how do we handle complex transformations using it, which is T, and how do we, uh, efficiently load data into a target system, which is L, um, Uh, we will also, uh, talk about performance.
[00:43:46] Janez Urevc: Performance is something that we care deeply about Tag1 and as we discussed during today's talk, uh, it applies to migrations as well. Um, we have a talk coming up about, uh, how to profile and tune a migration. including and especially the long running migrations. And we're doing as part of that, we will also talk about incremental migrations.
[00:44:14] Janez Urevc: Um, huge thank to the Tag1 team. Thank you, Lucas. Thank you, Michael. Thank you for joining me and sharing your expertise. Um, make sure to check out the other segments in this series. The links are down in the description below. Um, if you like this talk, please Remember to upload it, subscribe it and share it.
[00:44:40] Janez Urevc: Check out past team talks at tag1.com/ttt. That's three T's for Tag1 Team Talks. Um, and as always, we'd love to hear your feedback and topic suggestions. You can write us at [00:45:00] ttt@tag1.Com. Again, that's three T's for Tag1 Team Talks. A big thank you to Michael and Lucas and everybody who tuned in.
[00:45:09] Janez Urevc: Thanks for joining us.