This is an edited transcript. For the blog post and video, see A Guide to Estimating Migrations - How Much Will My Drupal Migration Cost? Part 1/3.
[00:00:00] Michael Meyers: Hello and welcome back to Tag1 TeamTalks, the podcast brought to you by Tag1 Consulting. If you've ever transitioned a website or application from one tech stack to another, especially a large scale system, you know, it's not a walk in the park. It's challenging. It's time consuming, and it's got a lot of moving parts.
[00:00:17] Michael Meyers: You need to plan, you need a strategy, and you've really got to know what you're doing to pull it off without a hitch. We're getting more outreach than ever before on migrating and upgrading Drupal sites. So we're doing a series of talks covering everything you need to know about Drupal migrations from planning and estimating through to executing and delivery.
[00:00:35] Michael Meyers: The Tag1 team, the people who actually built and maintain Drupal's migration tools, and the leaders behind some of the most complex and challenging migrations in Drupal history, are going to share their experiences, tips, and tricks, and help ensure that your Drupal migration and upgrades are a success.
[00:00:51] Michael Meyers: Today we're going to be talking about one of the most common questions we get, which is, How much will my Drupal migration cost? in this segment, which is part one, we're going to [00:01:00] talk about defining your scope of your migration and the questions you need to ask to outline exactly what you're going to do.
[00:01:06] Michael Meyers: And we'll talk about why or why you might not want to do things in different phases and at different times or at all so that you can hone in on that scope and then we can determine what you need to estimate. In the upcoming part two, we're going to talk more about how you actually do that estimation and covered a lot of the things and get into what we talked about today in more detail so you can start to build out your ballpark estimate for your migration.
[00:01:29] Michael Meyers: I'm Michael Meyers, the managing director of Tag1 Consulting, and I'm joined today by two really well known top contributors to Drupal. Including Janez Urevc, my co host for this series, and Lucas Hedding, who is one of the five current Drupal Migrate Core subsystem maintainers. Janez and Lucas, thank you so much for joining me.
[00:01:46] Michael Meyers: Welcome. Thank you.
[00:01:48] Lucas Hedding: Thank you for inviting us.
[00:01:51] Michael Meyers: Before we get started, I also want to thank Benji Fisher, who's another one of the Migrate API maintainers who we work with here at Tag1. He's putting together a presentation on a [00:02:00] related topic, and it helped inspire some of what we're going to be talking about today.
[00:02:04] Michael Meyers: So let's jump into things. You know, before you can even begin to do your estimation, you really need to get a sense of the scope, right? You need to define what it is that you're gonna be doing in your migration. Are you migrating your application, your data, your infrastructure, all of the above? Um, there are a lot of questions that you need to ask, and that's what we'll be covering today, starting with more business considerations and then diving more into the technical considerations.
[00:02:31] Michael Meyers: So one of the first things you need to think about, you know, you have an existing technology, you want to move it over a new one, um, are you going to be doing a lift and shift? Are you gonna be doing a rebuild? And you know, how much of your site do you want to keep or change in these processes? Um, Lucas, can you just give a, you know, a quick definition, what is the difference between, like, what is a lift and shift versus a rebuild?
[00:02:55] Lucas Hedding: Well, the way I'd define that, and everyone might have a slight tweak on this, [00:03:00] uh, lift and shift is you try to keep it as similar as possible without making any changes. So if you had bugs on the old system, If it was possible to keep the same system the same, you're not going to repair and fix those bugs, because the goal with a lift and shift is to get you from left to right, old system, new system, as quickly as possible, minimize the moving parts, so all that we're really dealing with is the movement, not any new bugs, new, uh, new, new issues with the site, uh, Obviously, it's not a usually it's usually not a pure lift and shift because there are a few things that you're going to resolve as you move.
[00:03:46] Lucas Hedding: But that's completely contrasted with a rebuild, where you start from the ground up, you burn the old system down and we're building from new, all that you're retrieving and using from the old system is maybe some [00:04:00] content, all your content, but you're really starting from what I like to call a green field, where there's nothing there, you envision what you're going to build, maybe you pull over All of your content, because that content was good, but the way it was presented, the look, the feel, what was promoted on the homepage, uh, needs to change because of your business, uh, priorities have adjusted since you built that old system.
[00:04:28] Michael Meyers: So... Lift and shift, minimizing changes, rebuild, typically a lot more changes and you need to figure out what those changes are going to be, theme functionality, removing technical, uh, that we're going to get more into some of those aspects later on when we get into the code aspect of things.
[00:04:45] Lucas Hedding: And if you knew what all of that rebuild was going to look like, it might actually be faster than doing a lift and shift.
[00:04:50] Lucas Hedding: It's not, I don't want to minimize that, that the rebuild is hard. It's, it's probably not any harder than lift and shift in a lot of cases, [00:05:00] as long as your rebuild isn't adding a lot of new features. The hard part with the rebuild, I think, It's figuring out what it is, that time that you're spent in meetings over weeks and with your designers and with your, your, your executive team to figure out what it actually is that you want to do that, uh, in Drupal world, we would call it bike shedding, right, where we're trying to figure out what color of the, we want to put of paint on this bike shed.
[00:05:29] Lucas Hedding: Who cares? Well, The executives do, the company does, and that rebuild definitely needs to know what the color of the homepage is going to be. Yeah, definitely. It goes into it, it's not any more highly complex or easy than a lift and shift in a
[00:05:47] Lucas Hedding: lot of cases.
[00:05:49] Michael Meyers: I think that's a theme that's going to come up a lot today.
[00:05:52] Michael Meyers: Um, especially when we get to theming, um, you know, the idea that, uh, you know, uh, your decisions aren't necessarily like a [00:06:00] lift and shift isn't necessarily faster and cheaper than doing a complete redesign. And that's one of the things that makes planning a migration amongst many other things so complicated is that there's, you know, a tremendous amount of detail and nuance.
[00:06:13] Michael Meyers: Behind these decisions. Sometimes it's a business decision. Sometimes it's a risk decision. Sometimes it's, you know, a cost decision, you know, and they may or may not be faster or cheaper. And these are all the things that we're going to dig into and figure out. Um, another aspect is How you approach your migration, right?
[00:06:34] Michael Meyers: So whether you're doing a, uh, a greenfield or you're doing a redesign, um, are you going to say, for example, migrate everything at once? You know, can you do this incrementally over time? There are a lot of different strategies that you could do to approach this. You know, Yanis, I know you've been involved in a lot of examples like this.
[00:06:57] Michael Meyers: Maybe you can give us just, you know, an overview of one concept or, you know, that sort of illustrates how you might do a more of an incremental migration.
[00:07:09] Janez Urevc: Yeah, sure. We recently worked with a client. It's one of the largest hospital groups in the United States. Um, and actually migrating from a legacy system to Drupal, um, they, uh, wanted to recreate their content, but they also were concerned about the timing and their ability to do it, uh, by the time.
[00:07:37] Janez Urevc: Uh, it was required to launch the new site. Um, so they decided not to migrate in the usual sense. Uh, but, uh, we instead we created. Uh, a custom module, a custom Drupal module that, um, if a, a [00:08:00] URL is accessed that doesn't exist in Drupal, it will try to fetch it from the legacy. Um. platform, which at that point doesn't, that is not publicly on the internet anymore.
[00:08:14] Janez Urevc: It's behind firewall and only Drupal has access to it. And this way, when there is new content, we can deliver new content when, uh, and when that's not the case. We can fetch the content from the legacy platform and inject it into a new redesigned website. And this allows them to do it incrementally for that reason, because when they are ready to create new content, they create it in Drupal and, uh, it switches automatically.
[00:08:44] Janez Urevc: So for them, it's really simple. It helped with, uh, it helped speed up the project. Um, cause they didn't need to recreate all the content, which in such a large organization would be quite hard to do in the given
[00:08:59] Janez Urevc: times of timeframe.
[00:09:01] Michael Meyers: They have an insane amount of content. Um, and if I remember correctly, their system was going end of life, their expensive licensing fees.
[00:09:08] Michael Meyers: And so building this content proxy to expedite the. Launch of the new system with only some of the content in the new system, um, save them a lot of money address the end of life issue. There's a lot of, uh, you know, benefits. And I think, you know, business reasons why they would want to do that. you know, there are so there are so many creative strategies that organizations take for different reasons like that.
[00:09:30] Michael Meyers: Lucas, I know you've been involved in in several as well. Um, can you give us a sense of the kind of creative solutions? You've been involved in to facilitate more incremental migrations and maybe why these organizations chose to take that approach.
[00:09:47] Lucas Hedding: The, there's a couple of ways to do it. Um, there's the Drupal incremental migration approach, which maybe we'll touch on in a minute, but we're really talking about not [00:10:00] migrating some content consciously right now, like this large hospital group.
[00:10:05] Lucas Hedding: Um, there's a large. missions organization that I've worked with for several years, um, supporting their, their regional network of sites. And they've got 40 or 50 different countries that have their own little, Hey, this is the country. Come to England and here's the, you know, here's the whole presence, um, because they, they do a lot of, um, recruiting and, and, and fundraising in England, uh, they opted to, to do it on a per URL basis.
[00:10:40] Lucas Hedding: So, uh, Drupal has the whole, uh, language prefix. Our country prefix, they're doing it on a per country basis. So UK gets to decide when they move over. And so the UK organization gets to, um, put the thumbs up when they're, when their content is ready on the [00:11:00] new platform, uh, we're moving from Drupal seven to Drupal 10, we were using, uh, a hosting provider, uh, that provides this capability, um, through its routing YAML, uh, You can do it with Amazon.
[00:11:14] Lucas Hedding: You can do it with a couple of different, um, software, um, hosting providers out there where you're doing this. If you're in a large fortune or a large company with a lot of capital, you might have a bunch of of Cisco load balancers. Or some other network devices like Cisco, that's what I'm familiar with when I worked at a large fortune 50 insurance company.
[00:11:37] Lucas Hedding: That's what we did all the time. We didn't even do it because we're migrating necessarily. That's just how their server farm was set up is their server farm was segmented out where if you came in with this set of URL prefixes on the URL. We, we sent you over to this server farm, which was Unix. And over here, we sent you to this [00:12:00] IIS server farm and over the, you know, that's how it was all done for years.
[00:12:04] Lucas Hedding: Uh, and there's no reason why you can't continue to do that. If, if you're able to chunk out your content on a URL basis. Uh, and and send people to your new site, uh, or or the old site.
[00:12:21] Michael Meyers: That's uh, that's exactly what The Economist did, major media company. They, uh, maintained a URL mapping, um, you know, similar, uh, conceptually, at least to what Jan has mentioned and what you're talking about, come to the site based on what URL load balancer sends you the correct server, uh, one is a legacy system, the other is Drupal, and it enabled them to do a migration over time, and that you looked the same, so that the end user, there was no difference, uh, and then once they were fully over, they were able to upgrade their All right.
[00:12:50] Michael Meyers: Um, they're designed to where they wanted to be. Um, another cool, you know, solution. And there are so many different ways that you could do this. Um, uh, Symantec, which is a company [00:13:00] we worked with for over a decade. Uh, they had, um, A lot of different, uh, CMSs, both different versions of Drupal that they were running simultaneously, as well as systems like AEM from different groups and departments, and they were locked into their AEM license for multiple years.
[00:13:16] Michael Meyers: Um, and so, uh, what they did or what we did with them is, uh, launch a decoupled front end that provided a unified, uh, a new updated interface from the start, um, and pulled in content from these different systems. And then over time, we were able to decommission, you know, pages and aspects of those systems to the point where we could then remove them in the back end and simplify things.
[00:13:42] Michael Meyers: So, there, uh, it's, it's pretty, you know, amazing how you could approach this. And a lot of it comes down to things like business needs, technology capabilities, and things of that nature. Um, another thing
[00:13:57] Lucas Hedding: I want to, I wanted to add that I think the, the, the reason why we're talking about this proxy, I think that the reason why maybe someone will correct me on our call on our, on our interview here today, I think it's because the urgency, like if we'd been having this conversation five years ago, we would have seen that, Oh, we've got tons of time to migrate over, but we've got all in the back of our minds here now that Drupal seven is going away.
[00:14:21] Lucas Hedding: We're going to lose security support. And if we can hide it behind firewalls. While we're trying to get our ducks in a row, then we might give ourselves a little bit more time to, to do that potential rebuild or do things more properly and correctly. And, and that's why we're talking about these all modified ways to do this migration or upgrade incrementally.
[00:14:54] Lucas Hedding: Uh, the other sort of traditional, quote, traditional, Incremental migration that has, has been bandied around in the Drupal community for a long time is, is, is quite a bit different. So let's define this approach here where we've got an old and a live site, uh, an old and a, and a, and a new site, a legacy, and a new site.
[00:15:18] Lucas Hedding: Both live serving customer traffic at the same time. Both in parallel, what traditionally, and you would have heard this five years ago. That's, that's all we were talking is this Drupal incremental migration, where we migrate the legacy site to the new site. And we incrementally update the, the new site from the legacy site while the legacy site is still serving traffic.
[00:15:52] Lucas Hedding: And. We give all of our stakeholders a chance to go to the new site for several weeks to do [00:16:00] quality assurance, testing, validation, all of our, executive stakeholders, making sure that it's all right, but we can't stop the business from still doing what it needs to do, publishing new content. And so we need to incrementally start moving that content over and now we just updated someone's password over on the old site.
[00:16:21] Lucas Hedding: Okay, we got to update that password over on the new site because we can't control when people are changing their passwords. That's the other sort of incremental, uh, migration approach. I think we're going to probably see less and less of that because we just don't really have time when we've got 15, 18 calendar months between now and when Drupal, uh, sunsets, we don't have the luxury of two months of quality assurance sitting there and making sure everything works.
[00:16:51] Lucas Hedding: Does, does that make sense?
[00:16:53] Michael Meyers: Yeah, we did that with examiner. com, which was the first top 100 website on Drupal, which we grew to a top 50 website. [00:17:00] Um, they had, you know, just insane amount of content, uh, and the volume of content coming in, and they had over a million users contributing to their site. It was a local news and information site.
[00:17:11] Michael Meyers: So we leveraged that strategy for, for many reasons. But we're, you know, for the reasons you talked about, right, like it enabled us to QA the new system in parallel, enabled us to give access to some editors and beta tests, you know, the new system before, you know, we gave broader access to get feedback and tweak things, you know, so, um, mitigated risk, um, there, you know, there are a lot of different reasons, you know, scale, risk, testing.
[00:17:37] Michael Meyers: Um, you know that, you know, some of the things we're talking about are more edge cases that are, uh, I think really large organizations, really large data sets, you know, a lot of the organizations that we deal with have very unique, large challenges that may not apply to. Most, you know, the average Drupal migration, right?
[00:17:58] Michael Meyers: So it really depends upon [00:18:00] where you're at. Um, I think one thing that a lot of organizations, regardless of your size, think about, and one of the questions we get a lot is, you know, to redesign or not to redesign, right? Organizations are moving their site, you know, um, you know, they, they, they don't necessarily want to do the lift and shift.
[00:18:18] Michael Meyers: They see this as an opportunity to make a lot of changes to improve their site and address a lot of the pain points of their users. And they want to change their design. Uh, some organizations don't want to change their design for many different reasons, you know, Janez, you, you've dealt with a lot of this, um, can you provide a little bit more background and context and the pros and cons?
[00:18:37] Michael Meyers: And like, um, you know, this is one of the things you're talking about earlier, where it seems like it might save you some costs, but maybe it doesn't like, you know,
[00:18:47] Janez Urevc: um, yeah, like the.
[00:18:52] Janez Urevc: The expectation would be that, uh, lift and shift approach here would be faster and, uh, and cheaper. [00:19:00] Um, but the fact is that when you're going from Drupal 7 to Drupal 10, or even if you're going from some other platform to Drupal, it's basically a very similar
[00:19:13] Lucas Hedding: situation. Um, even if you
[00:19:15] Janez Urevc: keep the, the, the look and feel of the, of the website, the same, you still probably have to rewrite quite a lot of it because, uh, the API has changed the approach to theming changed.
[00:19:27] Janez Urevc: For example, in Drupal 7, we, we had PHP template, I believe. And then in Drupal 8, Uh, and then 9 and 10, of course, we switched to Twig. So even if you're not changing anything, you will still have to rewrite all your template files into Twig, which could be, uh, quite a lot of effort. Um, so in a way, sometimes...
[00:19:54] Janez Urevc: It's not even cheaper at all to do a lift and shift approach. And from, you know, from this perspective, it makes quite a lot of sense to, to do a redesign when you are
[00:20:08] Lucas Hedding: doing a migration because
[00:20:10] Janez Urevc: a lot of the times will be better value for money, but then it also adds additional risk. Um, and as Lucas mentioned previously, all these decisions will need to be made, which can usually be very time consuming and, you know, might, might delay your project.
[00:20:31] Janez Urevc: So, um, it really comes down to what the priorities are, um, and what the organization needs. Is, uh, like getting the, the migration out as soon as possible. The main priority, then you will do a lift and shift. Uh, do you wanted to do a redesign anyway, at some point? And you have, um, possibility to, you know, think about it and to make decisions about new designs.
[00:21:01] Janez Urevc: Then probably going with a redesign makes, um, quite a lot of sense, at least
[00:21:09] Janez Urevc: financially.
[00:21:11] Michael Meyers: We're, uh, we're working on a migration for another fortune 500 company right now, and, you know, their original scope mandated no design changes, and when we talked to him about it, it wasn't that they don't want to change their design that they're going to down the line, and they understand the cost of keeping it like you mentioned, it came down to the fact that they have a lot of other internal priorities and systems that they're maintaining, um, you know, from a resourcing standpoint, and We They just weren't in a position to, to execute on aspects of the redesign, right?
[00:21:41] Michael Meyers: And it incorporates a lot of different people across the company. I want to get involved in the redesign process. So I don't, I don't necessarily use the term, like it opens a lot of can of worms, but it, it, it, you know, internally within the organization, even if we're doing. You know, all or most of the execution.
[00:21:58] Michael Meyers: It increases the [00:22:00] burden on the organization, making it harder. Um, you know, and and in some cases adds risk as well. So a lot of different factors, you know, like you said, cost resourcing are going to play into whether or not you do
[00:22:13] Lucas Hedding: this. But even if, and I think I'm thinking the same, uh, use case, Michael, uh, even when we got down to brass tacks and some of the, like, very specific, hey, could we do potentially something slightly different here if it made sense and would speed us up in Drupal 10?
[00:22:32] Lucas Hedding: So. While they initially came in the door with, Hey, we don't want to, we just want to do a lift and shift. We can't because of our deep knowledge. That sounds a little bit, you know, but we do know, I mean, we do know quite a bit about Drupal. Uh, and so, you know, our deep knowledge here, we said, uh, and everyone's going to have different set of knowledge going into, uh, an effort with, with a client, but could we do this different?
[00:23:00] Lucas Hedding: Could we slightly, it'll, it'll actually be longer to do it the old way. And, and in all, every single one of those conversations we had with that client, they're like, yeah, let's be realistic too. Like, our goal here was to save money, not to have the old warts and all system that we had previously.
[00:23:20] Lucas Hedding: So even in those, there's still a lot of one offs, Drupal 10. Uh, and and I think that's what what they'll end up doing, right? I think that it won't be as bad as it was. We will make incremental improvements. Even though it was originally said, let's just do a lift and shift.
[00:23:43] Michael Meyers: Yeah. No, it's, I mean, it highlights the, the, the depth of knowledge required to really evaluate this, you know, which makes it really hard.
[00:23:53] Michael Meyers: You know, they, they're, very sophisticated. You know, they're a very technology savvy organization. They're an awesome [00:24:00] partner. Um, but as we sat down with them and we, and we really dug into it, exactly. We were able to come up with strategic reasons. And specific places where it made more sense to actually make change and get to their goals faster.
[00:24:13] Michael Meyers: And they said, yeah, that makes total sense. You know, we didn't initially see that. And so, um, you know, for the organizations out there listening, you know, people that are at other agencies, I know that listen to this, there are people that are at, uh, you know, uh, users of Drupal businesses, you know, you have different factors and considerations.
[00:24:30] Michael Meyers: Um, you know, that's a that's a really important aspect, and I highly recommend anybody who's going through a migration at the very least, you know, consult with a partner on the early aspects of it right to make the most of it strategically, you'll, you know, make back that, you know, you'll get a return on that upfront investment, and that initial consulting to make sure that you take the right approach, even if you don't necessarily work with them for implementation, you'll, you know, [00:25:00] That that organization is a great example of very technology savvy and still had an opportunity to improve things and get a better outcome.
[00:25:10] Michael Meyers: So we talked a lot about the business considerations, and we don't have a ton of time, so I want to shift gears into some of the more technology considerations. Um, and again, you know, this is, um, this is hard because Uh, sizes and everything, right? Uh, some things are, are, are big and easy. You know, some things are, are, are complex but fast.
[00:25:34] Michael Meyers: Um, and so it's really hard to look at things and say, Oh, well, you have a lot of this worth it, you know, and say, you know, your time here is going to be easy versus hard, just like that evening migration example. Um, so let's jump into different aspects of a migration. We covered a little bit of this at a high level in our introduction to the series.
[00:25:52] Michael Meyers: We talked about sort of the anatomy or, you know, the jargon behind the components of a migration, um, things like code [00:26:00] themes, et cetera. Um, So from a what? What are one of the first things you need to consider when you're looking at the technology aspect? Of, uh, of a migration. Lucas?
[00:26:12] Lucas Hedding: Uh, the first one that every client brings up is, what about all my contributed code, uh, all these contribute modules that I'm using.
[00:26:18] Lucas Hedding: I've got 87 modules that I'm using from, from contri right now, or, or whatever, like some large number
[00:26:25] Michael Meyers: that's like 870, I think .
[00:26:28] Lucas Hedding: Some sites are, uh, actually I don't think I've ever run into 870, but I have run into several hundred before. I got to have all of those modules, right? And, and depending on the client's, um, understanding of how Drupal works, some of them come in with a little bit better understanding that some of that can be replaced with Drupal core.
[00:26:52] Lucas Hedding: Uh, with, uh, one contrib module instead of five, uh, or that could easily be, um, provided with just a [00:27:00] small theming tweak that was impossibly complex in Drupal 7, but now is just a couple of lines in a twig file. Uh, so the answer to what do I do with all these contrib modules? Usually ends up being less of an issue at the end of the conversation, because most of those, uh, contrib modules either don't have a replacement because there's no reason to have a replacement, or by now, we're like, how many, eight years into Drupal 8, 9, and 10.
[00:27:34] Lucas Hedding: Uh, yeah, there's, uh, instead of, uh, node label, now there's entity label. Instead of node queue, there's entity queue. And so on and so forth, like there's some more general solutions that are out there that will work not just for, uh, for nodes, but it'll work for taxonomy and for media and all these other things that are now built into core.
[00:27:59] Lucas Hedding: So then that brings you to the next part, right after contrib is, well, I've got all this custom business logic in my custom code. How do I handle that? And. Uh, you know what, that I don't want to minimize that. I don't want to minimize that at all. And there are definitely a lot of sites out there that for one reason or another decided that we were going, well, I know why they had some really complex business logic in their Drupal seven site.
[00:28:28] Lucas Hedding: And they wanted to get every last little penny out of that, that investment. Uh, I know here at Tag1 we've, we've, what was that a year ago, six months ago, we launched a new Drupal 7 site for a client because it was the right thing to do at the time and it was quick and easy. And they had a lot of assets that were able to be plugged into Drupal 7.
[00:28:49] Lucas Hedding: Um, so, you know, there, there's a lot of reason why Drupal 7 would be out there for a long time, because you're trying to get your investment out of that technology. But then you have all this complex logic that you need to upgrade, and there are tools out there. There are mechanisms to do it, uh, and it, you can't, you can't get around it.
[00:29:12] Lucas Hedding: It's going to be a tough nut to crack. Uh, my hope is that it becomes easier. My experience is that it becomes easier to do some of that custom logic, uh, entity access, node access, and Drupal 7 is a bear. It's quite easy to do in Drupal 10, relatively speaking, it's still hard work. All of these things are hard work.
[00:29:34] Lucas Hedding: I don't want to say, make it trivialized. I don't want to trivialize the work that we've put into these systems for the last 10 years. But when you're measuring relatives, it's so much less effort to do in Drupal 10 than it was in Drupal 7. And now you can actually test it with unit, um, unit level testing with PHP unit.
[00:29:57] Lucas Hedding: Uh, And I think as you go through your custom code, it's going to be a lot of work. A lot of it, even, even back to our example, uh, we just shared with this, this site that, um, uh, large, large tech stack, uh, where we, they want to do a lift and shift. We went through meticulously 60, 70 custom modules, an insane number of modules.
[00:30:23] Lucas Hedding: And it didn't seem like it was going to be all that, I mean, it's not proving to be all that hard to replace some of these things because a lot of it was form alters and other cosmetic changes, cosmetic changes that are, are, are not as one as needed, uh, or, um, as complex as they would have been in your old system.
[00:30:47] Michael Meyers: So this is about, you can take some percentage of your custom code, hopefully, and map it to core and contrib. And get rid of that custom code.
[00:30:56] Lucas Hedding: Some. And then the rest, like that, that, that, that [00:31:00] gold, the logic was the hard part in there. The if, this, then do that. None of that logic goes away. If this, if it's the 13th of the month, then it's a Friday, put a message, boom!
[00:31:13] Lucas Hedding: You know, that, that, that custom logic that was there, that doesn't go away. Your smile. Okay. Um, that logic is still there. You can just lift it from the old site and plop it into the new site right where you needed it to be. If you had some scoring system for insurance rating or something, right? That is probably there in the old site.
[00:31:38] Lucas Hedding: Still, if this if under 25 and there had five crashes with their car in the last six months, then, you know, charge them a lot more for auto insurance. That logic is already in the old site. So just move the logic over, put it into the new site and a [00:32:00] new container. But the logic is not needing to be
[00:32:04] Lucas Hedding: refactored.
[00:32:06] Michael Meyers: Interesting, so you have some savings. uh, I want to go back for a sec, because, Janez, you worked on a, uh, a performance and scalability project recently, and, and I've been doing Drupal for 20 years, and, and I, I, uh, I learn new stuff all the time from the team, but, but this really fascinated me.
[00:32:21] Michael Meyers: Um, correct me if I'm wrong, but, They had a lot of contributed modules and one of your, like, it was really impacting performance and scalability in some cases. And, and you had a recommendation where they could combine some things into one module to reduce the number of contributed modules. And that, that kind of went against everything I knew about Drupal in my head of like, Oh, there's a module for that.
[00:32:43] Michael Meyers: Of course you should use modules. And, you know, it never occurred to me that, you know, uh, too many modules would be a bad thing that, you know, that there might be other implications, uh, to having that.
[00:32:55] Janez Urevc: Modules are a great thing, . But if you, if you have too, too many of them, uh, then they start affecting performance.
[00:33:05] Janez Urevc: 'cause you just have to load all this code if nothing else. And, uh, sometimes people are using a module for, uh, some minor functionality, but the module itself does 10 times more. Other things. So you're installing this huge module to use 5% of it. Um, and in those cases, um, it could be useful to find some alternative that is lighter or maybe even, you know, um.
[00:33:42] Janez Urevc: In such cases, sometimes it makes sense to go with a very simple, small custom module. Um, and you get a lot of performance gains from that, yeah. And, um, exactly as you said, like the, there is a module for that mentality that we have in Drupal. It's great, um, and we should be proud of that. But, um, if performance is your consideration, um, then, then maybe, you know, you should stop and consider it twice, uh, not just blindly at all.
[00:34:18] Janez Urevc: Uh, first search result that comes up. Um, it's, it's also really depends on what kind of project you are. Like if you're building, uh, uh, brochure site for, uh, for a local sports club or something like that, then, then the performance aspect is probably not that important because you don't get that much traffic.
[00:34:41] Janez Urevc: But if you are dealing with, um, with an enterprise system, then performance needs to be considered seriously. And this is also another opportunity, I think, when you're doing migrations, um, and with regard to the custom code, um, over the years, there is probably technical depth that accumulated over the life of the project.
[00:35:05] Janez Urevc: And when you're when you're migrating, importing country code from, let's say, D7 to D10, it's a great opportunity to review all the custom code that you have and figure out if there is some that you can simply remove. Um, and that's great because you need, you don't need to deal with it. And it's also great for performance because you have less complexity.
[00:35:30] Michael Meyers: Performance should always be a line item in any project, specifically a migration. Um, you know, we've been doing a lot of work with Google to improve Drupal's performance and the performance of like large internet platforms to sort of speed up the web. Um, you know, they've done a ton of research. The faster your site is, the better the end user experience is, the better your business outcomes are, more page views, more time on site, more revenue if you're an e commerce site, um, so an investment into performance is going to pay significant dividends, it's one of the, you know, one of the key areas that you should be focusing on, and Unfortunately, it seems to be, um, at least up into a threshold, the kind of thing that a lot of organizations focus on only when it becomes a problem, right?
[00:36:20] Michael Meyers: Like when it's under that threshold, it's okay, even though it could be much better and really benefit their business if it was better. Um, and so I definitely encourage folks that are thinking about doing a migration and the success of their site and their future to seriously consider aspects of performance.
[00:36:38] Michael Meyers: Uh, in, in everything that they're doing, whether it's, you know, a number of modules and consolidating things and just performance tuning and optimization.
[00:36:48] Michael Meyers: one of the biggest factors in a, uh, tech stack migration is always the data and user migration. It tends to be one of the most complex, gnarly, time consuming, expensive, often very underestimated. Aspects of the larger migration, and I think that's because a lot of the details and understanding the complexity can be in, um, you know, Lucas is the, you know, one of the maintainers of the migrate API.
[00:37:21] Michael Meyers: Um, can you give us a little bit more insight into, you know, what's being migrated and how it's underestimated, you know, and, and some of the factors that you need to consider, like, do you have to migrate all these things? Um, you know, if you are, what, you know, what's the, the big concern?
[00:37:39] Lucas Hedding: Um, there's two parts to a migration.
[00:37:43] Lucas Hedding: There's the config and the configuration, and then there's the data. Let's, let's dive into that data. If you've got a lot of it, then you kind of have to do a force multiplier with your estimate. If you've got a hundred [00:38:00] users, well, go hire an intern. They can build those hundred users or those hundred pages, those hundred terms in four hours.
[00:38:10] Lucas Hedding: And sometimes that's the easiest way to do your quote migration. But if you've got a million users, it could take more than 24 hours to pull over those users because of the sheer volume of data that you're pulling over. Uh, hopefully you're somewhere in the middle. Uh, so you've got just a moderately complex site.
[00:38:32] Lucas Hedding: But if you've got any, any oddities in any of your data, which if you're still on Drupal 7, you've probably been on Drupal 7 for more than two years, you're starting to show a little bit of age here. You're, you're, you've got four or five, seven years worth of content and data. And back seven years ago, you wouldn't have.
[00:38:53] Lucas Hedding: Maybe done it that way. And so then you improved it. And so then now you're starting to see all the odd things in your data [00:39:00] surface. And now you've got to deal with the fact that images that were on your posts from seven years ago, we're still working on Drupal seven because it was cached. No one ever re revisited that node.
[00:39:13] Lucas Hedding: And resaved it, but now on the Drupal site that you've upgraded to, that cached thing isn't there anymore. So you're, you're, you're surfacing up all of these sort of latent things and all of the good and bad and changed business, um, decisions that you've made over the years. And what you end up getting when you migrate the content over is garbage in, garbage out, good data in, good data out.
[00:39:44] Lucas Hedding: If your dates are right in the old system, then they're going to be fine in the new system. If your content was all well structured in the old system, then it'll still continue to be consistent in the new system. If you were doing replacement of, of text with tokens and body fields, well, now you've got to, uh, make sure that continues to work.
[00:40:08] Lucas Hedding: And the images that were, like, raw images that are 4 meg images, Well, maybe you wanted to spend a little bit of time now in the new site and make sure that those are nicely image styled images, because that's now possible in Drupal 10. Oh, but now that makes the icons, those images small because they were raw and they're filling and then there's all of this sort of trickle down.
[00:40:36] Lucas Hedding: In fact, as you're touching the data, you're looking at it again for the first time in seven to six months, how do we handle that? And there's a lot more eyes on it. So it takes more time than you think, because data is actually the business of, of the website. And when you add more of it, then it [00:41:00] takes longer to do, more to review, more to migrate, more to review.
[00:41:05] Lucas Hedding: And it can be very, uh, time consuming to the point where some folks will be like, you know what, we don't need to bring over revisions. We don't want to have all of our old history because we might see some, it's just more to, to deal with. Just bring over the most recent published content. And that's actually probably one of the very best recommendations to doing a migration quickly is just pull over the published content that's live on the site right now.
[00:41:32] Lucas Hedding: Don't worry about revisions. Uh, Proceed with just that because it's less to review, less, uh, it's a lot faster to, to, to migrate a lot fewer moving parts and, and you've already got enough going on with, this upgrade anyways. So make your life easier.
[00:41:54] Michael Meyers: You brought up a really great point there about QA and many different aspects of migration, having a QA, all of that content.
[00:42:04] Michael Meyers: Um, Yana is, um, one of the things that, uh, we worked on. Uh, for that, uh, that fortune 500 company that, you know, we're lift and shift with some changes. Um, you know, we put together a unique idea, you know, or or an idea that maybe more organizations should consider to automate some of the quality assurance aspect because they just have so much content and they have, you know, compliance and other requirements that mandate that their content stay.
[00:42:34] Michael Meyers: You know, very much the same, you know, a QA burden of that from a manual standpoint would be, you know, just cost prohibitive. Could you talk a little bit more about, like, some of the things you can do on the QA side of things to automate and expedite?
[00:42:52] Janez Urevc: Yeah, sure. Um, not just that manual testing is time consuming if you have a large data set, but it's also, um, not so easily repeatable.
[00:43:06] Janez Urevc: Um, I mean, if you had automated system that, that checks, you can, you can run it daily. You can run it, I don't know, on every release or whatever your needs are. Um, and this is practically impossible to do, um, as thoroughly with manual testing. Um, and yeah, we, we, for this client that you mentioned, we proposed to do visual regression testing, which is.
[00:43:37] Janez Urevc: Basically a tool that first creates a baseline, basically creates screenshots of pages that you want to check. And then, uh, when you, when you need to test it, you do this again and then compare how much of the page changed. And if that is above certain threshold or... You know, there might be areas of the page that you don't want to change at all or, you know, whatever your needs are, then this can be flagged and easily identified.
[00:44:09] Janez Urevc: And that can, um, make the process more efficient, can surface up problems sooner. Um, problems that might otherwise would go. Unnoticed, which we don't want. Um, so yeah, I think people will be using more and more of that because, um, um, manual testing is often not sustainable.
[00:44:39] Michael Meyers: Migrations are an iterative process and you're constantly pulling over data.
[00:44:44] Michael Meyers: We're going to talk more about the migration tooling and the process and upcoming episodes. You're definitely running and rerunning, you know, aspects of your migration all the time. Uh, it would be impossible to QA things multiple times, even if you could do it once, could you do it, [00:45:00] you know, repeatedly is another factor.
[00:45:02] Michael Meyers: Um,
[00:45:03] Lucas Hedding: well, depending on the size and what you want to invest, you can write tests for the data migration. We did that. We did that with a very large telco. That I helped migrate and we got test data and then we would, and then we ran a full million, you know, full, large data set. And then we ran into bugs with that from point clicking.
[00:45:24] Lucas Hedding: And then we added that to our test cases, but that all comes at a cost. How much are you really willing to invest in having a certainty that all of your content is, is around? Uh, because another, another option is to test lightly. Maintain the old site in the mothball state, and then if you run into a, Oh my goodness, we missed, fill in the blank.
[00:45:50] Lucas Hedding: Well, the content's there still. It's not gone. You can always run another migration. A month, five months, [00:46:00] two years after you've. Upgraded the site.
[00:46:04] Michael Meyers: Definitely. It comes down to like, you know, you kind of touched on this earlier, you know, for some organizations, you might be able to manually upgrade things.
[00:46:11] Michael Meyers: You don't need automation, you know, for some organizations like a large pharmaceutical you talked about in the first episode, they have, you know, FDA compliance requirements that ensure pixel perfect, you know, uh, you know, between versions or they have to go through thorough review processes. And so, Yeah.
[00:46:28] Michael Meyers: For them, you know, making an investment in something like a visual regression testing system is well worth it. But for the organization with fun users, it would, you know, it'd be crazy, you know, it would cost more to do the visual regress testing system than it would be to migrate their entire site. And I think organizations fall somewhere on that spectrum and they need to make.
[00:46:49] Michael Meyers: Business decisions and cost benefit decisions is the where they're going to invest that money. And again, that's a big part of, you know, estimating your effort is [00:47:00] understanding the pros and cons of these different approaches and how they apply to your unique needs and where you want to make the most of the investment that you have, because no matter how big you are, you know, everybody has a limited amount of capital that they can put towards this effort.
[00:47:14] Michael Meyers: So it's about, you know, strategically leveraging that to get the most out of You know, your investment or or to meet certain requirements. Um, the last thing I want to touch on with respect to data and user migration before we move on, because I've personally experienced this with large migrations. The content of your legacy system and its structure versus the content of, you know, your new Drupal system can be very different.
[00:47:45] Michael Meyers: Um, and there are a lot of challenges, you know, say, um, you know, the content of an article, for example, or the content of your page in a legacy system may be laden. With heavy markup, HTML, things of that nature, um, and in order to import that into Drupal with a more modern structure and approach, you need to do a fair amount, maybe a lot more than a fair amount of processing, right?
[00:48:10] Michael Meyers: So, um, you know, uh, Lucas, I know you deal a lot of, you know, that's sort of the transform aspect of the ETL process in migrations, um, You know, is there any way that people can like quickly look at their existing system, you know, and what should they be looking at to say, Oh, my gosh, this is gonna be simple, trivial.
[00:48:34] Michael Meyers: It's not an issue or, oh, my gosh, you know, this is gonna take a lot of work to transform it into my new system. And maybe are there, like we said, you know, compromise decisions that they can make, um, to reduce that effort in, in, in doing that upgrade.
[00:48:51] Lucas Hedding: I think the most common use case that I see in those situations where they, they really want to get to a much better data model.
[00:48:58] Lucas Hedding: Maybe they were using one big body field in the old site and they realized that the use of paragraphs or layout builder or all these other new goody things that you could get that they didn't use in Drupal 7, uh, pull the content over to a legacy body field. They'll be on the new system, you'll pull over the old CSS even, just grab the css file on the old site, put that and attach it on, on, if that body field is populated, make sure that CSS is also loaded.
[00:49:35] Lucas Hedding: And it's a really quick win to get you off of your legacy system. And then If it's a really popular, uh, section of the site, maybe it's a, uh, a, a landing page that you really needed to make better, uh, after the migration, go ahead, do it, use layout builder, use paragraphs, use, uh, the 20 fields that you added to that content type, whatever it was that to make the structure.
[00:50:07] Lucas Hedding: And then you can, you have, you know, your, your dual monitor system where you copied. Some of the content from the legacy field into the different fields to make it more structured and Uh, into the layout builder blocks, you hit save, and maybe you're even using some of the newer, uh, moderation tooling where you can mark this as a draft.
[00:50:29] Lucas Hedding: Right. Draft were harder to do in Drupal 7. They're like built into Drupal core now. So you publish it as a draft, have your marketing team look at, have your, maybe you're a small organization, so it's you looking at it, maybe you don't even need the draft, right? So at that point, you've got them and you got, you're on the new system because you had to get there because the deadline for Drupal 7 is coming.
[00:50:55] Lucas Hedding: You're there, you copy, you paste, you publish, and now you're in the new data structure. And who cares about your old blog posts that were of the old system, um, old look and layout. You've got a new masthead at the top of the page. Your footer's still good. No one was going to look, I mean, yes, you poured your heart into that blog post, but it was seven years ago.
[00:51:16] Lucas Hedding: No one's looking at it anymore. So don't, don't invest your time there.
[00:51:21] Michael Meyers: It's a, it's a really great, you know, it's another approach to Incremental migrations, right? So it's there are other ways that you can do incremental migrations within a data migration. You can pull over your legacy content with that structure, put it in, make it look as good as possible.
[00:51:38] Michael Meyers: Um, but adhere to sort of the old structure. Reduce your effort and get there fast. And then over time, should you so choose, maybe do a subset to that, like high priority content, you might improve that, you know, so that you're moving the structure, making it look as good as possible, you know, putting things into, into fields in Drupal, that's, you know, not [00:52:00] dissimilar, you know, in many ways, exactly what that content proxy system is doing for the hospital group and in a, certainly in a twist.
[00:52:08] Michael Meyers: Right. In the sense that it's being run on less popular pages, right? The most popular, most important pages are the ones that were actually migrated to the new Drupal system. And, you know, there's millions of pages of content, some of which get like one view a year, you know, that, you know, um, and so, you know, some, they're not as important.
[00:52:28] Michael Meyers: Um, and so the, you know, they're handled by the proxy and over time, they'll get. added into the new system and, you know, so it's just a way to sort of manage your resources and your spend over time.
[00:52:41] Lucas Hedding: Those solutions, those proxy solutions, the example with the large, um, multinational missions, uh, Christian missions organization I mentioned where they're doing it by country, the downside, gotta put it out there.
[00:52:54] Lucas Hedding: Is it's really hard to then get those systems, those legacy systems to upgrade. That's what we're facing now. Is it so simple to keep the old site alive, that it really has a hard time dying.
[00:53:11] Michael Meyers: Janez, you created a monster.
[00:53:17] Michael Meyers: Um, no, I think that's a really fair point, you know, um. It works so well and so seamlessly and it looks so close to the actually migrated content. It sort of removed the pressing forcing function for the organization to get off of that legacy system, and they're just going to continue to prioritize, you know, what's most important to them, and I would be shocked if the proxy isn't around, you know, is meant to last, I don't, you know, I don't remember, six, nine months to facilitate, you know, a system less than a year.
[00:53:51] Michael Meyers: I'd be shocked if it doesn't end up being there for two years, right? And so there's definitely
[00:53:56] Janez Urevc: even more.
[00:53:57] Lucas Hedding: Yeah, well, in my case with this missions organization, they're now starting to do more of the carrot and stick. Well, they were doing the carrot. Now they're doing the stick approach where they're saying, here's the line in the sand.
[00:54:08] Lucas Hedding: We're going to shut your server down. We're not going to give you access. Just to get them over. Um, I mean, it's a, it's a not for profit. There's not a lot of extra resources, but it's been sitting there waiting for them to migrate to Drupal nine. And now it's Drupal 10. It's been sitting there for almost two years and there's just so many regions in the world that have had other priorities.
[00:54:30] Lucas Hedding: That they haven't prioritized the upgrade so that, you know, that the incremental comes with a double edged sword, I guess, is kind of what I'm leading to,
[00:54:41] Janez Urevc: but, for example, that the proxy approach that we were talking about when you have the legacy system air gap. It could be okay to keep it running even after the end of life.
[00:54:53] Janez Urevc: I mean, it's not ideal, but if it's air gapped, then, then maybe, maybe you could afford that for a while. So, um, in our case, the legacy system was not Drupal, but since they were also facing end of life, um, It's kind of a similar situation than Drupal 7 situation that, that many, many Drupal users are facing now.
[00:55:19] Janez Urevc: Um, so maybe this could be a one solution. It's not ideal, but it's, you know, it's an
[00:55:26] Michael Meyers: alternative. It was a really, uh, creative and it is a very effective solution. And for, you know, organizations and their migration needs. Range a massive spectrum. Um, you know, this was a, uh, a really awesome fit for this organization.
[00:55:45] Michael Meyers: They could afford to run it. It helped enable them to meet their business needs. You know, if they do end up running over two years, it's not the end of the world. If it's three or four, you know, maybe we can revisit this conversation. Um, you know, that that's, you know, it's it's there are trade offs to every approach and decision.
[00:56:05] Michael Meyers: And I think, you know, it's, it's really good to your point, Lucas, to just go in eyes wide open and understand what those trade offs are that you're making, what the repercussions of those decisions might be, and to try and have a plan and strategy in place so that, you know, you're not just passing the problem down the line, that you're actually, you know, completing your objective in a reasonable amount of time and reasonable amount of cost.
[00:56:30] Michael Meyers: Um, so I want to just quickly plug some upcoming talks. We touched on so many great things. Um, we're going to do a lot of talks in this migration series. We've got a three part series coming up on ETL, the, uh, extract, transform and load process that the Drupal Migrate API works, how you extract information out of a system.
[00:56:52] Michael Meyers: How you transform it, uh, whether it's the DOM transformation of the content in the body, uh, sorry, yeah, the content in the body of your article to, you know, manipulating dates and things into new formats, then loading it into your, uh, your new data store. We're going to talk about performance, something that's near and dear to our hearts, uh, optimizing and performance tuning and profiling, uh, migrations, including long running migrations, um, where it becomes a critical factor.
[00:57:22] Michael Meyers: Um, and we're also going to talk more about the incremental migration approaches that we talked about here today. Um, and then, uh, also, you know, the idea of. Porting in custom code, um, Drupal 7 to Drupal 10 and give you some ideas on, you know, how to do that, some of the tooling out there, some tips and tricks, uh, a lot more talks along that line coming down the line.
[00:57:44] Michael Meyers: So a huge thank you to Janez and Lucas for joining us today. Uh, please make sure you check out the other talks in this segment. all the links we mentioned today are going to be posted with the talk in the talk notes.
[00:57:58] Michael Meyers: If you liked this talk, please remember to upvote, subscribe, and share it out. You can check out our past Tag1 team talks at tag1. com./ttt. That's three T's for Tag1 talks. As always, we'd love your feedback and topic suggestions. If there's something that you want us to cover about migrations, please let us know.
[00:58:18] Michael Meyers: You can email us at ttt@tag1.com and big thank you to Janez and Lucas, Benji for his presentation and a huge thank you to everyone who tuned in and joined us today. Take care.