Migrating to Camunda Platform 8, Part 2: Data - “It’s a very exciting journey”

When planning a migration to Camunda Platform 8, there are three main aspects to consider: model migration, business logic migration, and data migration. In the first of our two-part series, we covered migrating models and business logic (Catch up on Part 1 here). Now, we are exploring the final pillar–data migration–as well as the benefits of migrating to Camunda 8. 

Listen as Senior Developer Advocate Niall Deehan,  Senior Consultant Manuel (aka Manu) Dittmar, and Consultant Jonathan Lukas dive into data migration, sharing tips for what to do when you have a lot of historical data or long-running processes. They then explore their favorite Camunda 8 features and the benefits of migration. 

Listen to learn more about: 

• How Camunda 8 stores history variables
• What to do if you have a terabyte of historical data
• If and when you should migrate runtime data for long-running processes
• The benefits of migrating to Camunda 8 including Connectors and SaaS
• The opportunities that come with migration

Share your questions or feedback about this episode on our forum.
Start your free trial of Camunda Platform 8.
Save your seat for CamundaCon 2023.

Additional Resources

• Camunda Platform 7 To Camunda Platform 8 Migration Tooling
• [guide] Migrate to Camunda Platform 8 in 6 Steps
• [docs] Migrating to Camunda 8 from Camunda 7
• [blog] What to do When You Can’t Quickly Migrate to Camunda 8
• [podcast] Migrating to Camunda Platform 8, Part 1: Process Models + Business Logic - “You can migrate any kind of implementation”

---

Visit our website.
Connect with us on LinkedIn, Facebook, Mastodon, Threads, and Bluesky.
Check out our videos on YouTube.
Tweet with us.

---

Camunda enables organizations to orchestrate processes across people, systems, and devices to continuously overcome complexity and increase efficiency. With Camunda, business users and developers collaborate using BPMN to model end-to-end processes and run sophisticated automation with the speed, scale, and resilience required to stay competitive. Hundreds of enterprises such as Atlassian, ING, and Vodafone design, orchestrate, and improve business-critical processes with Camunda to accelerate digital transformation.

---

Camunda presents this podcast for informational and entertainment purposes only and does not wish or intend to provide any legal, technical, or any other advice or services to the listeners of this podcast. Please see here for the full disclaimer.

Transcript

[NIALL DEEHAN] Originally separating history from runtime was, again a way of letting the engine be more efficient. It’s nice we’ve come to the, essentially, the final stage of that idea.


00:12 - Welcome + Recap of Part 1
[MIA MOORE]  Welcome to the Camunda Community Podcast. I’m Mia Moore, Senior Technical Community Builder and podcast contributor here at Camunda, and I’m covering for Niall while he’s on his summer break. Enjoy that vacation, Niall! You earned it. 

On our last episode which is linked in the show notes in case you’d like to catch up on the conversation, Camunda consultants Manu, aka Manuel, Dittmar and Jonathan Lukas joined Senior Developer Advocate Niall Deehan for a closer look at migrating to Camunda Platform 8 from Camunda 7.

Specifically, they discussed migrating models and business logic–what to look for, steps to take now for the smoothest migration possible, and the tooling that is available to help with your migration projects.

Quick reminder: Manu (aka Manuel) leads the migrations effort for our Consulting team by defining the overarching strategy and being directly involved in a lot of migration projects while Jonathan works on the tooling that helps our customers and community when it comes to actively doing the migration. 

When we last left them, Niall, Jonathan, and Manu (aka Manuel) were about to dive into the third pillar of migration: data. I know I can’t wait any longer to hear more about this topic so let’s hop back in! 

—

01:12 - Data Migration: History Variables

[NIALL] Jonathan, has jumped into a very important change between C7 and C8 courses around variables and data. And that brings us very nicely to the final section of migration, which is data. 

I would like to even create two subcategories here and say, running data and history data probably can be treated very differently.

Let's start with variables in general. Manu (aka Manuel), can you just give a quick overview of the considerations when it comes to how C7 stored, let's say, history variables versus how if you wanted to keep those the way they are and continue using them for reporting purposes, for example?

[MANUEL AKA MANU DITTMAR] In Camunda 7, it was always pretty easy to also access historical data, but just using the history service. In Camunda 8, that looks slightly different, right? Because Zeebe is really just taking care of the runtime data and not the historical data. That means, if you want to get this kind of historical information, you would need to use Operate and the Operate API for that. So here we have a clear separation.

So this is probably very important to consider when you're working somehow with the history service of Camunda 7 right now, that this cannot be translated one by one to the Zeebe context. But still you can get the very same information from the Operate API.

[NIALL] Exactly. So, as a quick reminder, C7 had two different schemas within the database. One was the runtime data, which was the only one the engine would read from. And one was the history data which contained all of the historical processes and the currently running ones. And that's where most of the, let's say, client queries would go through.

A big difference in how data is maintained by Zeebe, of course, is how it both stores and exports data in general, and how you access it. Manuel (aka Manu), do you want to give us a quick overview of how data of a process sort of exists lifecycle-wise from the moment a process is started to maybe by the time it's finished and shows up in Optimize?

[MANUEL AKA MANU] First of all, when you start a process instance, that one will be basically created within Zeebe. Zeebe internally uses the key value stored for that, to store the information about a process instance. And basically, all those kinds of events that are being sent to Zeebe will also be exported to Elasticsearch in the next step.

So that could be a process instance being created, a variable being set, a user task being completed. So everything that you can imagine, any kind of event that happens in the process instance– that will be exported to Elasticsearch, and then basically all the web applications will just consume the data from Elasticsearch.

So no matter if it's Operate, Tasklist, Optimize–this data will come from Elasticsearch.

The nice thing is that those web applications do not need to contact Zeebe for that anymore. And Zeebe can really focus on what it's good at, and that’s about handling running process instances.

Basically, as soon as those web applications basically consume the information in the way they need it from Elasticsearch, it will be presented to the end user’s web applications.

[NIALL]  And that's really a continuation from the ideas from C7. Originally separating history from runtime, was in again a way of letting the engine be more efficient. And it's nice we've come to the essentially final stage of that idea by completely removing the responsibility of history data from the engine for any means, which I think is a really good choice.

04:41: What to Do If You Have a Terabyte of C7 Process Data

[NIALL] With history data, Jonathan, what do we suggest people do when, if they have, let's say, a terabyte of C7 process data, burrowed away somewhere in a very slow relational database? What are our suggestions there for people who want to migrate then to C8?

[JONATHAN LUKAS] First of all, I would always just to review the existing data and decide whether it's still needed, or maybe some parts are irrelevant. So given there's 1 TB, that's a lot of history, and you can probably decide to throw away most of it. And one thing that we can do, or that I would always look into regarding securing historical data is the usage of Optimize.

Because Optimize has an engine-agnostic data model. But you can import process instances from C7 and from C8 and they will be stored in a very, very similar way,  the same way, and can be accessed from within Optimize and always, always with a view on on the process, of course.

That's also the only thing we can do about historical data from C7. So it cannot be preserved just like that in C8.

But I think that's a very good option that we have– using Optimize for this, because that's what Optimize is meant for–. It's for analyzing historical process instance data or decision instance data. And it's made for this. It's optimized for this and that's why you should always use it for this.

Given someone has 1 TB of historical data, you might also face situations where this data cannot be loaded from Camunda 7 anymore, where any connection to the REST API just times out because the database query is so slow. The data is basically unusable. It’s there. So it still exists that might go through compliance, but it's unusable.

By using Optimize. you can make it accessible again.

[NIALL] Hmm, that makes a lot of sense. So in most cases, people don't really need process history data, actually. It's it's as you mentioned already, Jonathan, compliance is one of the main reasons. And I think we can quite happily say that that's not a difficult thing for us to be able to maintain that data in an accessible way for compliance reasons.

And reporting is the other reason people would have it. And again, as you mentioned, Optimize is the great solution for that, because it can just take that data and give it to you in a very accessible way.

And history data–of the 2 types, runtime data and history data–history data is probably the easier one to deal with.

07:28: Migrating Runtime Data for Long-Running Processes

[NIALL] Now we have 2 types of, I would say, 2 categories of process. We have incredibly fast processes that maybe take milliseconds or a couple of seconds to run. Or we also have, because of the way BPMN offers a lot of flexibility of processes, processes that can last for months or years, or weeks, or whatever.

So with regard to the sheer difference in that and runtime data, Manuel (aka Manu), can you maybe describe some of the things people should consider looking at when it comes to considering data migration of runtime data and their processes?

[MANUEL AKA MANU] Yeah. So general recommendation would be: don't do it. So don't, don't try to migrate running instances from Camunda 7 to Camunda 8.

So here the idea is that you really have this parallel run pattern that basically, you can slowly phase out Camunda 7 while you start new processes in Camunda 8. And as soon as you don't have any running instances anymore on Camunda 7, you can also shut it down. But basically let the instance die where it actually started, instead of trying to migrate it to a new platform. 

Of course, there are some cases, as you said, where instances are running for multiple months or years, you could maybe think about trying to migrate that. So that you basically recreate the very same state of that process instance in Camunda 8. 

But yeah, with that one, I would be very careful, because you also need to make sure that during this migration, you don't receive any messages, or the user task will not be completed and anything like that. Otherwise, we have some inconsistency. So I would say that should be the last resort.

[NIALL] Yeah. Jonathan, you would agree, I think, with this. And can you maybe describe if people are forced? Let's say the migration is, they've decided it has to happen for some of the runtime data. What's the practical steps that people could take to make that happen?

[JONATHAN] So one possibility is to use our tooling. Here again, we have process instance, migration tooling. It's currently in a very early stage of development. I plan to develop it a bit further.

But what basically happens is that the process instances that need to be migrated are suspended so that we do not have the migrated in in some kind of dangling state where they are, are already advanced on C7, and they are started at an earlier point in C8, and they are probably things executed twice.

And then we extract all the process instance data. So what activities are currently active, what variables are set on which scope. And then we try to leverage the Zeebe API to bring the process instance exactly to this state again. 

Here, of course, we face some limitations that we should always review. So, for example, we cannot migrate call activities just like that, because call activity would always mean I start the root process instance, and then I have to make Zeebe execute the call activity, so that there's a new process instance spawned.

Then I have to get the ID of this process instance to actually modify it to the point where it is right now, and so on. So that, that's a very complex thing to carry out. 

And also, there are message start events, for example, that would need to be considered, just because message start events cannot be executed by the create process instance command in Zeebe. But you have to correlate a message. 

And there are some other limitations that are currently known, that are also documented. This process instance migration approach, which would, which would automatically do it.

What you could, of course, do, and I think we already did this once or twice with customers is writing your custom migration scripts, or doing a migration by hand, where you review a process instance, and then you make up a plan. And you perform the migration by hand, which is, of course, possible, but I would always agree with Manu (aka Manuel), saying that avoid it, by any chance, to migrate process instances that are already running. Just because you will, I think, you will lose context, you will lose data. It's just hard, a hard thing to do. 

[NIALL] It's a lot of work for not a huge amount of gain really, at the end of the day. I, I would agree. But I also like that it's still possible. I like the idea that even though we're saying, “Please don't do this.” Jonathan, of course, is going to spend some long-suffering days trying to make that tool work better.

If you want to join him, of course, you can also, take a look at the the migration tooling, which is linked in the show notes. And you can maybe even hey, who knows? Check out some of the issues he's raised, maybe help him by building some additional features in there.

12:15 - Benefits of Migrating to Camunda 8

[NIALL] I want to close out by with just some interesting thoughts you might have on something that we never really touched on. But it's worth discussing, which is: What are the main benefits of people migrating? Why do you think people will feel that all this effort is worth it by the end? 

I want to start with you, Manu (aka Manuel). Top your head. What are your feelings around like–this is beyond the fact that C7 is going out of support in a couple of years–What are the benefits we are going to see from using C8 over C7?

[MANUEL AKA MANU] I think, obvious answer for that one is the performance of Zeebe, yeah, because everyone who use Camunda 7 and more complex use cases, I think, face some limitations one day.

So for that, with Zeebe, with the performance, this is a very good reason to migrate.

Then, also I'm a big fan of the Connector architecture that we have in place. because it's not only limited to using the pre-built, out-of-the-box connectors that we provide, but you can actually also leverage those element templates that we provide to create your own ones, use the SDK to create your custom connectors. 

And I think this just helps you to scale up automation within your organization, that you can just share components and just leverage other people in your organization that are probably, that don't have multiple years of experience in enterprise application development on Java, and also making sure that also your Python developer, C# developers are on board.

Those are probably my 2 favorite features. 

[NIALL] I totally agree. Performance and connectors are both–they really open things up a lot for growth of processes within especially larger companies. But also, as you mentioned within a company that, let's say, has a limited number of people who can build processes. Some of the features around C8, like Connectors as you mentioned, allows a much vaster skill set of individuals to be able to contribute to building and maintaining process models.

How about yourself, Jonathan?

[JONATHAN] I think one of the great advantages of Camunda 8 is our service offering in form of SaaS. I think one of the biggest hurdles you have to overcome is operations. So either operating C7 wasn't that hard but it it has, it has its pitfalls still.

So providing a stable environment for running a process engine is still not too easy. So running on SaaS will just take the burden from you, and and make it easier for you, and give you the opportunity to focus on your business logic implementation and the operation of this.

And very closely related to this, is also the security concept of Camunda 8 where your business logic is separated from the process engine, and you will not be able to modify things in the process engine just like that on the run, access some private APIs, because it basically runs inside the same Java machine. 

And these are my additions to this.

[NIALL] I do agree. And that comes back to a point you made a while ago which is the different technologies that can be leveraged with Camunda 7 and Camunda 8 are important, right? You mentioned Kubernetes and Docker and stuff. 

But the other option is: you don't need to learn any of them. You could also just use the SaaS version and you don't actually need to worry about it. 

With Camunda 7, the minimum you would need to know is how to keep a Spring Boot application alive, which is not too much. But maybe if that's your only Java application in your whole company, that could actually be quite irritating if anything goes wrong. 

So as well as like the, as you mentioned, the security we’ve implemented, or the, I would say, more security architecture that now exists within C8.

And as well as that, the ability not to care about it, because it'll always be there running is also really valuable if you were to choose the SaaS version. So yeah, it makes it way more open and easier to use for people.

With that I would like to thank you very much for talking to me for the last little while. I greatly appreciate it. Thanks for taking your time away from customers who desperately need your help migrating, and instead talking to me about it. yeah. Do you have any last thoughts about this topic before we go, Manu (aka Manuel)?


16:40 - Final Thoughts on Migrating to Camunda 8

[MANUEL AKA MANU] It's a very exciting journey of the migration. And also, if our existing customers are having a look into Camunda 8, they probably would just also identify a lot of new use cases, right? Because also, if you have a look at the case studies out there and see how people are using Camunda 8, it can be pretty exciting to see what is possible with that engine now, and maybe also, next to migration, also creating new projects–probably something very interesting for you.

[NIALL] Yeah, I agree. And Jonathan, any last thoughts?

[JONATHAN] I would like to invite you to have constructive discussions with us about features that you are still needing in Camunda 8 because we always try to help you, to make you successful. And that's how we reach it: by communicating with you.

[NIALL]  I completely agree, and I will attach Jonathan's personal mobile number to the show notes. So if you have any thoughts about Camunda, anytime, night or day, he is willing to help you out. He thoroughly enjoys when he gets calls. Not much of a sleeper. So if you want to call 2 a.m., 3 a.m., that's ideal for him, that’s when he's at his peak. So yeah, thanks for that, Jonathan. Greatly appreciate it. 

Good-bye all. See you later.

---

17:54 - Conclusion + CamundaCon 2023

[MIA] Oh, Niall. Needless to say, no personal numbers will be linked in the show notes but you are always welcome to head to our forum with your questions, feedback, and any success stories that you want to share, at any time--our forum will be linked in the show notes. 

And there you have it–the 3 pillars of migration: #1 model migration; #2 business logic migration, and #3 data migration. I know I learned a lot about each one from this conversation, and I hope you did, too.

If you’re inspired to dive right in and use some of the tools that were discussed, head to the GitHub repo that is linked in the show notes. It includes all the tooling we have created, and will also include any updates and new tooling that will be built in the future.

If you’re interested in creating issues on those with your questions or adding your own contributions, this is a great place to start. 

Before we wrap things up, I’d like to invite you–yes, you!–to CamundaCon 2023, happening both in New York City, in-person, and online on September 27-28. Camunda experts all around the world will be gathered in person and online to share what they’ve learned, things they’ve built, problems they’ve solved, and lessons they’ve learned with Camunda. 

I got to go to CamundaCon last year when it was in Berlin and I had a delightful time. It was so wonderful getting to connect with the community and to hear really cool stuff that’s going on with Camunda. 

Head to camundacon.com to learn more, and I hope to see you there.

And to make sure you're among the first to listen to our newest podcast episodes, be sure to subscribe. 

Thanks for listening, and I have not been Niall the whole time–I've been Mia. Until next time!

.
.
.
.
.
.
.
.
.
.
.
.

19:30

[NIALL] Hi, everybody. Welcome to the worst electronic assistant you'll ever meet, who, whatever you ask, will generally suggest is maybe getting a beer or just stop working.