Robot Framework is an open source framework for automation used for testing and robotic process automation. This episode's guest, Markus Stahl, has been using Robot Framework with Camunda 7 and Python, using the external task worker pattern.
In this conversation, Markus and Niall Deehan discuss Robot Framework, as well as some of the key differences between Camunda 8 and Camunda 7.
As mentioned in the episode, you can check out Markus and Asko on Twitch.
Josh Wulf: Welcome to the Camunda Nation Podcast. My name is Josh Wulf. I'm a Developer Advocate at Camunda and your host for this episode. In this episode, my colleague, Niall Deehan talks with Markus Stahl about robot framework and open source automation framework used for testing and for robotic process [00:00:30] automation, aka RPA. Markus has been using robot framework with Camunda 7, but doing it in such a way that it translates easily to Camunda Platform 8, using the External Task worker pattern, whether you're a Python programmer, interested in RPA, or wanting to learn more about Camunda Platform 8, this conversation is for you. Now, one thing to note, we've recorded this episode before the official launch of Camunda Platform 8. So when Niall and Marcus talk about [00:01:00] Camunda Cloud, they are referring to the product formally known as Camunda Cloud, and now known as Camunda Platform 8. So with that in mind, let's listen in.
Niall Deehan: Hello, I'm Niall, and in this episode of the podcast we're going to be discussing robot framework, and how it integrates with Camunda. Robot framework is an open source RPA tool written in Python. It's often used for testing front end applications, but more recently it's been used for creating bots that interact with other less accessible systems without APIs and [00:01:30] things. I'm not alone, of course, I'm joined with an expert in the topic, which is Markus. Would you like to introduce yourself?
Markus Stahl: Yes, of course. Yeah. My name is Markus Stahl, and I'm a tech evangelist at [inaudible 00:01:42] Systems, where I take care of developers and bring them together to collaborate. And my deal with robot framework that I, once upon a time, was a Java developer and was working with testers together and realized they have a lot of manual work to do and thought maybe [00:02:00] it can help them. And that's some ideas how I would assist them with Java tools. And although it made things easier, it was not quite the right thing for testers because they were not developers. And they were hesitating using Java tools and had a little bit of distance to it. Then one day I came across the robot framework and thought, okay, this is it. I don't need to stay in Java for helping my testing colleagues, but robot framework [00:02:30] actually does the job for me. It hides all the complexity from the programming language and makes it really easy for testers to program without realizing your programming.
Niall Deehan: Which is perfect. That's actually kind of part of the first topic I want to bring up with you because one of the things that I learned from joining the robot framework was the different ways robot framework can solve different problems. So you're discussing there the first problem you came with this [00:03:00] with robot framework, which was this testing thing. Can you explain a little more details to sort of the... How robot framework solved that problem and the kind of benefits that you would've seen immediately from it?
Markus Stahl: Yes. So robot framework is actually not an RPA tool. It's really only a framework. It hides the complexity of Python. So as you may know, everything on this planet is written in Python or has some Python API, or Python module written for it. So if [00:03:30] you want to automate anything, then there's some Python module out there. The problem is that you have to know how Python works. So you have this benefit that if you want to test something, you want to access another system, you have a developer ready, API or developer ready module that you can use, but you're not a programmer. You are a business person, you know all the use cases that you want to cover. And then robot framework is a layer in between the programming language and the business layer. So you can describe [00:04:00] your business case in your own words, and then similar, like JB does it in Java, it translate the keywords into Python code, but you don't... Usually you don't see the Python code.
Niall Deehan: Yeah, I've actually, that's one of the things I really liked about robot framework was I, of course, am not a Python developer, very much Java background. And I have used a bit of a robot framework for some random little projects [00:04:30] and it's very readable. That seems to be a very key point of robot framework is to be able to remove the barrier of requiring understanding of the programming language itself and be able to not just write those things, but also read them really easily.
Markus Stahl: Yes, exactly. So when I started with using robot framework also for RPA, I was in contact with consultants from Robocorp [00:05:00] and showed them a few examples of mine. And they were surprised that I used... We have in German those weird letters that nobody else has and...
Niall Deehan: I know them really well. Yeah.
Markus Stahl: ... And you're capable of using them. And you can even use special characters that you usually save for your passwords or something. So hashtags and quotation marks. So they can use them in your test case description. So you can program, [00:05:30] in quotation marks, program your use case with your own words and your own special letters and so on. And they were surprised that yes, of course you can do this. And I think I even saw somewhere on the internet use cases written in, I don't know what it was, Mandarin or something, or no Latin letters at all. This is amazing that you can, wherever you are in the world, I think if you have characters that are in UTF 8 or something, you can describe it in robot framework.
Niall Deehan: Yeah. That's a really [00:06:00] good point. I remember when we first did DMN, like when we first built the DMN engine, and I remember I went to Denmark to a government institution there to try and help them build some rules. And one of the main requirements was it had to handle Danish characters as well. And it was, and again, DMN is the same idea because DMN is intended to be, which is a rules engine, tend to be implemented and maintained by end users. So you don't have the same... You can't give the same restrictions [00:06:30] like you could to a developer and just tell them, sorry, this is the characters that you're allowed to use. In the same way you need to say, okay, well, if we want this really to be readable and really to be usable by regular human beings, then we actually need to take into account that weird Danish letters exist and also weird German letters exist. And if you don't do that, you're really not catering for the majority of use cases.
Speaking of those use cases, we're talking actually a lot about the use case of testing. Robot framework [00:07:00] came to my knowledge, whenever people started to use with Camunda in combination with RPA tools. And you mentioned there's not a... Robot framework's not an RPA tool, which is absolutely correct, but because the framework can essentially do what an RPA bot can do, and because it's open source, it's found its way in there. Can you talk a little more about robot framework as it has entered the RPA market in the way it has?
Markus Stahl: So I [00:07:30] think it for 10 years or something, a robot framework made it easy for testers to become test automation engineers. I joined the testing team and the testers they were really comfortable in automating test cases. And at one point I went to this conference, you mentioned Robocon, and there people are talking about that people abused kind of test cases for buying concert tickets or something, which is not really a test case. You write a test case, but the test case is [00:08:00] buy me the YouTube concert tickets that are sold out within two minutes or something. I think the CEO from Robocorp, he went on stage and said, everyone who can automate test cases can do RPA and I thought, yes, of course, of course you can because what's the difference. I mean, there are a few differences.
Niall Deehan: Just to interrupt you very slightly there, to introduce Robocorp into the conversation, do you want to quickly give a one or two line description of Robocorp as opposed [00:08:30] to robot framework? They very often confuse things.
Markus Stahl: Yes. They very often confused. The difference is maybe... So robot framework is like the language, you can consider robot framework as a programming language as it is only a framework. It's not a company behind robot framework. There's a robot framework foundation, it's funded. There's no company behind it. Robocorp is a company that does a business out of the capabilities and offers a platform and tools that you can use to [00:09:00] utilize robot framework for process automation or robotic process automation and task automation.
Niall Deehan: And so, and you were saying that it was the point where Robocorp kind of said and sort of said this thing, Hey, if you can use this for automating test cases, you know how to use RPA bots, right?
Markus Stahl: Yes. I think when they said it, it was like half a year before they actually went public. So the difference between a test case and a task or process is that when a test case fades, [00:09:30] you want the complete process to end and just want to have a report and look at and say, oh, yes, okay. I have to change the test data. Okay. I found the bug. You don't want to have this in a process. So, in BPMN, you have those BPMN errors and you have incidents and retry and stuff. So you have some mechanisms, you want to save the process because the real world process is more complicated than the test case. And you have to take into account eventualities. So if something is slightly different than [00:10:00] you expect, you still want the process and the task to go through and to pass.
Niall Deehan: Hmm. And so we started talking about robot framework, and then we moved on to Robocorp. We just kind of built on top of that. And now we've mentioned processes for the first time, because the other aspect of being able to integrate robot framework, as you said, outside of the test case scenario, is when it comes to processes. So a lot of people started using robot framework as [00:10:30] an alternative to other RPA tools as our Robocorp in particular, as opposed to RPA tools, because it's way easier to use as part of a process. So before we talk about Camunda in particular and how that works with Robocorp, do you want to just give me some idea about the kind of reasons why let's say processes and Robocorp framework has tended to go together rather than for [00:11:00] instance, some of the other bigger RPA tools?
Markus Stahl: I believe one of the key potentials is that you have a much broader potential community and much broader group of users and developers who can do RPA. The classic RPA tools, in my opinion, you need a training for this. So they start with a workshop and say, okay, everyone can automate. And then they start, everyone [00:11:30] starts automating. And then management says, okay, we go with it and we do RPA now. And then the next thing you see is that they ask for a lot of RPA developers and looking for RPA develops that are particularly specialized in this one product. So you have really rare or small amount of people that you try to hire. With robot framework, you are much broader. You can hire basically anyone who is not afraid of any program languages because [00:12:00] you don't need to be an expert in Python to start with, and you don't even need to be an expert in programming to start with automated.
Niall Deehan: Yeah.
Markus Stahl: And then, because there's Python underneath, everything starts up, in my experience, much faster than the solutions that I've seen with the other products.
Niall Deehan: Yeah. I agree. I've never actually built anything for production. Well, I've done many years now, but like I never attempted to with any the RPA things, but I spent a good few months exploring the RPA [00:12:30] community and trying to get started with stuff. And for me, just as someone getting started, one of the things I noticed was the lack of proprietary stuff cluttering up your development workflow with RPA, with a robot framework. As you said, robot framework is you open up an ID that you want that can deal with Python, you install Python, and then you're basically good to go. Then everything else basically works. I know from using [00:13:00] UI path and Automation Anywhere that it took me days to just get a development environment set up, especially if you wanted to use their cloud thing, which is quite complicated.
Now, notably, I was coming from a position of being, I'm just trying to explore this and see the potential, but I was able to get started so much faster as a developer myself, who already has all of this development stuff installed in my laptop, it was so much easier to get started with robot framework. And that's kind of why [00:13:30] I started to see that is the better option for RPA integration with Camunda. And actually speaking of that with RPA integration with Camunda, there's a lot of people who see at the added benefit of an RPA bot and a process engine like Camunda, you've got experiences already having been somebody who put this sort of stuff to the test. Can you talk a little bit about what kind of benefits or what kind of experience that you had whenever you combined [00:14:00] these tools?
Markus Stahl: So we, at one point we realized in the testing team that we were, that our test cases were actually business processes only with test data. So one day we thought, okay, we use production data and provide it to our test cases. And suddenly we had an automated business process. We thought, okay, from now on we go out and ask colleagues, do you have business process we can help you with? And we got processes and we automated them in task. And then we had [00:14:30] to schedule the task somehow. So we used what we had. So we used Jenkins as the ICD environment. And then you have like a few dozen Jenkins shops, and you cannot see in which order they are running, which job is depending on who and which jobs are running in parallel, which one, which job is waiting for the other. So after a few processes, it got really messy and we thought, okay, we have to stop here. We don't automate more because it's getting harder to maintain.
And we need the [00:15:00] visualization. And then of course, Camunda came into our view and then we made an experiment and it actually had much more benefits than only the visualization. Because you have this workflow engine underneath, underneath the visualization, and you have the BPNM and suddenly you can move the exception handling up to the workflow engine. You visualize the exception handling and you're done. You can concentrate in your task on the actual logic [00:15:30] of executing the task and leave the rest to the engine. It really increased the development, efficiency and it made things so much easier. When we migrated then actually the Jenkins jobs to Camunda, so we transformed the jobs into external tasks and...
Niall Deehan: Oh right.
Markus Stahl: ...Run them. And suddenly we could even wipe out single tasks. They were just taking care of making, getting exceptions or something or starting process [00:16:00] or something. And we had actually less jobs and then task after we move to Camunda because there are things that BPMN just does for you and you don't have to invent the wheel again.
Niall Deehan: Yeah. That's a really good point. I think in the beginning with Camunda, people would've... We came across loads. We used to be replacing home built process engines and we used to always tell people like, why would you bother building one yourself? You don't build your own databases, that the process engine was perfectly fine in it's based [00:16:30] and standards and its open source and things. One of the things you mentioned there that I think is kind of important to bring up is the external task pattern. Because without that, I don't think will be way possible to integrate those. And just to explain from the Camunda perspective how that works, and then we can talk a little bit about how we're able to, how important it is for the integration is that Camunda, of course, in the engine in particular is a Java application. It's a really small little [00:17:00] jar about four megabytes that you can embed into stuff and which is great.
So that means you can talk to it with Java and you can orchestrate Java code with it. And that's actually how it started, the Camunda engine. And then I think a couple of years ago, many years ago now, and we decided to look into releasing a feature that would allow us to abstract people's business logic away from the engine. And we were still thinking in Java terms at that stage as well. And one of the things [00:17:30] that happened was the external task pattern then became essentially a way to solve... Thinking back, it sounds crazy, but it was solving the problem of long running services. That was the problem we had. Because when we were... One of the... Camunda would have worked great, but when we had some Java that ran for a really long time, because it runs on the engines thread, it would time out and crash. Right?
So the problem we were trying to solve external tasks originally was like, how can we abstract a Java [00:18:00] class or service so that it can run on its own thread or its own thread pool for as long as it needs to? And then it can tell us afterwards. And an example would've been video processing where people had like this like process and a part of it was rendering a video that took like five hours. There was the engine basically couldn't deal with that. It was way too long. So external task pattern originally was designed to solve that problem, which is crazy because that has completely disappeared as a use case at this point, because [00:18:30] the side effect was that we created a rest API that any language could use to be able to get work from the engine and then in its own environment, be able to run stuff.
And not only tell the engine when stuff was completed, but as you said, the error handling was super. You could then say, Hey, we attempted to do a thing and we failed. And not only that we failed, but we failed in a way that we think if you retry in like 10 minutes, it'll probably be okay. So you can tell the engine [00:19:00] all of this, you can even tell the engine, Hey, this is a BPMN error. So the process continues, just a different route. So we ended up finding this way of decoupling the business logic from the process while keeping a huge amount of referencing. And then that's when we ended up getting communities like the C sharp and Python community jumping on board and being able to say, oh, we can generate clients really easily. And now we have come under running on RPA. Now we have come under running with using robot framework. [00:19:30] So external tasks are core to how we bumped into this whole community, ironically, for trying to make long running Java processes run. And that's where robot framework actually connects.
So if people are wondering how exactly does Camunda run a Python robot framework thing, it comes back to this external task pattern, which I'm sure you're very well aware of.
Markus Stahl: Yes. It's so funny that you, basically you created Camunda platform by accident.
Niall Deehan: [00:20:00] We kind of did. I mean, we definitely... Here's the thing that is actually the most interesting is in the early days there was so few of us, I think when I joined, there was 20 people in Camunda, now there's like maybe 300 and something. So I don't even know how many. But I remember really clearly that we put out this external task thing and we were so focused on the community reaction. And we always had these discussions, how are people using it? Why are people using it? And actually it was [00:20:30] the C Sharp community that first showed up out of nowhere, because we weren't really trying to talk. We didn't even discuss how C Sharp could be integrated, but for whatever reason, C Sharp people started showing up and started saying to us, oh, this is great. And we internally then thought, oh this actually we need to pivot this use case that's actually used for calling poly... Orchestrating polyglot architectures.
And then we ended up discussing well, okay, let's [00:21:00] think about the next couple of years of this product if that's the use case. And that's kind of how we built it. It was entirely by listening to the community about how they were really using a product in a way that we did not intend for them to use initially. So it, which is very funny, because it mirrors robot framework really closely, which was originally a testing tool that found people using it for something completely different. And now we have this scenario where we have, again, community [00:21:30] focused stuff. We need to listen to how people use the software and then not just listen to it, but act on it internally. Then you end up getting a really cool thing.
Markus Stahl: Yeah. And really it tells a lot about the, about all you community managers and about the company, because you listen to the community, you became aware of how you, your users use your product and abuse it in a, in a way and create new opportunities for you. Right?
Niall Deehan: Yeah.
Markus Stahl: Today I started with a Camunda platform and [00:22:00] because I was so late into all the Camunda stuff, now the external aspect for me, like the logical thing to use because then you view Camunda and the workflow engine as a service.
Niall Deehan: Yes.
Markus Stahl: And I cannot imagine why I would like to embed this in my application. It's so comfortable to have at the service that I have in my view have monitoring on it.
Niall Deehan: Yeah. You're right, actually. It's probably one of the reasons why Camunda Cloud exists is because once people realize that they could treat an orchestrator as a service, [00:22:30] the next logical step was, well make that service cloud available. So now we have Camunda Cloud. But then we needed to think, well, if we're building a new product and it's cloud based, why don't we just rebuild the engine to be cloud native? So that's actually why we build ZB. So one of the things that's kind of interesting about the way Camunda platform works is it uses this really great engine, which is honestly solidly it's a really, really good [00:23:00] piece of technology, but it has a few flaws based on the time it was built. Now I wouldn't say flaws. I would say it has limitations because most people would never even come across these in most use, which is, it has a relational database.
And that means that it has a certain amount of connections it can make. It has a certain amount of speed of transaction based on committing to this database. And this is, and also scalability. You can only really scale [00:23:30] as big as your database will allow you. So, and again, if we were thinking about this as a service, what we want is a completely elastic service that can say for very small use cases, we just need a few little nodes and it's fine. But as it gets bigger, we want to be able to scale it infinitely. And so we designed then ZB based on the idea of what if we just took away the whole idea of transactions. And we took away the whole idea of a database in the back end and just say, let's say, well, we can, how [00:24:00] fast can we make this go?
Then we made something very cloud native. You just stick it in the cloud. You don't need any other sort of systems around it. It's self-healing. It has a self perpetuum clustering mechanism internally. It's really, really good. And it's perfect for cloud stuff. So that's where that came from. And to your point, it's still, if you look on it from the outside, exactly the same as Camunda platform, because you just use external tasks in exactly the same way. It's a little more efficient in fact. It's built explicitly [00:24:30] for external task pattern. As you said, why would you want to embed the engine? It's a lot of work. But I see that the RPA and robot framework stuff is going to be very easily transitioned to Camunda Cloud in the future. Do you see any benefits in sort of this sort of move yourself, knowing what you do right now about Camunda Cloud?
Markus Stahl: About Camunda Cloud. I haven't used Camunda Cloud. [00:25:00] I want to, so I have already created repository bot and that's called I think robot framework ZB, because I want to have a library in an API for robot framework accessing ZB and Camunda Cloud. But I think back when I did this, the technical terms for process and stuff, they were different from the Camunda platform and the, I don't know the GRPC protocol. So I thought, okay, this is a lot of work. And [00:25:30] my couch time is really, really rare at the moment.
Niall Deehan: I think at the moment, I think we're releasing a open API, rest API thing so you can auto generate clients for Camunda Cloud. That's probably coming pretty soon. And I think as soon as that happens, we're going to see the same boost in people who want to build these random workers and random languages showing up. GRPC is way more efficient for [00:26:00] stuff, but it's also not really very well used. So I think giving people the option to use either a rest called just like that or just building GRPC clients, cause that's one of the things you can do as well. GRPC makes it a little easier to just build a Python client just by pointing at here is the GRPC thing, but people still love rest. And we also actually added a very interesting thing recently, which is for UI. So right now with UI and Camunda platform, it uses [00:26:30] the rest API of course, right? Which for something as generic as Camunda can be a bit of an issue because you'll end up getting way more data than you need when you query.
Also, we found this all the time, when you're querying from a front end, you maybe want a task, specific variables associated with that task. And this is a whole bunch of stuff that happens across different tables that usually makes in Camunda platform about three or four calls required to generate a single thing for a user. So we started investing in GraphQL [00:27:00] as an API for Camunda Cloud, which means you can query and say, I want incredibly specific data associated with this user task. And then all of that joining is done on the server side. And then you get back exactly what you've asked for rather than either too much data or having to query more than one time for a thing. So that's, we're looking at all the different things that people complained about for the last like seven years about Camunda [00:27:30] and we're trying to already look into fixing them before they become another reason to complain on the forums to me.
Markus Stahl: I have really high hopes in this ZB client that Nela and Josh tried because if this works, then it's really easy to wrap a robot framework cloud around that client. So really looking forward to the next Twitch session.
Niall Deehan: Yeah. And for those listening who are interested, you can always, [00:28:00] it's a joy to behold Josh and Nela, two people from the Camunda team, trying their best to fight their way through a language they don't fully understand. I really enjoy it. It's a really fun learning experience as well. But in the end they're trying to build something worthwhile as well. So that's actually a pretty fun thing. And of course, Markus, yourself and Osco also do Twitch streams as well. If you want to plug that.
Markus Stahl: Yes, absolutely. So Osco [00:28:30] is a software architect from the University of [inaudible 00:28:33] in Finland and he and I, we had around the same time, the idea of about combining robot framework and Camunda, but we had different approaches. I was coming from the tester and robot framework side. So I was thinking about robot framework, developers and users. And I think Osco was coming from the Camunda side. So I implemented a library for robot framework users to access Camunda, to try Camunda. Okay, I want to start this process. [00:29:00] Okay, well now I want to have a process instance. Osco, on the other hand, he built an external task client and that does all the stuff and the robot framework implementation doesn't know that it's working with Camunda at all. It just receives values and processes.
Niall Deehan: Runs. All right. Okay.
Markus Stahl: And so the use case is a different, so I think my Camunda library it's for quick stuff. It's really good. And it's also works [00:29:30] when you have more processes. So that's okay. What I think will happen in the future is if the setup that Osco is doing, that this is actually the way to go, that you have really flexible robot framework task, and that you can run anywhere. If you can run it in the robot cloud from Robocorp or on your own machine, in your own cloud, wherever, and you have those clients from Osco that takes care of sending. If a test case or a task fails in robot, [00:30:00] making a BPMN error out of it or making an incident out of. So you don't actually realize in your robot task that you're dealing with BPMN.
Niall Deehan: Yeah. That's ideal.
Markus Stahl: So I think that's the future and the Camunda library, I now saw use case where a company used it for testing Camunda actually. So because you have access to the rest API, you can test if you deployment of processes worked. If you run some tests on your process, check if the variables really are set right in [00:30:30] the process flow. So this is where reward favor comes back to the roots again and uses the Camunda library that I implemented. So I think this is probably the, in my opinion, what will happen in the next year. What's probably the best potential.
Niall Deehan: That sounds like fun. It sounds like a lot of potential for both frameworks.
Markus Stahl: And you wanted me to drop this in so I just wanted to say that Osco he's the software architect in Finland, in [inaudible 00:30:57] and he's this year he's on paternity [00:31:00] leave. So he uses sometimes his lunch break on a Friday to teach me his Camunda client for robot framework. And he streamed this on Twitch so I think now two or three weeks in a row, and I don't know where this ends, because this is so much fun to do.
Niall Deehan: Yeah. And I'm watching myself. It is a lot of fun watching Osco talk through just everything he's built. And I kind of like as well that he has a certain amount of eccentricities with how he codes stuff, that he likes things [00:31:30] a certain way. And I kind of love that. It's a really fun thing to watch. So yeah, we'll probably, we'll try to add a link to that in the description.
Markus Stahl: Awesome.
Niall Deehan: So, yeah. And with that, I think I shall leave you there. Markus, thank you very, very much for joining. It was a pleasure to chat to you again, and I'm sure we will again soon.
Markus Stahl: Thank you, Niall.
Niall Deehan: Yeah, no problem. For everyone else, thank you very much for listening. And hopefully if you have any questions about robot framework or about Camunda you can visit the forum. If you're interested in events that are coming up, we have a Camunda.com\events page. [00:32:00] And if you have any questions, myself and Nele have a fun little question corner at most, once a month on end of a Thursday. Again, that'll be on the events page. There's lots of ways of getting in contact with us. So I'll see you then. Bye. Bye.
Markus Stahl: Bye bye. Roll the credits.
Josh Wulf: Credits. We got credits. First of all, shout outs to Niall and Markus for that conversation. As mentioned, Asco and Markus stream live on Twitch. Check the show notes for the link [00:32:30] and like, comment, subscribe to these guys. Props to Inclusion Audio for the editing and to the legendary Shingo Nakamura for keeping a progressive house. Keep automating all the things and stay safe out there. Seacrest out.