When Camunda Platform 8 launched earlier this year, we announced connectors. This month, I talk with Principal Software Engineer Nico Rehwaldt about connectors–what they are, how they relate to process modeling, and how you can get started.
Nico also shares how connectors are really designed for "the future me" and even makes the bold claim connectors in C8 are reusable behaviors done right, incorporating learnings and feedback from Camunda Platform 7 element templates and keeping developers and their future selves at the center of the design from the very beginning.
- A Technical Sneak Peek in Camunda's Connector Architecture by Bernd Ruecker
- Camunda 8 Connector SDK on GitHub
Josh Wulf (00:00)
Welcome to this month's episode of the Camunda Nation podcast. I'm your host. My name is Josh Wulf. I'm a developer advocate at Camunda. This month is all about the exciting Connectors Framework for Camunda Platform 8.
What are connectors? Simply put: if BPMN symbols are reusable representations of elements in a process, then connectors are reusable integration code to integrate specific IT systems with a model.
Don't worry - we'll unpack this in this episode where I'm speaking with Nico Rehwaldt. Nico, in addition to being the lead for bpmn.io, is also one of engineers who have been working behind the scenes on the Connector Framework for Camunda 8.
The Connectors SDK is now public on GitHub, so you can start playing with it right now, or you might want to listen to our conversation first.
Here it is!
Nico Rehwaldt (01:06)
I am Nico. I'm a principal software engineer at Camunda here. Historically, I've been working a lot in the modeling domain, like building bpmn.io and also the Camunda desktop model. Altogether, it was a strong team.
Nico Rehwaldt (00:38)
But recently, since more than half year, I'm working in the connectors team, also as a principal software engineer, overseeing what we do makes sense end to end. That's basically what I'm doing. I have a little bit of context in connectors and also what we'll see as a modeling as very strong component of our C8 connectors story, so those topics are actually very much related.
Josh Wulf (01:47)
Yeah. We had you on talking about bpmn.io and I had no idea that you were involved with connectors, but it makes perfect sense, right? Because a big part of connectors is the modeling piece of things. Maybe for people who aren't familiar with what are connectors, could you give us the top down view, the 10,000 foot view? What are connectors?
Nico Rehwaldt (02:11)
Well, the basic pitch for connectors is if we look at our C7, Camunda 7 success rate, one of the biggest success factors is that we're extremely friendly for developers, right? We have a very strong framework for develop to automate processes, right?
Nico Rehwaldt (02:30):
But then when we see how... That involves a lot of things. For example, a strong story around testing, right? A strong story about process applications, deployment, deploying this whole thing as a block and then it's actually running. Also providing behavior to the engine, very strong for developers.
Nico Rehwaldt (02:51)
But then when we look at bigger organizations using Camunda, what they usually want to do is they want to allow to automate more things with a little bit less developer capacity.
Nico Rehwaldt (03:05)
For example, instead of always rebuilding a certain integration into Kafka or into another system that they would generally use, just build this integration once and then have a strong story around reusing that integration across the organization. Right?
Nico Rehwaldt (03:23)
I'm building certain integrations in my stack, in my architecture, in my enterprise architecture, and I want to reuse this integration across different processes in my organization.
Nico Rehwaldt (03:36)
This is a high level view of connectors, where we want to make this reuse possible. What I'm always talking about is we are building a framework foundation to allow you to build those domain specific reusable building blocks. It doesn't matter a lot for us what those building blocks are. Some people may interface with external systems.
Nico Rehwaldt (04:00)
We are settled on using Stripe for payment processing, so they will have some dedicated building blocks for talking to Stripe. But then from the person that actually models, it's actually not important if they're talking to Stripe or if they talk to other payment providers at some point what's actually just important is that payment processing happens here. What are the relevant input data? What do I have to feed into payment processing?
Nico Rehwaldt (04:24)
Because this is basically where, from the modeling perspective as well, right, it just becomes... You don't model in service tasks anymore, but you actually model in a payment, process payment task, right? You would configure it accordingly.
Nico Rehwaldt (04:37)
This is basically the very strong story, right? From the modeling side of things, we think in reusable building blocks. You don't think in user task, service task, business role task anymore. But you talk, think in “send a message” task. Maybe you don't even need to know if it's Kafka or whatever, process payment, and you don't need to know what's behind it. This is part one.
Nico Rehwaldt (04:59)
Part two is, what we'll talk about in a little bit, is also providing the framework on the SDK as we call it to conveniently build these building blocks, like in terms of runtime behavior.
Josh Wulf (05:13)
Yeah, I got it. I was just thinking exactly of that example, the process payment one. I got that from Bernd, rather than the technical implementation of charge the credit card, it's process payment. Because it's that it's at that domain. Which domain is it in? Is it in the technical domain? The implementation? No, it's in the business domain of what's actually happening.
Josh Wulf (05:36)
It makes sense to me that if you have a large, very large organization, for example, with many different departments, different processes taking place, rather than duplicating a lot of that work even across teams, even for myself in doing things. If you put Slack into a process once, you immediately start thinking, "Hey, I need a reusable genericized, Slack worker." Then it's configuration driven, custom headers, that kind of thing.
Josh Wulf (06:05)
Then the very next thing I was wishing for was, is there a way I can make that into a template for the modeler so that I don't have to consult the documentation and hand type in the custom header names and get them wrong.
Nico Rehwaldt (06:18)
Yeah. When we talk about who is templates, who is connectors for? It's for the future me. The stuff you're mentioning on Slack, right? I want to build that integration once. Personally, I have a very bad memory. Whatever I build, and I build a lot of stuff, I will forget that. I will forget how to properly configure it, right?
Nico Rehwaldt (06:40)
We want to enable more people to safely actually use that without the need to write a test case using any of the great testing utilities that are out there. Right? We actually want to allow you to safely just model and use it. When you're back in three months, that you forgot what actual are the input variables, right? This is what the element templates or connector templates are part of in this whole story, provide you with.
Josh Wulf (07:07)
Okay. Just to give us some historical context, we ran a sentiment analyzer over the forum, an AI type thing, to find out “what's people's mood” talking about different things. We discovered something very interesting. People had a largely negative mood talking about connectors in Camunda 7. Do you have any thoughts on why that was? Or what's different about connectors in C8 from C7?
Nico Rehwaldt (07:36)
Let's do a bold claim. I think connectors in C8, they do it right. They do it differently. Connectors in C7 was a mean to plug in also a specific run time. For example, we had an HTTP connector at some point, right? You would be able to provide this as an additional behavior to your process application and/or to your engine, and then you would make it available.
Nico Rehwaldt (07:59)
But what was missing in C7, the whole aspect around element template, say reuse during drawing, and your model that was missing. In C7, connectors were always a niche. There was a very specific and special way how to configure them, for example. I've built a properties panel, so I can tell there's a lot of engineering effort on our end involved to basically build this custom, like connector specific editing.
Nico Rehwaldt (08:30)
For C7, it was more like, "Yeah, nice. I can kind of create, plug in my own behavior. But I have to learn a lot of things in addition." Right? Then when people want to use those connectors, they don't actually get the support they need during modeling.
Nico Rehwaldt (08:44)
I guess this was a few of the reasons why connectors in C7 weren't as successful. But I guess the idea was very similar. It was one attempt to go into this reusable behavior. Just what we realized with C8 is we need to tackle connectors from both ends. We need to provide this SDK to build behavior.
Nico Rehwaldt (09:08)
Also, consider what's important. I've mentioned, I didn't mention but stuff like secrets, for example, is very important. If I would talk about reusable connectors, I need to have a way to safely inject secrets. If I want to have a reusable Slack client, I obviously need to safely inject the OAuth token. Same for if I want to talk to a Lambda function. All those external systems are somehow authenticated, so secrets handling is a core story that has also not been there in C7 connectors, I guess.
Nico Rehwaldt (09:35)
But the other important building block is really modeling. How, once I create those connector behaviors, what is the simple and straightforward way to create those reasonable building blocks on the modeling experience? Right? In the web modeler, then make it possible for me to safely discover it, but also use it in two months from now when I forgot about the implementation. I don't care about the implementation anymore. I just want to send the message like I did send the message 15 times ago in the last year, right? Or I just want to process a payment. I don't actually care. I once figured it should be Stripe. I know I have an account, but I don't care on this level anymore.
Nico Rehwaldt (10:17)
And this higher flight height, so to say, right? This gets us closer to the core of BPM, where we want the processes to be meaningful from the business perspective. What is actually happening here? Well, the payment is processed. We don't necessarily want to capture all the technical details.
Nico Rehwaldt (10:38)
I know there's different ways how you can play Camunda. If you, for example, use Camunda upon microservice orchestration, maybe you're a little bit more technical actually orchestrating those different systems, right? But when you're really from the business process perspective, you just care about you want to process a payment, as you mentioned, right? You don't want to be bothered about the details.
Nico Rehwaldt (10:57)
C8 gives you the framework. You can easily compose element templates, either in a technical, depending like... You can be very technical and very deep in terms of what you want to reconfigure. Then you can also have different behaviors and maybe the template actually doesn't change drastically. But suddenly, we're processing with Stripe, with whatever, some other provider like PayPal if it's possible, right, and not Stripe anymore.
Nico Rehwaldt (11:23)
But still, from the user's point of view, it's still the same building, not just a new version where we migrated to PayPal under the hood. There's a lot of flexibility in C8 and we also consider the modeling aspect from the beginning. This is the main difference, I think.
Josh Wulf (11:37)
Okay, yeah. People were obviously very passionate about it. It's obviously highly sought after, which is why people were talking about it a lot. I like your bold claim that this time, we did it right in C8.
Josh Wulf (11.53)
Can we talk a little bit about the technical way that this is implemented in C8? Because of course with Camunda Platform 8, we have both self-managed and we have SaaS as well. We have the fully-hosted, fully-managed-by-Camunda offering, and then we have the self-managed offering as well. Connectors. Obviously, with the SaaS one at the very least, there's going to be a sort of like selected set of connectors that are going to be provided in there - I would imagine. Can you give us a sense of how are these things implemented? How do they work? Which ones will be available initially, that kind of thing?
Nico Rehwaldt (12:34)
Just as a little bit disclaimer, right? What we're talking about here is given the large scale of things brand new, right?
Nico Rehwaldt (12:42)
There has been a lot of work, obviously in the foundations, a lot of stuff that I've just presented. It should be straightforward. Well, it's just pragmatic. It makes sense, right? There's a lot of demand in the market and a lot of people. It resonates. The story resonates well with people.
Nico Rehwaldt (13:02)
But the more we talk about the actual implementation and the more the actuals, what we're going to provide, the more of this is actually still in the making, right?
Nico Rehwaldt (13:12)
For example, connectors, one thing that we didn't talk about, or which what we maybe just quickly mentioned, but let me talk about it again, there's different kinds of connectors. We talk about inbound and outbound connectors.
Nico Rehwaldt (13:27)
Inbound connectors is everything. An external system change, for example. A new record in a database or a new message available via Kafka or a “payment has successfully been processed” notification by Stripe, right? This does something with your process. It does a change in your process. It correlates a message. It triggers a BPM error, or it starts a new process. It's this inbound connectors. External system changes cause something in my process to change or a new process to start.
Nico Rehwaldt (14:01)
Then we have outbound. Outbound is very similar to our existing job worker pattern that we already have. Or in C7, external task workers. Right? Outbound means I am actually, as the engine, causing some external system state change. I am actually sending a message, submitting a message to a Kafka queue. I am actually basically triggering the processing of the payment with Stripe, right? Or I am sending a message on Slack, if you want to have Slack as your convenience communication platform, right?
Nico Rehwaldt (14:31)
Those are the two different aspects. Now, as we look at what we already have at C8 in terms of connectors, we already offer some very basic out of the box outbound connectors. If you try out C8 right now, go into the web modeler, you can out of the box use the REST connector. Basically, you can model a task, put in a REST task, and then you configure what domain you want to talk to, what actual HTTP message you want to use, provide query parameters. Then you get the result back and then you can map the result back into your process, right? Easy outbound connector.
Nico Rehwaldt (15:08)
Then we have a little, a few domain specific ones. Like you can send an email because sending email is hard right now. You can send it via SendGrid and you can also, I think that's in the making about to be published, send a message to a Slack channel. We both have like the first iteration of a very reasonable, very powerful connector in terms of the rest outbound connector, but also some more domain specific hooks into existing communication protocols/platforms.
Nico Rehwaldt (15:37)
This is in SaaS, right? We offer this out of the box to everyone spinning up a new C8 cluster in Camunda 8 SaaS. We have very specific requirements there for outbound connectors, so we need to make sure we are able to offer the platform to as many people as possible to a reasonable price. Right?
Nico Rehwaldt (15:58)
One thing, for example, when we build connectors and the connector SDK, like we allow you to build connectors, we need to make sure that it's possible to operate those connectors in a SaaS environment, where you only want to execute the connector as they're actually being used.
Nico Rehwaldt (16:15)
Given, let's say we have 100 customers and only one of them is actually using Slack. We don't want to pay for 99 boxes of Slack running all the time, because no… one of them is using them, right? Still, maybe for the one person or maybe for the 5% of people, Slack is very meaningful, right? We need to have this architecture that supports us.
Nico Rehwaldt (16.37)
Internally what we're doing in SaaS, for example, connectors are actually just lambda functions that are being executed. Only as you're using, for example, Slack, this connector behavior is actually being spun up, right, started and executed.
Nico Rehwaldt (16:51)
For self-managed, it's a little bit different because self-managed, you as a user, you are in control of your own cost. You can decide to build it in the exact same way, right? Building a run time for connectors in which is lambda function based, right, on demand. It's trivial using the SDK that we're about to provide, right, about to publish.
Nico Rehwaldt (17.15)
But you can also just spin it up as a job worker, hosting behavior for Slack connector, for Stripe connector, for Kafka and so on. Then it's basically just a single box that is running all the time. In self-managed, you have a lot of control what you want to do.
Nico Rehwaldt (17:30)
You can also go our route. Let's say you are a huge corporate, huge company, and you have the same kind of issues at some point, right, when you offer an automation cloud based on Camunda, right? You have the same kind of issues. But if you want to start quick on self-managed, you can also just spin those connector run times up as a job worker.
Nico Rehwaldt (17:50)
This is the flexibility you get. I think this is also historically one of the strong points of Camunda actually, that we offer our users a lot of architecture options, right? There's so many different ways for our customers to use us. For single automation projects for large scale centers of excellence and so on. This kind of architectural freedom, how you want to deploy, this is something that we enable with the connector SDK. This is something that we also benefit for C8 SaaS.
Josh Wulf (18:22)
That make sense for the outbound connectors. I totally get it. You can run just a continuously running job worker or job workers to service those tasks. That's the standard Camunda Platform 8 pattern.
Josh Wulf (18:35)
Or you can run these Lambda functions. I imagine that there must be some generic Lambda trigger job worker that has to run. In order to make that happen as a technical implementation for that.
Nico Rehwaldt (18:47)
Yeah, exactly. Obviously, there has to be one job worker, right, And what we call the connector bridge. For C8 SaaS, to go a little bit into detail, there is always one beacon, one job worker running. That knows all the existing functions, so this is also basically how we explicitly make certain functions available in C8 SaaS.
Nico Rehwaldt (19:12)
At some point in the future, anyone will be able to create those outbound connectors very easily. But still, we want to make sure what is the actual list of outbound connectors that we support for all SaaS users out of the box? It will always be possible for you to, fairly straightforward add your own additional, but then those will not be, at least for the beginning, those will not be automatically provisioned by us.
Nico Rehwaldt (19:35)
But again, I can just spin up a job worker next to my cluster and that provides behaviors, right? It's architectural freedom. Yeah. But we have the connector bridge in the middle, so we have one component in the middle that knows, okay, those are the functions and will also register job workers for each of those.
Josh Wulf (19:54)
What about, for the inbound connectors? That's a different story for those?
Nico Rehwaldt (20:01)
You could call inbound a different beast from the nature of it. Like in inbound, thinking in reasonable building blocks from the modeling perspective again, right? I want to configure the start event to trigger like the process instance to start when a certain message arrives via Kafka. That's a level on which I actually want to configure that on the diagram, so how to make that happen?
Nico Rehwaldt (20:27)
The nature of inbound is an entirely different beast and we still need to get through all the details of it. But long story short, the framework that we want to provide our users should still be as simple as possible. Right?
Nico Rehwaldt (20:41)
But to give you one of the challenges. Inbound, it requires that I configure that on the diagram. I need to be aware when a process is deployed of this configuration on the diagram, on the process that has been deployed, so inbound requires a listener that the process is actually being deployed. Then I want to spin up those components, like the connector behavior, right?
Nico Rehwaldt (21:03)
For example, if I configured it to react to a certain message on a Slack channel, I have to subscribe to Slack or spin up my Slack app, right, to actually get notified by Slack if that condition actually happens. Inbound is always stateful. Outbound, we have this very easy stateless, easy to conveniently execute using Lambda functions, right, in the stateless. Inbound will always have some strong stateful components in there. We need to still figure out how to build it in the best way.
Nico Rehwaldt (21:36)
But from the user's point of view, as you build your connector, for example, as you define your inbound Slack connector reacting to a message, we try to provide you with a programming framework where you don't need to worry about the complexity. Because also, that enables us again to give you but also us a little bit of architectural freedom how it's being executed, right?
Josh Wulf (21:58)
Wow. This sounds cool. I'm just imagining here. Like a listener on the Elasticsearch?
Nico Rehwaldt (22:04)
Josh Wulf (22:04)
Then like a life cycle for the inbound connector and it gets a configuration call back.
Nico Rehwaldt (22:10)
Yes. Think about it. Think about it like this. I have the Slack connector runtime for inbound. The framework will call me, activate for this given process or for this given configuration, right? Also deactivate because at some point, it's un-deployed right or some other change that triggers it. Yeah.
Nico Rehwaldt (22:30)
If you think it like this, actually building the connector is still very simple, but there is a lot of heavy lifting going on under the hood. For example, we need to clarify how to get actually access to internal information, right? We also want, in any case, both outbound and inbound, we want to give our users the opportunity to run those co-located next to their actual systems. Right? Because co-location is something very powerful. If I run next to my Kafka queue in my data center, I don't actually need to expose the Kafka queue to the internet. This is something extremely powerful, and we're seeing a lot of customers use it. For inbound, it's the same thing. Right?
Nico Rehwaldt (23:13)
We'll get to some of the details. I think Bernd also planned a blog post on connectors, so that could be also good collateral to read for everyone who wants to look into that stuff. That should hopefully give you a feel how simple it eventually will be for you to build the connectors. Then we, on our end, have to do some heavy lifting to make it possible. Fingers Crossed.
Josh Wulf (23:36)
This is kind of like a... This is kind of like a technical preview, preview.
Nico Rehwaldt (23:40)
Yeah. Preview of the preview.
Josh Wulf (23:42)
Yeah. We've got outbound connectors now available in SaaS. Are there any inbound connectors available yet?
Nico Rehwaldt (23:51)
There are none because the architecture is really in the making. There's a couple of plans. But again, you should have all the plans... You should take all those plans with a grain of salt.
Nico Rehwaldt (24:00)
Right now, we are really working through some of the technical challenges. Whatever we want to make available, we want to make it available for you, for anyone to adopt on SaaS at scale, but also for self-managed. Because self-managed, so many people have been asking for connectors for self-managed.
Nico Rehwaldt (24:17)
There's a lot of desire. We just, right now, we are architecting inbound. Right now, validating also the architecture and then we take it from there. Obviously, we want to be as quick as possible, but you can't be quicker than that.
Josh Wulf (24:36)
Yeah. You can't rush greatness. Like you said, the bold statement, this C8 connectors is “connectors done right”, so they'll be ready when they're ready. There's a strong modeling piece there, element templates, configuration, configuration by configuration. There's like a meta configuration, so it's like you have a configuration file of some kind that you deploy that describes the configuration that's required for a particular connector run time.
Nico Rehwaldt (25:04)
You refer to the element template, right?
Josh Wulf (25:07)
Yeah. The element template is like a... That's been around for quite some time, hasn't it?
Nico Rehwaldt (25:12)
Yes. It has been around for quite some time. But given how powerful it is, you can argue it's actually incredible that we didn't pick it up.
Nico Rehwaldt (25:22)
For example, element template has been around. I've actually created the foundations in 2016, I think. We had some customers pick it up. Everyone who looks into reuse would naturally pick it up despite the fact that we never blogged about it, despite the fact it was hidden somewhere in the documentation, some page. But people knew it. People building, in larger organizations building larger systems, they knew it and they would naturally pick it up. This is already the success on its own, right?
Nico Rehwaldt (25:50)
An element template was fairly limited, if you look at what we've improved on in element templates for the C8 launch. We added grouping. We added, obviously, element templates for C8, because until then it was only a C7 feature. We added icons on the diagram, which is very important if you think about in reusable building blocks. You want to have those also recognizable on the diagram. There should be a difference between a plain service task configured using input and output and headers, and this mail task or this money task. I want to get money. This should be whatever dollar symbol on it. It's actually your freedom to do, right?
Nico Rehwaldt (26:33)
We've invested heavily in making element templates much more meaningful, and we will also evolve that. On one end, it actually makes me very happy that there was a strong connector focus on reusable building blocks, are also very heavily investing in element templates nowadays and evolving that.
Josh Wulf (26:54)
Awesome, great work. Sounds like a lot of pieces are finally coming together in the perfect kind of synergy.
Nico Rehwaldt (27:02)
Yeah. Let's hope. Let's hope that's the case.
Josh Wulf (27:05)
I know you've already spoken to it, really. It's like, when is this coming? When it's ready. Outbound connectors, there are some available now in SaaS. At some point, that'll make it into self-managed. Inbound connectors are going to show up at some time. There's an SDK coming to enable you to build your own connectors. And Bernd has a blog post about the whole technical architecture coming. That might even be available at the time that this podcast episode goes to press. If it is, I'll put it in the show notes so people can check it out.
Nico Rehwaldt (27:38)
There's a couple of things. There's a couple of things that are already in the making very close to publish, right? Those that know element templates, they will always... They would wonder how to configure them, right?
Nico Rehwaldt (27:50)
Then element templates was a feature that was available in the Desktop Modeler, right? It a well-defined way how to configure them. Until now, it wasn't available in the Web Modeler. But next versions of the Web Modeler will actually ship with a convenient element template editor. You can actually, in the Web Modeler, next to create your BPM diagrams and forms and also DMM, you can also just design your reusable building blocks.
Nico Rehwaldt (28:19)
For example, if I wanted to work with you and you just wanted to use the Slack connector, I can already create this element template, this reusable building block. I can create that and I can provide that for you so you can use it in your same project.
Nico Rehwaldt (28:34)
This is one thing very exciting that is coming up in the connector realm. Right? The other one, as you talked about, is the SDK, which is currently an internal preview, SDK for outbound. We're going to publish it sometime in the near future. SDK for inbound, that will take a while. But already having the SDK for outbound, I think that makes our whole vision around connectors, building the connector run time, much more tangible.
Josh Wulf (28:58)
Epic. I look forward to getting my hands onto it and trying it out. Anything else that you wanted to share with us today about connectors, Nico?
Nico Rehwaldt (29:08)
I will just encourage everyone, the moment you see some news, you see some extra source code, you see some extra SDKs on the Camunda GitHub organization, just gets your hands on it. We definitely explore the space we're operating in a lot, right? We see a lot of demand by people, right?
Nico Rehwaldt (29:24)
But especially with something that is early in terms of product, right? It's your chance to also give feedback. If you give feedback early and often, it allows us to shape that one and make it even better. If you build your own element templates, you feel there's features missing, right, we always have an open ear for the C8, on the C8 forum, but also other channels. Just basic feedback. If you have interesting ideas how to make that stuff even more powerful, just approach us. We're really, really happy to dive into those topics.
Josh Wulf (29:56)
Awesome. Thank you for your work on this connectors framework, Nico, and thanks for coming on the show.
Nico Rehwaldt (30:03)
Thanks for having me.
Roll the credits! Audio engineering provided by Inclusion Audio production, check them out at inclusionaudio.com. The soundtrack is provided by the legendary Japanese Progressive House, DJ Shingo Nakamura, courtesy of monstercat.com.