Architecture

Kaoto is designed following an Hexagonal Architecture¹.

We have a domain defined in the model project . This domain contains generic classes needed to define and build flows. Those flows will be composed by ordered steps. Those steps will have constraints and configuration properties that are defined in a generic way but are able to adapt to different Domain Specific Languages (DSL).

There is a set of interfaces that define the features the different DSL need to implement as use cases. For each language or framework we want to support, we will need a specific implementation of these interfaces. The main responsibility of that implementation is to act as translator between the specific DSL and the generic domain objects.

Following good practices of Quarkus, we have a separated set of services that connect the Application layer with the Framework. There is an API that acts as the main adapter to connect frontend and backend. We also have different parsers that feed the application with metadata describing the catalog of available steps or the catalog of available extensions. This way, we can configure different flavours of Kaoto with the same build.

Domain

A flow is an ordered list of steps. A step is the minimum building block in a flow. It usually contains one action (or meta-action). For example, a step may be a query to a database to extract data, connecting to a remote storage to store data, or sending an email. You can find more information on the Step autogenerated documentation .

Some steps are complex and divide the flow into separated branches. Each branch is conceptually another flow by itself, with a list of steps. Steps usually contain some configuration properties defined following the schema validation .

All flows also contain metadata associated to it, like the name or some annotations. If the flow has input or output parameters, this will also be defined in the metadata.

Separation of Concerns

Kaoto has a clear Separation of Concerns to maximize decoupling and therefore extensibility of the application. Following the Single Responsibility Principle, the backend offers an API that handles every transformation and parsing needed to support each DSL. The frontend will not care about the specifics of the DSL, and will only use the generic objects as defined on the Domain. Both frontend and backend share the same domain.

This means that the frontend is agnostic of the underlying DSL used. The frontend understands the generic abstract concept of steps organized on a flow. But it does not parse or understand the source code that is going to be generated. It focuses on the visual canvas and the high level human definition of what flow we want to implement. The steps in the flow have properties that need to be configured and have constraints that define when and how they can be used. But all those constraints and properties are defined in a generic way, so we could plug in different DSLs on the backend and the frontend will handle all of them the same way.

Dependency Inversion

Each Domain Specific Language (DSL) that we want to support in Kaoto has an independent implementation in the backend. Each DSL will have a plugin or extension that follows the defined interfaces (or use cases, in the diagram) that will allow the API of the backend to handle that DSL.

Using the Quarkus built-in dependency injection , the Dependency Inversion is easily approached by allowing different DSL implementations to be plugged-in into the backend.

Implementation details

Kaoto source code is divided in several repositories , where the most relevants are the frontend and the backend.

Frontend

The frontend is designed as a micro-frontend architecture ^{2 3}.

The core frontend interacts with the backend and is able to visualize and edit a flow seamlessly, working with the generic domain classes.

But sometimes we need a finer grain detail to edit properly a flow. This is when the extensions and view definitions come into play. Extensions can be configured on the fly, without having to rebuild or redeploy Kaoto.

The reasons to want to extend the default generic handling of steps are varied. It could be because we want to offer some kind of wizard to configure a step in a flow. Imagine we want to connect to a database, we may want to offer the user some kind of helper to build the SQL query or a list of tables available in the database. It could also be a complex step, like a data transformation, or a loop over elements, and we want to offer some helpers to the user.

Each step extension is a micro-frontend that is tied to a specific DSL and step. They are configured outside the frontend source code, allowing us to hot-plug step extensions into our running Kaoto instance.

It is the backend who will return the list of applicable extensions to the flow that is being edited. Frontend will then load the corresponding step extension from the Extension Catalog, delegating on the code of the extension all the needed custom code to configure that step.

Backend

The Backend relies on a series of configuration metadata catalogs to guide the frontend.

These catalogs contain:

• All the information about the available steps
• When to use each view definition (extensions)

The backend is the piece that will know what DSLs are supported at any given time. You can find more information about adding Domain Specific Languages to Kaoto on the documentation .

Steps

The responsibility to list all the available steps lies on the backend. Each step will be tied to one or more DSLs, and it is the backend who will list for the frontend which steps are available and permitted at any time. Usually the catalog of available steps is based on a local or remote catalog configured by the administrator of Kaoto.

The backend will do all the transformations between the generic domain objects that define the flow (which the frontend understands) and the generation of the source code ready to be deployed.

View Definitions

Given the list of steps generated with the previous service, Kaoto can decide what micro-frontend extensions can be used with the flow the user is editing.

The view definitions are the configuration bits to know which views and extensions to load on the frontend. These view definitions are also configurable and usually stored on some local or remote repository that the administrator of Kaoto configures.

You can find more information about adding Custom Step Extensions to Kaoto on the documentation .

Backend and Frontend interaction

Let’s see some interaction diagrams detailing how the backend and frontend interact.

Deploy a Flow

Once the user has finished editing a flow, they may want to deploy it on a cluster. There is a button on the frontend that will send the flow (in generic domain objects) to the backend so the backend is the one that, based on the Dependency Inversion Principle, will delegate into the specific DSL deployment implementation.

On the previous figure, the backend is deploying the flow into a cluster.

Watch logs

On a similar note, the user may want to watch a log of an already deployed flow. The backend will first return the list of deployed flows (again, based on different DSL implementations), so the user can choose which flow to watch.

Finally, the log of the flow will be sent back to the frontend so the user can watch it.

Edit a Flow

When editing an orchestration, Frontend will send Backend the modified full flow and the Backend will return all the information needed to refresh the full Frontend interface.

There are two different versions of this flow depending on what action is done.

If you modify the source code, what will return the visualization:

If you modify the visualization, that will refresh the source code on the frontend:

In both cases, the Backend will return the same information (visualization, view definitions, and source code), so the frontend will have everything synchronized and aligned at all times. This is important when editing the source code, as invalid parts of the source code will be ignored to be able to generate a valid visualization. This invalid parts of the source code will not be present on the returned response, allowing a constant clean up of the source code.

You can find more general information on the Backend autogenerated documentation .