Performance tuning

Learn how to tune the Bonita Engine to optimize the performance of Bonita Runtime in your environment.

In this page, we assume that you are familiar with Java threads, concurrent execution, XML, DB connection pools, your DB instance, cache policies, scheduling, connectors, network speed, I/O, Java Virtual Machine configuration, JTA and transaction management.
You need to know how to install and configure Bonita Runtime.

This information applies primarily to the Enterprise and Performance editions, though some details also apply to the Teamwork and Efficiency editions.

It is also possible to use gzip compression on your application server to improve performance.

Summary

This section is a summary of the key recommendations from the rest of the page. You can use it as a checklist of things to consider when tuning system performance.
To understand these recommendations in detail, read the sections below.

Two key definitions:

  • The maximum possible number of parallel threads that could be required at any given time is the sum of:

    • the number of workers

    • the number of scheduler threads

    • the number of external API calls

  • The processing capacity is the desired number of parallel threads, and is the sum of:

    • the number of workers

    • a percentage of the number of concurrent scheduler threads

    • a percentage of the number of concurrent external API calls

Performance tuning checklist of best practises:

  • Access Engine APIs in Local mode whenever possible.

  • Make sure that the network between your client and the Bonita Engine server is fast if you access the APIs remotely.

  • Install your Database on a powerful machine (hardware that meets your DB vendor requirements: sufficient memory, powerful CPU, and fast I/O).

  • Make sure that the network between your Bonita Engine server and your database server is very fast.

  • Set the Work service threadpool and Connector service threadpool to a size according to the usage of your processes.

  • Set the maximum DB connections for the bonitaDS datasource to the desired processing capacity number of parallel threads.

  • Configure a suitable sequence manager range size for your typical process designs and expected volume.

  • Make sure that the maximum DB connections for sequenceManagerDS datasource is more than 1 and is appropriate for the SequenceManager range size configuration.

  • Tune the application cache to your hardware capabilities (if you have more memory available, increase the cache size).

  • Set the basic JVM options to make it fast and well-sized.

  • Optimize your database configuration for your most frequent usage (read or write) and the level of robustness that you want.

  • Configure your transaction manager for the level of robustness that you want.

  • Tune the log levels of Bonita Engine and all its dependencies.

  • Design your processes following the best practises.

  • Add a reasonable number of well-developed event handlers.

  • Tune the Bonita Engine cron jobs for your needs.

  • If your database is PostgreSQL, follow our recommendations.

Engine access

This section deals with performance impact of your choice of Engine access mode.

There are various ways to access the Engine APIs provided by Bonita Engine. Choose the most suitable access mode for your deployment, requirements, and preferences.
The access modes rely on different technologies and have different benefits and drawbacks. In this section, we will describe the performance characteristics of each mode.

Local access

This is undoubtedly the fastest way to access this engine, because it means a direct Java call with nothing additional between client and server.
The deployment constraint is that the client of the engine must be located in the same JVM as the engine server.

Remote access

The remote access modes enable you to have an engine client connected remotely to the engine server.

All remote access modes share a set of common benefits and constraints. There is a loss of performance mainly because of serialization between client and server.
If your client is not located on the same machine than your server JVM, the network becomes an additional potential source of performance reductions to monitor.

In some deployments, it is possible to benefit from the best of both local and remote modes.
The engine server access mode is defined per client and does not need to be the same for all clients.
If you have a client located in the same JVM as your server, configure it to use the local access mode.
You can then configure other clients to use one of the remote modes but you do not penalize the client able to leverage the local access performance.

HTTP

The HTTP access mode is available using our natively provided bonita-client library. It can also be used from other technologies like PHP, .Net or Javascript (in that case, you need to develop your our client).

We do not guarantee to keep the http protocol stable, so we strongly recommend that you use

  • the standard bonita-client library

  • the same version for both the client and the server

This mode can be easily used inside a web container like Tomcat or Jetty.

The bonita-client library

Data sent is serialized using a Java library called XStream. This serialization also has a cost.

REST

This method of accessing the Bonita capabilities is not yet integrated as an engine service but exists as a web application service accessed using the Web REST API. No details are provided here as it is currently out of scope.
In general, the constraints are almost the same as for the HTTP mode, but we do not provide any Java client for this access mode.

Concurrent execution

This section describes some aspects of engine configuration that have a performance impact if there is a high level of concurrent execution.
Before you read this, make sure you are familiar with the engine execution sequence, states, and transactions.

There are two main entry points for load on the engine:

  • API calls coming from outside the engine

  • Engine-generated calls for internal processing, specifically the Work service and the Scheduler service

Bonita Engine is an asynchronous BPM process engine. This means that every thread that deals with process execution applies the following rule: do the minimum that makes sense in the current transaction to get to a stable state, and then continue in another transaction inside another thread.
The great benefit of this is that the caller is not locked while the engine processes something that might be long (such as a long sequence of tasks with connectors.).

Client Threads

Client threads are responsible for a large part of the load generated inside the engine.
The number of client threads is related to the number of parallel users.

If you are running your own application, you have one thread if your applicaiton is not multi-threaded, or you have the number of threads you decided to create explicitly in the application or using your own threadpool.

If you are running Bonita Engine inside a container, the maximum number of client threads is defined by a parameter of the container. For example: Apache Tomcat maxThreads set in `Tomcat_folder``/conf/server.xml`.
Default value 20.
See the Tomcat documentation for information about the maxThreads parameter.

Work service

The work service is responsible for asynchronously processing execution of process instances. The work service has its own thread pool, which can be configured for each tenant.
This is one of the key configurations to optimize, because even though there are many client threads, client threads are held only for a short time before being released, and then execution flow continues using work service threads.
A thread from the pool of the work service is known as a worker.

The work service is configured in bonita-tenant-community-custom.properties.

bonita.tenant.work.terminationTimeout=30
bonita.tenant.work.corePoolSize=25
bonita.tenant.work.maximumPoolSize=25
bonita.tenant.work.keepAliveTimeSeconds=60
bonita.tenant.work.queueCapacity=10000

It is very similar to the constructor provided in the default JDK ThreadPoolExecutor.
For a reminder of how the threadpool behaves, see the Queuing section of the ThreadPoolExecutor documentation.

In the default Bonita configuration, corePoolSize is equal to maximumPoolSize because we have observed that the default implementation of the threadpool executor allocates work to available threads using a round robin algorithm.
Therefore, if the maximum is reached, the thread pool size is unlikely ever to reduce to corePoolSize, because work is always allocated to available threads.
The current implementation of the RejectedExecutionHandler queues the work, and reduces the system load because it does not release the caller (normal behaviour for a BlockingQueue).

After a lot of profiling, we have concluded that having an arbitrarily high number of threads in the work service does not positively impact the performance of the whole system, because it leads to a lot of contentions, mostly on the database (see Database connections).

The size of the threadpool (corePoolSize in the default configuration) is key, and correlates to the number of process instances the engine can handle in parallel.
In other words, if you want the engine to be capable of handling X process instances concurrently, you should set the corePoolSize value of the work service to X.
You then need to ensure that your platform infrastructure can handle X concurrent instances, checking that all other engine dependencies including the network and the database are able to process all incoming requests without loss of performance.

Setting a high queueCapacity limit means that more work can be queued, but can reduce throughput as work is queued rather than causing a new thread to be created.
It is essential to ensure that the queue never becomes full (queueCapacity is never reached).
If the queue becomes full, the application restarts in order to force the engine to generate all work from the database. This means that work is lost.

SQLServer-specific work configuration

When Bonita platform is under high volumetry on work execution and database transaction, sometimes when one work commits its data and next transaction tries to access it, this information is not yet visible.

This issue happens only when using Bonita and BDM XA resources ( XAMultipleResource ) and because the transaction isolation level is configured as ALLOW_SNAPSHOT_ISOLATION and READ_COMMITTED_SNAPSHOT. These isolation levels are mandatory to avoid a deadlock.

To avoid the issue described above, by default, a 100 ms work execution delay is added when the database is SQL Server and if the previous transaction has multiple XA Resources ( Bonita + BDM ). This small execution delay allows database to handle the commit and update the information out of the isolated level, so next request out of the write transaction can get the updated data. The work execution delay is configured in bonita-tenant-community-custom.properties.

# Add a delay on work when the transaction that registers the work has multiple XA Resources
# This is an SQL Server specific property to ensure all data commit are visible when the next work is executed.
bonita.tenant.work.sqlserver.delayOnMultipleXAResource=100

Connector service

The connector service executes connectors. To improve tenant isolation (and to protect against denial-of-service attacks), the default implementation of the connector service has its own thread pool and requires executes connectors in a separate thread from the worker.
The configuration of the thread pool of this service is independent of the configuration of the work service. If you have processes that use a lot of connectors, then you can have more threads to execute connectors. See the connector execution page for details on how connectors are executed.

The Connector service is configured in bonita-tenant-community-custom.properties and bonita-tenant-sp-custom.properties (cf platform setup)

Community:

bonita.tenant.connector.queueCapacity=10000
bonita.tenant.connector.corePoolSize=5
bonita.tenant.connector.maximumPoolSize=100
bonita.tenant.connector.keepAliveTimeSeconds=100

Subscription only:

bonita.tenant.connector.timeout=300

For details of these parameters, see Work service.

In addition, connectors longer that 10 seconds produce a log at warning level named: org.bonitasoft.engine.core.connector.impl.ConnectorExecutionTimeLogger. This log contains all references to find exactly which connector is slow.

Another log at the debug level prints all input parameters of this connector.

Here is a sample log produced using a connector that does a Thread.sleep(15000)

WARNING: Connector 15 sleep with id 20002 with class org.mycompany.connector.SleepImpl of process definition 6587226372021992905 on element flowNode with id 20003 took 15001 ms.
FINE:  Input parameters of the connector with id 20002: {seconds: [15]}

The 10 seconds threshold can be changed in the configuration file bonita-tenant-community-custom.properties

bonita.tenant.connector.warnWhenLongerThanMillis=10000

Scheduler service

The Scheduler service is responsible for executing jobs.
A job is executed inside a thread of the scheduler service.
There are various kinds of jobs, some resulting from internal requirements such as API session cleaning, or batch deletion of a table row, and some related to process design such as BPMN2 events.
The Bonita Engine Scheduler service uses the Quartz Scheduler. Quartz takes the size of the threadpool as an input parameter. Quartz uses threads to execute jobs concurrently.

The Scheduler service configuration is in bonita-platform-community-custom.properties. You can configure:

bonita.platform.scheduler.quartz.threadpool.size=5
bonita.platform.scheduler.batchsize=1000

Database connections

Two datasources are defined:

  • bonitaSequenceManagerDS is used for distributing ID requests

  • bonitaDS is used for everything else

Note that the sum of the maximum values configured for bonitaDS and bonitaSequenceManagerDS should be less than or equal to the maximum number of simultaneous connections allowed to your database.

bonitaSequenceManagerDS

This datasource needs only a few connections: between 5 or 10% of bonitaDS number should be sufficient. However, this is closely correlated to the range size.

bonitaDS

This datasource requires a higher value, because Bonita Engine stores almost everything in the database. This means that every single thread from any of the entry points requires a database connection through bonitaDS.
To make sure that this datasource is not a bottleneck, define the maximum number of database connections to be equivalent to the desired number of parallel processing threads.
The desired number of parallel processing threads is the sum of the number of workers (see Work service) plus a percentage of the number of scheduler threads (see Scheduler Service) plus a percentage of the number of concurrently external API calls (see Client threads).

Datasources settings

You need to configure the maximum pool size for datasources (the following paths are for bundle users).

For Tomcat, edit the file setup/tomcat-templates/bonita.xml:

  • For bonitaSequenceManagerDS, set maxTotal=”yourvalue”.

  • For RawBonitaDS, set maxTotal=”yourvalue”.

  • If necessary, for the Business Data feature, do the same for the datasources 'RawBusinessDataDS' and 'NotManagedBizDataDS'.

Volume

This section deals with some aspects of the engine configurations that have a performance impact in the case of high volume.

Sequence manager

Bonita Engine manages a dedicated sequence for each table for ID generation. This implementation allows fast delivery of IDs and a single point of usage inside the application: the persistence service.

The sequence manager keeps in memory a range of reserved IDs by table.
This range size is configurable by sequence so that it can be adapted to the volume you have.
The bigger a range is, the less frequently the sequence manager will have to query the database for a new range, because it is managed in memory for as long as possible.
However, all the IDs that are reserved in memory are lost when the JVM is shut down, so the number should not be too big or you might reach Long.MAX_VALUE too quickly.

The sequence manager allows you to set the range size for each sequence and a default range size value, which is applied to any sequence that does not have a specific range defined. If you want to tune these values, you have to understand the correlation between them.
For example, if you have an average of 20 steps in your process, then it would be reasonable to set the ActivityInstance range size to be 20 times bigger than the ProcessInstance range.

The sequence manager configuration is in bonita-platform-community-custom.properties.

The sequence manager has its own database connection.
This should be appropriately sized for the number of times the sequence manager will query the database, which is a consequence of the range size values. See Database connections.

Persistence cache

For Subscription editions, Bonita Engine has a cache providing a persistence layer using Hibernate caching.

EhCache configuration for this persistence layer is defined in a file named bonita-platform-hibernate-cache.xml and bonita-tenant-hibernate-cache.xml.
It is possible to modify the cache settings in those files for each kind of object.

Before going into production, we encourage to finely tune the "Level-2" object cache in a pre-prod environment:

  • activate Hibernate cache statistics by setting to true the parameter bonita.platform.persistence.generate_statistics in file bonita-platform-community-custom.properties

  • activate logs at INFO level:

<logger name="org.bonitasoft.engine.persistence" level="INFO"/>
<logger name="com.bonitasoft.engine.persistence" level="INFO"/>
  • run load tests to simulate a production environment

  • analyse the "2nd Level Cache Ratio" log messages generated, combined with the "soft-locked cache entry was expired" warnings messages to change the configuration in file bonita-tenant-hibernate-cache.xml.
    For instance, if on entity org.bonitasoft.engine.core.document.model.impl.SDocumentImpl, the "soft-locked cache entry was expired" warning message occurs, it means the size of the maxElementsInMemory parameter must be increased, provided it is a reasonable memory size and provided the "2nd Level Cache Ratio" is not low for this element.
    If the "2nd Level Cache Ratio" is low or even 0, it means the cache is never used to read several times the same entity, which means the timeToLiveSeconds parameter should be increased, or the cache should be completely deactivated for this entity.

Below is an example of a "soft-locked cache entry was expired" warning message:

WARNING: Cache org.bonitasoft.engine.core.process.instance.model.impl.SFlowNodeInstanceImpl Key org.bonitasoft.engine.core.process.instance.model.impl.SFlowNodeInstanceImpl#org.bonitasoft.engine.persistence.PersistentObjectId@25505ff
Lockable : null
A soft-locked cache entry was expired by the underlying Ehcache. If this happens regularly you should consider increasing the cache timeouts and/or capacity limits

Application cache

Bonita Engine uses an application cache to store specific objects. The default implementation of this service relies on EhCache. It is configured in these files:

  • bonita-platform-community-custom.properties

  • bonita-tenant-community-custom.properties

  • bonita-platform-sp-cluster-custom.properties

  • bonita-tenant-sp-cluster-custom.properties

The following cache configurations can be defined:

Configuration Purpose

connector

stores connector implementations for a given connector definition

processDefinition

stores process definition objects

userFilter

stores user filter implementations for a given user filter definition

groovyScript

stores compiled versions of Groovy scripts

transientData

stores transient data

parameterCacheConfig

stores process parameters

Java Virtual Machine

You can configure the JVM settings for the engine to tune performance.
Check the JVM documentation for details of the available settings.

Notably, we recommend you to set the initial (-Xms) and maximum (-Xmx) heap sizes to the same value.
This reduces the likelihood of the JVM garbage collector starting.
While the garbage collector is running, it prevents creation of new objects, which slows down the application server.

Garbage Collector

We recommend to update the Garbage Collector parameters to use the following ones (UseParallelGC has been proven more eficient than UseParNewGC in our experience):

  • Edit file setup/tomcat-templates/setenv.sh

  • In the line CATALINA_OPTS=, add: -XX:+UseParallelGC -XX:ParallelGCThreads=10 -Xlog:gc*:file=<CATALINA_BASE>/logs/gc-$(date +%Y_%m_%d-%H_%M).log:time,uptime,hostname,pid:filecount=5,filesize=20M

Hardware and network

This section deals with performance impact of hardware elements.

Bonita performance is very correlated to the database connectivity and its behavior.
Almost everything (API call, internal processing using workers, jobs scheduling, and so on) requires a database access.
Two elements are critical: network latency, as in most cases your database is located on another server, and the I/O of your hard drives.
In case of issues, you should monitor these two elements and consider improvements. For example:

  • locate your database in the same datacenter as Bonita Engine, using gigabit network connections

  • use SSD hard drives, and RAID configuration with striping

Network connectivity also impacts access to the engine APIs when you are not using local access, that is, if you are using HTTP, REST.

Database, Transaction Manager, and logs

This section is a reminder about some of the main dependencies Bonita Engine has that have a strong impact on the performance of the whole system.

Bonita Engine relies on several other components that each has its own performance tuning options. Some of them are key for the system and you should pay a lot of attention to them.
In most cases, the key things to consider are the database, transaction manager, and logs.

Database

Bonita Engine uses the database heavily, so in consequence a slow database makes the engine slow.

It is essential that the hardware configuration of the server hosting the DB is powerful, considering resources like CPU, memory or others depending on your database instance.

In addition to this, make sure that your database instance is well configured.
Most database softwares provide many options for tuning, and some of them are easy to set up.
Others may be more difficult and present choices between robustness and performance, fast read or fast write, etc.
Your database configuration must be correlated with Bonita Engine usage pattern. To find the right characteristic to optimize, one good starting point is to consider whether you are creating a lot of process instances (in which case optimize database writes) or you are executing a lot of read queries like getTaskList (in which case optimize database reads).
Specific PostgreSQL performance tuning is given as a database tuning reference.

Transaction manager

Bonita Engine is natively compatible with the Java Transaction API. This means transaction management relies on a transaction manager.

Bonita Runtime embeds Narayana, an open source transaction manager.

It uses the following configuraton file server/conf/jbossts-properties.xml. The most common configuration to change here would be com.arjuna.ats.arjuna.coordinator.defaultTimeout that is the timeout for transactions. More details on the configuration can be found in the Narayana documentation.

Logs

In general, increasing the log level is useful for debugging but has a performance cost.
With this in mind, define the log level for technical logs, queriable logs and archives.

Remember that Bonita Engine dependencies also have their own log and debug options that may impact strongly the system performance.
Be sure to configure these appropriately.

Connector time tracker

It is possible to track the duration of actions in a connector using a time tracker. The tracker service tracks several connector lifecycle operations.
This service can impact performance so it is disabled by default.
It is configured by editing the following parameters in bonita-tenant-community-custom.properties:

## Time tracker
#bonita.tenant.timetracker.startTracking=false
#bonita.tenant.timetracker.maxSize=1000
#bonita.tenant.timetracker.flushIntervalInSeconds=30
#bonita.tenant.timetracker.csv.activateAtStart=true
#bonita.tenant.timetracker.csv.folder=$ {java.io.tmpdir}

#bonita.tenant.timetracker.csv.separator=;
#bonita.tenant.timetracker.memory.activateAtStart=false
#bonita.tenant.timetracker.memory.maxSize=1000000

To activate connector time tracking:

  1. Uncomment all the previous lines except ## Time tracker.

  2. Change the value of startTracking from false to true.

The other parameters can be left at their default value, left commented, or set to the desired value. What each of them does:

  1. maxSize maximum of records that will be saved by the time tracker before a flush. If the maximum number of records is reached before the scheduled flush, the older ones are discared. To avoid the loss of information, a number sufficiently big in comparison with flushIntervalInSeconds should be chosen

  2. flushIntervalInSeconds the interval beetween two flushes on the timetracker thread

  3. csv.activateAtStart wether to save the result of the timetracker into a csv file

  4. csv.folder the folder where to save the csv file

  5. csv.separator the separator character in the csv file

  6. memory.activateAtStart whether to save the result of the timetracker in memory

  7. memory.maxSize maximum amount of records saved in memory. If the maximum number of records is reached before the scheduled flush, the older ones are discared. To avoid the loss of information, a number sufficiently big in comparison with flushIntervalInSeconds should be chosen

The non-relevant options will be ignored at execution. Note that memory and csv can both be activated at the same time.

Process design, event handlers, and cron jobs

Process design

There are several things you can do during the process design to reduce performance overheads.
This is mostly related to reducing usage of extension points when possible.
Consider carefully your usage of connectors, groovy scripts, XML and serializable data.

Event handlers

Events handlers are extensions of the engine configuration.
You can add event handlers for several purposes and you can configure which events you want to catch.
We strongly recommend that you add only appropriate handlers and carefully code the handler filters to handle only those events that you are interested in.

BPMN Timers execution

Bonita Engine uses the Scheduler service to trigger timers.

The Bonita Scheduler service implementation uses the Quartz Scheduler. Some quartz properties can be modified to fine tune quartz jobs execution. These properties can be found in bonita-platform-community-custom.properties.

org.quartz.jobStore.misfireThreshold
org.quartz.jobStore.maxMisfiresToHandleAtATime
org.quartz.jobStore.acquireTriggersWithinLock
org.quartz.scheduler.batchTriggerAcquisitionMaxCount
org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow

Details on these properties can be found in the Quartz documentation.

They are not read subsequently, so changing the values in bonita-tenant-community-custom.properties after the engine has been started has no effect on Quartz. For value definition, and information about how to update the Quartz trigger tables, see the Quartz documentation about Cron Triggers.

PostgreSQL performance tuning

Here is Bonita advice to finely tune PostgreSQL database server performance.

In this example, we assume you have:

  • 12Gb of RAM

  • fast SSD storage

Update memory configuration in file postgresql.conf (typically /etc/postgresql/11/main/postgresql.conf) with the following values:

# MEMORY PARAMETERS:
# shared_buffers SHOULD be set to 1/4 of the total memory available on the server, with a maximum of 8GB:
shared_buffers = 3GB
work_mem = 16MB
maintenance_work_mem = 256MB

# QUERY PLANNING PARAMETERS:
# cost of non-sequentially-fetched disk page. 2 for fast RAID0 disks, higher value for slower disks:
random_page_cost = 2
# cost of a disk page fetch. Value is correlated with random_page_cost. See Warning below. :
seq_page_cost = 2
# effective_cache_size SHOULD be 2/3 of the total memory available on the server
effective_cache_size = 8GB
# effective_io_concurrency is the number of current disk operations. 200 is a good value for SSD.
effective_io_concurrency = 200
checkpoint_completion_target = 0.9

properties random_page_cost and seq_page_cost should have values relative to each other thoroughly set, in order for PostgreSQL query planner to choose the right execution plan.
See PostgreSQL Planner Cost Constants for more details on how to set those values.

If you want to be able to restore live PITR (Point-in-Time Recovery) backup of the database, ensure archiving is activated:

# SHOULD already be the default value:
wal_level = replica
# archiving is off by default, set it to on:
archive_mode = on

Update kernel configuration in file 10-postgresql.conf (typically /etc/sysctl.d/10-postgresql.conf; create the file if it does not exist yet) with the following values:

# KERNEL PARAMETERS:
vm.swappiness=10
vm.zone_reclaim_mode=0
vm.overcommit_memory=2
vm.overcommit_ratio=80
vm.dirty_ratio=40
vm.dirty_background_ratio=30