Performance tuning
Learn how to tune the Bonita Engine to get optimize the performance of your platform.
It assumes that you are familiar with Java threads, concurrent execution, XML, DB connection pools, your DB instance, cache policies, scheduling, connectors, network speed, I/O, Java Virtual Machine configuration, JTA and transaction management.
You need to know how to install and configure Bonita.
This information applies primarily to the Enterprise and Performance edition, though some details also apply to the Teamwork and Efficiency editions.
It is also possible to use gzip compression on your application server to improve performance.
Summary
This section is a summary of the key recommendations from the rest of the page. You can use it as a checklist of things to consider when tuning system performance.
To understand these recommendations in detail, read the sections below.
Two key definitions:
-
The maximum possible number of parallel threads that could be required at any given time is the sum of:
-
the number of workers
-
the number of scheduler threads
-
the number of external API calls
-
-
The processing capacity is the desired number of parallel threads, and is the sum of:
-
the number of workers
-
a percentage of the number of concurrent scheduler threads
-
a percentage of the number of concurrent external API calls
-
Performance tuning checklist of best practises:
-
Access Engine APIs in Local mode whenever possible.
-
Make sure that the network between your client and the Bonita Engine server is fast if you access the APIs remotely.
-
Install your Database on a powerful machine (hardware that meets your DB vendor requirements: sufficient memory, powerful CPU, and fast I/O).
-
Make sure that the network between your Bonita Engine server and your database server is very fast.
-
Set the Work service threadpool and Connector service threadpool to a size according to the usage of your processes.
-
Set the maximum DB connections for the bonitaDS datasource to the desired processing capacity number of parallel threads.
-
Configure a suitable sequence manager range size for your typical process designs and expected volume.
-
Make sure that the maximum DB connections for sequenceManagerDS datasource is more than 1 and is appropriate for the SequenceManager range size configuration.
-
Tune the application cache to your hardware capabilities (if you have more memory available, increase the cache size).
-
Set the basic JVM options to make it fast and well-sized.
-
Optimize your database configuration for your most frequent usage (read or write) and the level of robustness that you want.
-
Configure your transaction manager for the level of robustness that you want.
-
Tune the log levels of Bonita Engine and all its dependencies.
-
Design your processes following the best practises.
-
Add a reasonable number of well-developed event handlers.
-
Tuned the Bonita Engine cron jobs for your needs.
-
If your database is PostgreSQL, follow our recommendations.
Engine access
This section deals with performance impact of your choice of Engine access mode.
There are various ways to access the Engine APIs provided by Bonita Engine. Choose the most suitable access mode for your deployment, requirements, and preferences. The access modes rely on different technologies and have different benefits and drawbacks. In this section, we will describe the performance characteristics of each mode.
Local access
This is undoubtedly the fastest way to access this engine, because it means a direct Java call with nothing additional between client and server.
The deployment constraint is that the client of the engine must be located in the same JVM as the engine server.
Remote access
The remote access modes enable you to have an engine client connected remotely to the engine server.
All remote access modes share a set of common benefits and constraints. There is a loss of performance mainly because of serialization between client and server.
If your client is not located on the same machine than your server JVM, the network becomes an additional potential source of performance reductions to monitor.
In some deployments, it is possible to benefit from the best of both local and remote modes. The engine server access mode is defined per client and does not need to be the same for all clients. If you have a client located in the same JVM as your server, configure it to use the local access mode. You can then configure other clients to use one of the remote modes but you do not penalize the client able to leverage the local access performance. |
HTTP
The HTTP access mode is available using our natively provided bonita-client library. It can also be used from other technologies like PHP, .Net or Javascript (in that case, you need to develop your our client).
We do not guarantee to keep the http protocol stable, so we strongly recommend that you use
|
This mode can be easily used inside a web container like Tomcat or Jetty.
The bonita-client library
-
sends data over the network using the HTTP protocol using the Apache HttpComponents. open source library
-
uses a maximum of 20 connections. To change this value refer to the page Configure connection to Bonita Engine.
Data sent is serialized using a Java library called XStream. This serialization also has a cost.
REST
This method of accessing the Bonita capabilities is not yet integrated as an engine service but exists as a web application service accessed using the Web REST API.
No details are provided here as it is currently out of scope.
In general, the constraints are almost the same as for the HTTP mode, but we do not provide any Java client for this access mode.
Concurrent execution
This section describes some aspects of engine configuration that have a performance impact if there is a high level of concurrent execution.
Before you read this, make sure you are familiar with the engine execution sequence, states, and transactions.
There are two main entry points for load on the engine:
-
API calls coming from outside the engine
-
Engine-generated calls for internal processing, specifically the Work service and the Scheduler service
The Bonita Engine is an asynchronous BPM process engine.
This means that every thread that deals with process execution applies the following rule: do the minimum that makes sense in the current transaction to get to a stable state, and then continue in another transaction inside another thread.
The great benefit of this is that the caller is not locked while the engine processes something that might be long (such as a long sequence of tasks with connectors.).
Client Threads
Client threads are responsible for a large part of the load generated inside the engine.
The number of client threads is related to the number of parallel users.
If you are running your own application, you have one thread if your applicaiton is not multi-threaded, or you have the number of threads you decided to create explicitly in the application or using your own threadpool.
If you are running Bonita Engine inside a container, the maximum number of client threads is defined by a parameter of the container. For example:
-
Apache Tomcat
maxThreads
set in `Tomcat_folder``/conf/server.xml`.
Default value 20. See the Tomcat documentation for information about themaxThreads
parameter.
Work service
The work service is responsible for asynchronously processing execution of process instances. The work service has its own thread pool, which can be configured for each tenant.
This is one of the key configurations to optimize, because even though there are many client threads, client threads are held only for a short time before being released, and then execution flow continues using work service threads.
A thread from the pool of the work service is known as a worker.
The work service is configured in bonita-tenant-community-custom.properties
.
bonita.tenant.work.terminationTimeout=30
bonita.tenant.work.corePoolSize=25
bonita.tenant.work.maximumPoolSize=25
bonita.tenant.work.keepAliveTimeSeconds=60
bonita.tenant.work.queueCapacity=10000
It is very similar to the constructor provided in the default JDK ThreadPoolExecutor.
For a reminder of how the threadpool behaves, see the Queuing section of the
ThreadPoolExecutor documentation.
In the default Bonita configuration, corePoolSize
is equal to maximumPoolSize
because we have observed that the default implementation of the threadpool executor allocates work to available threads using a round robin algorithm.
Therefore, if the maximum is reached, the thread pool size is unlikely ever to reduce to corePoolSize
, because work is always allocated to available threads.
The current implementation of the RejectedExecutionHandler queues the work, and reduces the system load because it does not release the caller (normal behaviour for a BlockingQueue).
After a lot of profiling, we have concluded that having an arbitrarily high number of threads in the work service does not positively impact the performance of the whole system, because it leads to a lot of contentions, mostly on the database (see Database connections).
The size of the threadpool (corePoolSize
in the default configuration) is key, and correlates to the number of process instances the engine can handle in parallel.
In other words, if you want the engine to be capable of handling X process instances concurrently, you should set the corePoolSize
value of the work service to X.
You then need to ensure that your platform infrastructure can handle X concurrent instances, checking that all other engine dependencies including the network and the database are able to process all incoming requests without loss of performance.
Setting a high queueCapacity
limit means that more work can be queued, but can reduce throughput as work is queued rather than causing a new thread to be created.
It is essential to ensure that the queue never becomes full (queueCapacity
is never reached).
If the queue becomes full, the application restarts in order to force the engine to generate all work from the database. This means that work is lost.
Connector service
The connector service executes connectors. To improve tenant isolation (and to protect against denial-of-service attacks), the default implementation of the connector service has its own thread pool and requires executes connectors in a separate thread from the worker.
The configuration of the thread pool of this service is independent of the configuration of the work service.
If you have processes that use a lot of connectors, then you can have more threads to execute connectors. See Connector execution page for details on how connectors are executed.
The Connector service is configured in bonita-tenant-community-custom.properties
and bonita-tenant-sp-custom.properties
(cf platform setup)
Community:
bonita.tenant.connector.queueCapacity=10000
bonita.tenant.connector.corePoolSize=5
bonita.tenant.connector.maximumPoolSize=100
bonita.tenant.connector.keepAliveTimeSeconds=100
Subscription only:
bonita.tenant.connector.timeout=300
For details of these parameters, see Work service.
In addition, connectors longer that 10 seconds produces a log at warning level named : org.bonitasoft.engine.core.connector.impl.ConnectorExecutionTimeLogger
.
This log contains all references to find exactly which connector is slow.
Another log at the debug level prints all input parameters of this connector.
Here is a sample log produced using a connector that does a Thread.sleep(15000)
WARNING: Connector 15 sleep with id 20002 with class org.mycompany.connector.SleepImpl of process definition 6587226372021992905 on element flowNode with id 20003 took 15001 ms. FINE: Input parameters of the connector with id 20002: {seconds: [15]}
The 10 seconds threshold can be changed in the configuration file bonita-tenant-community-custom.properties
bonita.tenant.connector.warnWhenLongerThanMillis=10000
Scheduler service
The Scheduler service is responsible for executing jobs.
A job is executed inside a thread of the scheduler service.
There are various kinds of jobs, some resulting from internal requirements such as API session cleaning, or batch deletion of a table row, and some related to process design such as BPMN2 events.
The Bonita Engine Scheduler service uses the Quartz Scheduler. Quartz takes the size of the threadpool as an input parameter. Quartz uses threads to execute jobs concurrently.
The Scheduler service configuration is in bonita-platform-community-custom.properties
.
You can configure:
bonita.platform.scheduler.quartz.threadpool.size=5
bonita.platform.scheduler.batchsize=1000
Database connections
Two datasources are defined:
-
bonitaSequenceManagerDS is used for distributing ID requests
-
bonitaDS is used for everything else
Note that the sum of the maximum values configured for bonitaDS and bonitaSequenceManagerDS should be less than or equal to the maximum number of simultaneous connections allowed to your database.
bonitaSequenceManagerDS
This datasource needs only a few connections: between 5 or 10% of bonitaDS number should be sufficient. However, this is closely correlated to the range size.
bonitaDS
This datasource requires a higher value, because Bonita Engine stores almost everything in the database.
This means that every single thread from any of the entry points requires a database connection through bonitaDS.
To make sure that this datasource is not a bottleneck, define the maximum number of database connections to be equivalent to the desired number of parallel processing threads.
The desired number of parallel processing threads is the sum of the number of workers (see Work service) plus a percentage of the number of scheduler threads
(see Scheduler Service) plus a percentage of the number of concurrently external API calls (see Client threads).
Datasources settings
You need to configure the maximum pool size for datasources (the following paths are for bundle users):
For Tomcat, edit file setup/tomcat-templates/bonita.xml
:
-
For bonitaSequenceManagerDS, set
maxTotal=”yourvalue”
. -
For RawBonitaDS, set
maxTotal=”yourvalue”
. -
If necessary, for the Business Data feature, do the same for the datasources 'RawBusinessDataDS' and 'NotManagedBizDataDS'.
Volume
This section deals with some aspects of engine configurations that have a performance impact in the case of high volume.
Sequence manager
Bonita Engine manages a dedicated sequence for each table for ID generation. This implementation allows fast delivery of IDs and a single point of usage inside the application: the persistence service.
The sequence manager keeps in memory a range of reserved IDs by table.
This range size is configurable by sequence so that it can be adapted to the volume you have.
The bigger a range is, the less frequently the sequence manager will have to query the database for a new range, because it is managed in memory for as long as possible.
However, all the IDs that are reserved in memory are lost when the JVM is shut down, so the number should not be too big or you might reach Long.MAX_VALUE too quickly.
The sequence manager allows you to set the range size for each sequence and a default range size value, which is applied to any sequence that does not have a specific range defined. If you want to tune these values, you have to understand the correlation between them.
For example, if you have an average of 20 steps in your process, then it would be reasonable to set the ActivityInstance range size
to be 20 times bigger than the ProcessInstance range.
The sequence manager configuration is in bonita-platform-community-custom.properties
.
The sequence manager has its own database connection.
This should be appropriately sized for the number of times the sequence manager will query the database, which is a consequence of the range size values. See Database connections.
Persistence cache
For the Teamwork, Efficiency, Performance and Enterprise editions, Bonita Engine has a cache providing a persistence layer using Hibernate caching.
EhCache configuration for this persistence layer is defined in a file named bonita-platform-hibernate-cache.xml.notused
and bonita-tenant-hibernate-cache.xml.notused
.
To apply the configuration of those files, remove the '.notused' suffix.
It is possible to modify the cache settings in those files for each kind of object.
Before going into production, we encourage to finely tune the "Level-2" object cache in a pre-prod environment:
-
activate Hibernate cache statistics by setting to true the parameter bonita.platform.persistence.generate_statistics in file bonita-platform-community-custom.properties
-
activate logs at INFO level:
<logger name="org.bonitasoft.engine.persistence" level="INFO"/>
<logger name="com.bonitasoft.engine.persistence" level="INFO"/>
-
run load tests to simulate a production environment
-
analyse the "2nd Level Cache Ratio" log messages generated, combined with the "soft-locked cache entry was expired" warnings messages to change the configuration in file bonita-tenant-hibernate-cache.xml.
For instance, if on entity org.bonitasoft.engine.core.document.model.impl.SDocumentImpl, the "soft-locked cache entry was expired" warnings message occurs, it means the size of the maxElementsInMemory parameter must be increased, provided it is a reasonable memory size and provided the "2nd Level Cache Ratio" is not low for this element. If the "2nd Level Cache Ratio" is low or even 0, it means the cache is never used to read several times the same entity, which means the timeToLiveSeconds parameter should be increased, or that the cache should be completely deactivated for this entity.
Below is an example of a "soft-locked cache entry was expired" warning message:
WARNING: Cache org.bonitasoft.engine.core.process.instance.model.impl.SFlowNodeInstanceImpl Key org.bonitasoft.engine.core.process.instance.model.impl.SFlowNodeInstanceImpl#org.bonitasoft.engine.persistence.PersistentObjectId@25505ff
Lockable : null
A soft-locked cache entry was expired by the underlying Ehcache. If this happens regularly you should consider increasing the cache timeouts and/or capacity limits
Application cache
Bonita Engine uses an application cache to store specific objects. The default implementation of this service relies on EhCache. It is configured in these files:
-
bonita-platform-community-custom.properties
-
bonita-tenant-community-custom.properties
-
bonita-platform-sp-cluster-custom.properties
-
bonita-tenant-sp-cluster-custom.properties
The following cache configurations can be defined:
Configuration | Purpose |
---|---|
connectorCacheConfig |
stores connector implementations for a given connector definition |
processDefCacheConfig |
stores process definition objects |
userFilterCacheConfig |
stores user filter implementations for a given user filter definition |
migrationPlanCacheConfig |
not yet used |
breakpointCacheConfig |
not yet used |
groovyScriptCacheConfig |
stores compiled versions of Groovy scripts |
synchroServiceCacheConfig |
used by the benchmark test infrastructure (and has no meaning outside of it) |
transientDataCacheConfig |
stores transient data |
platformCacheConfig |
used to store platform object, which contains general platform information such as the version, or start date |
parameterCacheConfig |
stores process parameters |
Java Virtual Machine
You can configure the JVM settings for the engine to tune performance.
Check the JVM documentation for details of the available settings.
Notably, we recommend you to set the initial (-Xms
) and maximum (-Xmx
) heap sizes to the same value.
This reduces the likelihood of the JVM garbage collector starting.
While the garbage collector is running, it prevents creation of new objects, which slows down the application server.
Garbage Collector
We recommend to update the Garbage collector parameters to use the following ones (UseParallelGC
has been proven more eficient than UseParNewGC
in our experience), depending on wether you use Java 8 or Java 11:
-
Edit file
setup/tomcat-templates/setenv.sh
-
To add in the line
CATALINA_OPTS=
:-
If you use Java 8 add:
-XX:+UseParallelGC -XX:ParallelGCThreads=10 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:CATALINA_BASE/logs/gc-$(date +%Y_%m_%d-%H_%M).log
-
If you use Java 11 add:
-XX:+UseParallelGC -XX:ParallelGCThreads=10 -Xlog:gc*:file=CATALINA_BASE/logs/gc-$(date +%Y_%m_%d-%H_%M).log:time,uptime,hostname,pid:filecount=5,filesize=20M
-
Hardware and network
This section deals with performance impact of hardware elements.
Bonita performance is very correlated to the database connectivity and its behavior.
Almost everything (API call, internal processing using workers, jobs scheduling, and so on) requires a database access.
Two elements are critical: network latency, as in most cases your database is located on another server, and the I/O of your hard drives.
In case of issues, you should monitor these two elements and consider improvements. For example:
-
locate your database in the same datacenter as the Bonita Engine, using gigabit network connections
-
use SSD hard drives, and RAID configuration with striping
Database, Transaction Manager, and logs
This section is a reminder about some of the main dependencies Bonita Engine has that have a strong impact on the performance of the whole system.
Bonita Engine relies on several other components that each have their own performance tuning options.
Some of them are key for the system and you should pay a lot of attention to them.
In most cases, the key things to consider are the database, transaction manager, and logs.
Database
Bonita Engine uses the database heavily, so in consequence a slow database makes the engine slow.
It is essential that the hardware configuration of the server hosting the DB is powerful, considering resources like CPU, memory or others depending on your database instance.
In addition to this, make sure that your database instance is well configured.
Most database software provides many options for tuning, and some of them are easy to set up.
Others may be more difficult and present choices between robustness and performance, fast read or fast write, etc.
Your database configuration must be correlated with the Bonita Engine usage pattern.
To find the right characteristic to optimize, one good starting point is to consider whether you are creating a lot of process instances (in which case optimize database writes) or you are executing a lot of read queries like getTaskList
(in which case optimize database reads).
Specific PostgreSQL performance tuning is given as a database tuning reference.
Transaction manager
Bonita Engine is natively compatible with the Java Transaction API. This means transaction management relies on a transaction manager.
Bonita Platform embed Narayana, an open source transaction manager.
It uses the following configuraton file server/conf/jbossts-properties.xml
.
The most common configuration to change here would be com.arjuna.ats.arjuna.coordinator.defaultTimeout
that is the timeout for transactions.
More details on the configuration can be found in the Narayana documentation.
Logs
In general, increasing the log level is useful for debugging but has a performance cost.
With this in mind, define the log level for technical logs, queriable logs and archives.
Remember that Bonita Engine dependencies also have their own log and debug options that may impact strongly the system performance. Be sure to configure these appropriately.
Connector time tracker
It is now possible to track the duration of actions in a connector using a new time tracker. The tracker service tracks several connector lifecycle operations.
This service can impact performance so is disabled by default.
It is configured by editing the following parameters in bonita-tenant-community-custom.properties
.
## Time tracker
#bonita.tenant.timetracker.startTracking=false
#bonita.tenant.timetracker.maxSize=1000
#bonita.tenant.timetracker.flushIntervalInSeconds=30
#bonita.tenant.timetracker.csv.activateAtStart=true
#bonita.tenant.timetracker.csv.folder=$ {java.io.tmpdir}
#bonita.tenant.timetracker.csv.separator=;
#bonita.tenant.timetracker.memory.activateAtStart=false
#bonita.tenant.timetracker.memory.maxSize=1000000
To activate connector time tracking:
-
Uncomment all the previous lines except
## Time tracker
. -
Change the value of
startTracking
fromfalse
totrue
.
The other parameters can be left at their default value, left commented, or set to the desired value. What each of them does:
-
maxSize
maximum of records that will be saved by the time tracker before a flush. If the maximum number of records is reached before the scheduled flush, the older ones are discared. To avoid the loss of information, a number sufficiently big in comparison withflushIntervalInSeconds
should be chosen -
flushIntervalInSeconds
the interval beetween two flushes on the timetracker thread. -
csv.activateAtStart
wether to save the result of the timetracker into a csv file. -
csv.folder
the folder where to save the csv file. -
csv.separator
the separator character in the csv file -
memory.activateAtStart
wether to save the result of the timetracker in memory. -
memory.maxSize
maximum amount of records saved in memory. If the maximum number of records is reached before the scheduled flush, the older ones are discared. To avoid the loss of information, a number sufficiently big in comparison withflushIntervalInSeconds
should be chosen
The non-relevant options will be ignored at execution. Note that memory
and csv
can both be activated at the same time.
Process design, event handlers, and cron jobs
Process design
There are several things you can do during the process design to reduce performance overheads.
This is mostly related to reducing usage of extension points when possible.
Consider carefully your usage of connectors, groovy scripts, XML and serializable data.
Event handlers
Events handlers are extensions of the engine configuration.
You can add event handlers for several purposes and you can configure which events you want to catch.
We strongly recommend that you add only appropriate handlers and carefully code the handler filters to handle only those events that you are interested in.
BPMN Timers execution
Bonita Engine uses the Scheduler service to trigger timers.
The Bonita Scheduler service implementation uses the Quartz Scheduler. Some quartz properties can be modified to fine tune quartz jobs execution. These properties can be found in bonita-platform-community-custom.properties
.
org.quartz.jobStore.misfireThreshold
org.quartz.jobStore.maxMisfiresToHandleAtATime
org.quartz.jobStore.acquireTriggersWithinLock
org.quartz.scheduler.batchTriggerAcquisitionMaxCount
org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow
Details on these properties can be found in the Quartz documentation.
They are not read subsequently, so changing the values in bonita-tenant-community-custom.properties
after the Engine has been started has no effect on Quartz.
For value definition, and information about how to update the Quartz trigger tables, see the Quartz documentation about Cron Triggers.
PostgreSQL performance tuning
Here is Bonita advice to finely tune PostgreSQL database server performance.
In this example, we assume you have:
-
12Gb of RAM
-
fast SSD storage
Update memory configuration in file postgresql.conf
(typically /etc/postgresql/11/main/postgresql.conf
) with the
following values:
# MEMORY PARAMETERS:
# shared_buffers SHOULD be set to 1/4 of the total memory available on the server, with a maximum of 8GB:
shared_buffers = 3GB
work_mem = 16MB
maintenance_work_mem = 256MB
# QUERY PLANNING PARAMETERS:
# cost of non-sequentially-fetched disk page. 2 for fast RAID0 disks, higher value for slower disks:
random_page_cost = 2
# cost of a disk page fetch. Value is correlated with random_page_cost. See Warning below. :
seq_page_cost = 2
# effective_cache_size SHOULD be 2/3 of the total memory available on the server
effective_cache_size = 8GB
# effective_io_concurrency is the number of current disk operations. 200 is a good value for SSD.
effective_io_concurrency = 200
checkpoint_completion_target = 0.9
properties |
If you want to be able to restore live PITR (Point-in-Time Recovery) backup of the database, ensure archiving is activated:
# SHOULD already be the default value:
wal_level = replica
# archiving is off by default, set it to on:
archive_mode = on
Update kernel configuration in file 10-postgresql.conf
(typically /etc/sysctl.d/10-postgresql.conf
; create the file
if it does not exist yet) with the following values:
# KERNEL PARAMETERS:
vm.swappiness=10
vm.zone_reclaim_mode=0
vm.overcommit_memory=2
vm.overcommit_ratio=80
vm.dirty_ratio=40
vm.dirty_background_ratio=30