After deploying a BDB JE HA application, you may later want to upgrade to a new version. Berkeley DB JE supports hot upgrade of a replication group, by allowing mixed version operation. That is, replication nodes running the newer software version can inter-operate with older version nodes and both can be available for user operations. However, in some cases, there are certain constraints to performing such a hot upgrade.
Each release of Berkeley DB JE is tied to a given log file on-disk format. Log file formats do not necessarily change every release. The Change Log for each release specifies whether the log file format has changed or not. There are no restrictions on upgrades across releases that use the same log file format. For best performance and to take advantage of bug fixes, ensure all nodes in a replication group run with the same BDB JE version during normal operations. Occasionally, a new release of Berkeley DB JE includes a log file format change. The constraints that apply to upgrades which introduce new log file formats are explained in this section.
In a replication group, the Master transmits log records that must be read by the replicas. If a group is operating with mixed version nodes, the Master must be running an older or equal version to that of the Replicas, so that all Replicas are able to understand the replication stream. To support this requirement, BDB JE always elects a node that runs with the lowest log version as the Master, until the majority of nodes in this group are upgraded. After that point, JE can elect a node with any version as the Master.
User specified node priorities can overrule this default policy, so you should consider their impact when upgrading.
The recommended steps for upgrading Berkeley DB JE HA when log file formats have changed are as follows:
Bring all Replicas up to date with the Master. That is, all Replicas must have the same environment content as the Master. You can ascertain this by using the DbGroupAdmin.dumpGroup() utility, or programmatically using the ReplicaConsistencyPolicy.
Perform the following upgrade procedures on each of the environments that are part of the replication group.
Shut down the old version of the application.
Install the new BDB JE jar file.
Restart the application.
Upgrade each individual Replica and restart them to join the replication group.
Shut down the Master node and upgrade that node last.
When the Master is shut down, one of the Replicas running the new version is elected as the Master. From that point onward, the Master writes data in the new format. After the former Master is upgraded it is restarted as a Replica.
During a hot replication upgrade:
Adding new (empty) nodes to the replication group is not allowed while nodes are running at different versions. Add new replication nodes after the entire group is upgraded.
Upgrading BDB JE 4.0 directly to BDB JE 5 and higher versions is prohibited. Upgrade BDB JE 4.0 to BDB JE 4.1 first, and then upgrade BDB JE 4.1 to higher versions. There is no constraint if you upgrade from BDB JE 4.1 or later versions to a higher BDB JE version.
There are exceptions that you may run into during the upgrade process. The following exceptions may be thrown when a replication group node is restarted during the upgrade, and a ReplicatedEnvironment object is instantiated:
This exception can be thrown by a Replica when its log is ahead of the current Master and the Replica is unable to rollback without a recovery. As a consequence, one or more of the most recently committed transactions may need to be rolled back, before the Replica can synchronize its state with that of the current Master. This exception can also be thrown if the current Master crashes. To solve this exception restart the ReplicatedEnvironment with the new JE version.
During synchronization, a Replica that has a newer log, may have to roll back a number of committed tranactions. If the number of rolled back transactions exceeds the limit defined by TXN_ROLLBACK_LIMIT, the Replica throws a RollbackProhibitedException. To solve this exception you may have to truncate logs manually by using DbTruncateLog and restart the ReplicatedEnvironment with the new JE version.
EnvironmentFailureException is thrown due to log version incompatibility between the Master and the Replica. This exception is thrown with the message:
"Incompatible log versions. Feeder log version: xxxx, Feeder JE version: xxxx, Replica log version: xxxx, Replica JE version: xxxx".
To solve this exception restart the ReplicatedEnvironment with the new JE version.