Hot upgrades and downgrades¶
A word of caution¶
Hot upgrades are one of the most powerful capabilities of OTP, but with that power comes a lot of associated complexity. Hot upgrades are so impressive because of the technical feat required to swap out code at runtime, and while OTP makes this process quite easy to accomplish, it also expects you to fully understand what you are asking it to do. In other words, you need to be very aware of how hot upgrades are performed, and how to manage change in your applications such that you can perform upgrades seamlessly.
Distillery makes hot upgrades deceptively simple to use, because it
automatically generates appups for you (see Appups for more).
This may lead you to believe that you can let Distillery handle this process
entirely for you, and you need to understand that this is not the case.
Distillery does this for you in order to make things easier on you, by only
requiring you to make small adjustments to the appups before building a release,
rather than having to write every appup from scratch. It is still on you to
review these generated appups, understand when they are correct, or when
something has been missed, or when you simply need to do your part in the
process (such as implementing
code_change in processes you want upgraded).
Always remember though, that if an upgrade goes badly, you can always reboot your application and get back to a good state, so if in doubt, just perform a rolling release - only the most critical of applications will require hot upgrades, and in those cases it will be worth your while to understand them in great detail. I would recommend using rolling releases always, rather than undertaking hot upgrades, but when you need them, they are there.
It is not possible to use hot upgrades and
include_erts: false in
conjunction with one another. This is due to how
information about releases as they are unpacked and installed. ERTS must be
packaged in the release in order for hot upgrades/downgrades to properly
You cannot perform upgrades from within the
_build directory -
you must deploy the tarballs as described in the
Walkthrough. Hot upgrades/downgrades
mutate the release directory, and some of the files which are required for
building releases will be missing if you do a build after upgrading a
release there, resulting in errors when you attempt to upgrade/downgrade
from that release. Always deploy to another local directory (for example,
When building upgrade releases, Distillery will ensure that all modified
applications have appups generated for them. Appups are what tell
how to generate the
relup file, which contains low-level instructions for the
release handler, so that it knows how to load/remove/change modules in the
system during a hot upgrade.
Without an appup, an upgrade cannot succeed, because the release handler will
not know how to upgrade that application. Distillery takes great care to provide
good default instruction sets for the appups it generates, by ordering
instructions based on module dependencies, and whether things have been added,
removed, or modified; it makes sure to take advantage of special processes (such
GenServer and friends) by using the
code_change handler, and even handles
custom special processes started via
:proc_lib by leveraging the
However, while Distillery’s appup generator is quite good, it can’t be perfect
for all applications, or all situations. Any time you modify the internal state
of a process, say the state parameter of a
GenServer, you will need to make
sure that the new version of the code knows how to transform the old state to
the new format, by implementing
code_change, or what will happen is that the
new module will start executing using the old state, and things will blow up in
your face. It is important that you be aware of how your application has
changed, and ensure that you handle cases like these, either in your own code,
or by using instructions in the appup file Distillery generates (or even provide
When you upgrade from one release version to another, if something goes wrong during the upgrade past the point of no return, the node will be restarted running the original version of the code, and if something goes wrong early, the changes will be rolled back. In both cases, the error will be printed to standard output.
Since I have received many questions on this topic, I want to take a moment to discuss hot upgrades in conjunction with migrations.
You need to consider migrations and application upgrades as two separate, distinct deployments. Migrations should be backwards compatible with the old version of the application, and should be deployed in advance of application upgrades, so that you have a chance to vet, and roll back if necessary, the migrated changes. Once the migration has been applied and confirmed to be good, you then proceed with applying your application upgrade. If a problem with the new application code occurs, you can then safely roll back the application without needing to also roll back the migration (if even possible).
The above strategy does require that you have strict change management, so that you incrementally apply changes, rather than try to do them all at once. This has implications in your application code as well, since you need to usually allow for two different code paths (one to support old schemas and one for new). This seems onerous, but in practice it makes deployments easier to manage, and changes can be introduced at a steady pace.
I say all of this because I see people wanting to have OTP apply migrations as part of the hot upgrade process, but that is not what it was designed for, and is not intended to make any guarantees about external systems - it only makes guarantees about how it applies upgrades to application code. Trying to mix these concerns together will only lead to pain, so avoid doing so at all cost!