Rescue broken asciinema-server upgrade (PostgreSQL 12 => 14)

How to rescue broken asciinema-server upgrade that caused by incompatible files in the volume of the old PostgreSQL 12 container with the latest upstream version
On this page

Today, I’m experiencing some problem with my recent self-hosted asciinema server upgrade. After following the upgrade process according to the asciinema-server documentation page on GitHub, the phoenix and postgresql containers failed to run and kept restarting.

I’m trying to find information on what causes this to happen. From the container log, I got the following error:

(DBConnection. ConnectionError) connection not available and request was dropped from queue after xxxxms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

1By tracking down slow queries and making sure they are running fast enough
2Increasing the pool_size (albeit it increases resource consumption)
3Allow requests to wait longer by increasing :queue_target and :queue_interval
4See DBConnection. start_link/2 for more information

Then from the log of the posgreSQL container itself, I found information that the files in the volume of the PostgreSQL container are not compatible with the one that currently running :

FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 14

So, the root cause likely due database upgrade from PostgreSQL 12 to PostgreSQL 14.

From these informations, I decided to:

  1. Doing a temporary downgrade so I can back up the previous database.
  2. Uninstall and reinstall the appropriate version of PostgreSQL from the master repository.
  3. Restore the database that I have backed up and hope that the restore process can run smoothly.

And it worked!

Rescue process

Before begining, I assume that you follow the official installation process.

Enter the asciinema-server directory, and stop all containers associated with asciinema-server by running docker-compose down.

Download the latest asciinema-server docker image:

1docker pull asciinema/asciinema-server

Download the asciinema config from upstream and merge it into your branch (if you haven’t already):

 1git fetch origin
 2git merge origin/master
 3
 4# The commands below are optional
 5# and depending on your initial asciinema-server installation
 6git stash
 7git stash pop
 8git add .
 9git commit -m "Local upgrade"
10git merge origin/master

1. Downgrade PostgreSQL to previous running version and back up the old data

After that, edit docker-compose.yml, temporarily revert postgres image from version 14 to your previous version (for me, my previous version was at version 12).

1version: '2'
2
3services:
4   postgres:
5     image: postgres:12-alpine
6     container_name: asciinema_postgres
7     ### blah blah blah

This is necessary so that we can dump the PostgreSQL database.

Turn on PostgreSQL, but leave the other asciinema-server services off.

1docker-compose up -d postgres

Perform a backup by running the following command:

1docker exec -it <DOCKER_CONTAINER_ID> pg_dump postgres -U postgres > asciinemadump.sql

Note: Your DOCKER_CONTAINER_ID information can be obtained from docker ps command.

After the database dump process is complete, turn off the PostgreSQL container by running this command:

1docker-compose down

delete all volume data used by the PostgreSQL container (default is ./volumes/postgres).

IMPORTANT: I recommend that you back up the directory before permanently deleting it!

2. Run the upstream version of PostgreSQL

After that, delete the docker volume for PostgreSQL 12 and use the same PostgreSQL version from upstream (v14).

Again, edit docker-compose.yml and revert the config according to the upstream version:

1version: '2'
2
3services:
4   postgres:
5     image: postgres:14-alpine
6     container_name: asciinema_postgres
7     ### blah blah blah

Turn on PostgreSQL 14, but leave the other asciinema-server services disabled.

1docker-compose up -d postgres

3. Restore PostgreSQL backups

After PostgreSQL that is compatible with the upstream version is running, do the database restore process that we have backed up before.

Copy the asciinemadump.sql file to the Postgres volume ( default: ./volumes/postgres) and change the file permissions so that the root user inside the container can read the file.

Then run the following command to restore the database:

1docker exec -it <DOCKER_CONTAINER_ID> psql -d postgres -U postgres -f /var/lib/postgresql/data/asciinemadump.sql

Note: Your DOCKER_CONTAINER_ID will be different from the previous DOCKER_CONTAINER_ID. Make sure to use the correct docker container ID.

4. Run asciinema-server as usual

After that, run all the asciinema-server services as usual (docker-compose up -d), and you should be able to access your self-hosted asciinema-server again.