Today, I’m experiencing some problem with my recent self-hosted asciinema-server upgrade. After following the upgrade process according to the asciinema-server documentation page on GitHub, the
postgresql containers failed to run and kept restarting.
I’m trying to find information on what causes this to happen. From the container log, I got the following error:
(DBConnection. ConnectionError) connection not available and request was dropped from queue after xxxxms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1By tracking down slow queries and making sure they are running fast enough 2Increasing the pool_size (albeit it increases resource consumption) 3Allow requests to wait longer by increasing :queue_target and :queue_interval 4See DBConnection. start_link/2 for more information
Then from the log of the posgreSQL container itself, I found information that the files in the volume of the PostgreSQL container are not compatible with the one that currently running :
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 14
So, the root cause likely due database upgrade from PostgreSQL 12 to PostgreSQL 14.
From these informations, I decided to:
- Doing a temporary downgrade so I can back up the previous database.
- Uninstall and reinstall the appropriate version of PostgreSQL from the master repository.
- Restore the database that I have backed up and hope that the restore process can run smoothly.
And it worked!
Before begining, I assume that you follow the official installation process.
asciinema-server directory, and stop all containers associated with
asciinema-server by running
Download the latest
asciinema-server docker image:
1docker pull asciinema/asciinema-server
Download the asciinema config from upstream and merge it into your branch (if you haven’t already):
1git fetch origin 2git merge origin/master 3 4# The commands below are optional 5# and depending on your initial asciinema-server installation 6git stash 7git stash pop 8git add . 9git commit -m "Local upgrade" 10git merge origin/master
1. Downgrade PostgreSQL to previous running version and back up the old data
After that, edit
docker-compose.yml, temporarily revert postgres image from version
14 to your previous version (for me, my previous version was at version
1version: '2' 2 3services: 4 postgres: 5 image: postgres:12-alpine 6 container_name: asciinema_postgres 7 ### blah blah blah
This is necessary so that we can dump the PostgreSQL database.
Turn on PostgreSQL, but leave the other
asciinema-server services off.
1docker-compose up -d postgres
Perform a backup by running the following command:
1docker exec -it <DOCKER_CONTAINER_ID> pg_dump postgres -U postgres > asciinemadump.sql
DOCKER_CONTAINER_IDinformation can be obtained from docker
After the database dump process is complete, turn off the PostgreSQL container by running this command:
delete all volume data used by the PostgreSQL container (default is
IMPORTANT: I recommend that you back up the directory before permanently deleting it!
2. Run the upstream version of PostgreSQL
After that, delete the docker volume for PostgreSQL 12 and use the same PostgreSQL version from upstream (
docker-compose.yml and revert the config according to the upstream version:
1version: '2' 2 3services: 4 postgres: 5 image: postgres:14-alpine 6 container_name: asciinema_postgres 7 ### blah blah blah
Turn on PostgreSQL 14, but leave the other asciinema-server services disabled.
1docker-compose up -d postgres
3. Restore PostgreSQL backups
After PostgreSQL that is compatible with the upstream version is running, do the database restore process that we have backed up before.
asciinemadump.sql file to the Postgres volume ( default:
./volumes/postgres) and change the file permissions so that the root user inside the container can read the file.
Then run the following command to restore the database:
1docker exec -it <DOCKER_CONTAINER_ID> psql -d postgres -U postgres -f /var/lib/postgresql/data/asciinemadump.sql
DOCKER_CONTAINER_IDwill be different from the previous
DOCKER_CONTAINER_ID. Make sure to use the correct docker container ID.
4. Run asciinema-server as usual
After that, run all the
asciinema-server services as usual (
docker-compose up -d), and you should be able to access your self-hosted asciinema-server again.