Eureka! How to use Docker Compose, External Volumes, and Volume Options to Persist Data

precursor: this is about getting docker compose, docker volumes, and volume drivers to let you persist data in a user defined path (not the docker virtual path). so if you want to use compose, volumes, and have it write to a location like your development folder, read-on.

I’ve been playing around more with software development lately and decided to build some flask apps. I think I learn the most when going through a class and getting stuck on something that isn’t covered by the instructor but something I want to make happen. Well I hit a barrier when transitioning from docker run to docker compose.

When working in docker run the basic docker commands I ran were:

Build the Container: docker build -t miniblog:latest .
Run an elastic container: docker run --name elasticsearch -d -p 9200:9200 -p 9300:9300 -v /host/path/for/elasticdata:/usr/share/elasticsearch/data -e "discovery.type=single-node" --rm docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
Run a mysql container: docker run --name mysql -d -v /host/path/for/mysql/data:/var/lib/mysql -e MYSQL_RANDOM_ROOT_PASSWORD=yes -e MYSQL_DATABASE=miniblog -e MYSQL_USER=miniblog -e MYSQL_PASSWORD=<db_pass chnge this> --rm mysql/mysql-server:5.7
Then Run the docker commnad with -e variables defined: docker run --name miniblog -d -p 8000:5000 -e SECRET_KEY= -e MAIL_SERVER= -e MAIL_PORT= -e MAIL_USERNAME= -e MAIL_PASSWORD= -e MS_TRANSLATOR= --link mysql:dbserver -e DATABASE_URL=mysql+pymysql://<db_user>:<db_pass>@dbserver/<db_name> --rm miniblog:latest

This worked well but was three separate commands to run and wasn’t elegant, so I decided to switch it to compose.

The issue I hit with docker compose was related to how to use a volume which survived a docker compose down while also letting me define the host path to that location. I know we can let docker manage it, but I like it all in a single development workspace so I can bind with a run or use compose. Well I hit a myriad of issues when trying to do this but ultimately found a solution which required defining an external volume before running docker compose up . Here are the steps and the .yaml…

Making Docker Compose External Volumes work with User Defined Paths

Have your .env file defined which is referenced by the ${} section in the .yaml
Create a local volume in the cli: docker volume create -d local -o type=none -o o=bind -o device=/path/to/elastic-data elastic-data-miniblog and docker volume create -d local -o type=none -o o=bind -o device=/path/to/mysql-data mysql-data-miniblog. This step can’t be done in the .yaml file as you cannot define the driver and options while also setting external: true. The driver options were particular messy and pieced together from much crawling around developer forums.
Define volumes in .yaml and services (see below). In the services under volumes you map the name of the volume to the path in the container and under the volumes section near the bottom you define a named volume, set external to true, which references the volume previously made.

version: '3.8'
services:
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
        networks:
            - privnet
        environment:
            discovery.type: single-node
        ports:
            - 9200:9200
            - 9300:9300
        volumes:
            - elastic-data-miniblog:/usr/share/elasticsearch/data
        container_name: elasticsearch
    mysql-db:
        image: mysql/mysql-server:5.7
        networks:
            - privnet
        environment: 
            MYSQL_RANDOM_ROOT_PASSWORD: 1
            MYSQL_DATABASE: ${MYSQL_DATABASE} 
            MYSQL_USER: ${MYSQL_USER}
            MYSQL_PASSWORD: ${MYSQL_PASSWORD}
        volumes:
            - mysql-data-miniblog:/var/lib/mysql  
        container_name: mysql-db 
    miniblog:
        image: miniblog:0.10
        pull_policy: always
        networks:
            - privnet
        environment:
            SECRET_KEY: ${SECRET_KEY}
            MAIL_SERVER: ${MAIL_SERVER}
            MAIL_PORT: ${MAIL_PORT}
            MAIL_USERNAME: ${MAIL_USERNAME}
            MAIL_PASSWORD: ${MAIL_PASSWORD}
            MS_TRANSLATOR_KEY: ${MS_TRANSLATOR_KEY} 
            DATABASE_URL: ${DATABASE_URL}
            ELASTICSEARCH_URL: ${ELASTICSEARCH_URL}
        ports:
            - 8000:5000
        container_name: miniblog
        depends_on:
            - "mysql-db"
            - "elasticsearch"
networks:
    privnet:
volumes:
    mysql-data-miniblog:
        external: true
    elastic-data-miniblog:
        external: true

Done

So this finally worked by first creating an external volume in the docker command line and marking the named volume as external. After much pain trying to get mysql write written to a file location while using docker compose (and getting the changes to persist after a docker compose down) at last the data persisted with mysql writes – just make sure you don’t comment out the db connection string 😉