Cannot connect to db host within docker containers from api-service to db-service in order to do migration using goose in golang

Question

goose is the migration tool that helps me to run all the *sql files and run up queries within database. I want to automate migrations(create tables and stuff) using this tool within the docker container on my api service. The problem is when docker runs command "goose run" it gets an error -goose run: dial tcp: lookup db on 192.168.63.6:53: no such host.

docker-compose

services:
  db:
    build: ./db
    volumes:
      - ./db/pgdata:/pgdata
    image: postgres
    ports:
      - "5432"
    restart: always
    environment:
      - POSTGRES_USER=user
      - POSTGRES_DB=dbname
      - POSTGRES_PASSWORD=123
      - PGDATA=/pgdata

  api:
    build:
      context: ./api
    restart: always
    volumes:
      - ./api:/go/src/github.com/gitlees/todoapp/api
    ports:
      - "5000:8080"
    links: 
      - "db"

Api Dockerfile

RUN go get -u github.com/pressly/goose/cmd/goose

RUN goose postgres "host=db user=user dbname=dbname sslmode=disable password=123" up

Answer 1

First of all, let us have a deeper look into a Dockerfile. I have set up a repository for demo purposes for this question, too.

# We use a so called two stage build.
# Basically this means we build our go binary in one image
# which has the go compiler and all the required tools and libraries.
# However, since we do not need those in our production image,
# we copy the binary created into a basic alpine image
# resulting in a much smaller image for production.

# We define our image for the build environment...
FROM golang:1.11-alpine3.8 as build

# ...and copy our project directory tp the according place.
COPY . "/go/src/github.com/mwmahlberg/so-postgres-compose"
# We set our work directory...
WORKDIR /go/src/github.com/mwmahlberg/so-postgres-compose
# ...and add git, which - forever reason, is not included into the golang image.
RUN apk add git

# We set up our dependency management and make sure all dependencies outside
# the standard library are installed.
RUN set -x && \
    go get github.com/golang/dep/cmd/dep && \
    dep ensure -v
# Finally, we build our binary and name it accordingly    
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /apiserver

# Now for the second stage, we use a basic alpine image...
FROM alpine:3.8
# ... update it...
RUN apk --no-cache upgrade
# .. and take the binary from the image we build above.
# Note the location: [/usr]{/bin:/sbin} are reserved for
# the OS's package manager. Binaries added to an OS by
# the administrator which are not part of the OS's package
# mangement system should always go below /usr/local/{bin,sbin}
COPY --from=build /apiserver /usr/local/bin/apiserver
# Last but not least, we tell docker which binary to execute.
ENTRYPOINT ["/usr/local/bin/apiserver"]

The last line should actually do the trick: ENTRYPOINT specifies the command to be executed when a container is started. Arguments are appended to this command. So to add your connection string, you could do the following

api:
  build: .
  restart: always
  command: "host=db user=user dbname=dbname sslmode=disable password=123"
  ports:
    - 8080:8080
  links:
    - db

The last thing you should do is to have a static configuration for a docker image, like you show in your example. You basically set a static connection string, which deprives you of much of the flexibility using containers.

However, imho it is bad practice to use command line flags to configure containers, the first place. A much more flexible way is to use environment variables. For example, in kubernetes you would use a config map to set up environment variables which in turn configure a pod. However, environment variables can be used with docker-compose or docker swarm, too.

Hence, I would change the docker-compose.yml to something like this:

version: '3'
services:
  db:
    volumes:
      - ./db/pgdata:/pgdata
    image: postgres
    ports:
      - "5432"
    restart: always
    environment:
      - POSTGRES_USER=user
      - POSTGRES_DB=dbname
      - POSTGRES_PASSWORD=123
      - PGDATA=/pgdata

  # adminer added for convenience
  adminer:
    image: adminer
    restart: always
    ports:
      - 8081:8080
    depends_on:
      - db

  api:
    build: .
    restart: always
    ports:
      - 8080:8080
    depends_on:
      - db
    environment:
      - POSTGRES_USER=user
      - POSTGRES_DB=dbname
      - POSTGRES_PASSWORD=123
      - POSTGRES_HOST=db
      - POSTGRES_PORT=5432    

and use the environment variables to configure the binary.

I added a simple example how to use environment variables in Go to the repo. Note that projects like https://github.com/spf13/cobra/cobra or https://gopkg.in/alecthomas/kingpin.v2 make it much easier to work with environment variables, as they provide automatic type conversion, the ability to easily set default values and much more.

As for a more in depth reasoning why to use environment variables, you might want to read part 3 of the 12 factor app methodology.

hth

Answer 2

I made a tiny docker wrapper for the pressly/goose for my own purposes, maybe it will be useful for someone: github.com/kukymbr/goose-docker.

For example, pure docker call:

docker run --rm -v ./migrations:/migrations --network host \
-e GOOSE_DRIVER="postgres" \
-e GOOSE_DBSTRING="host=localhost port=5432 user=postgres password=postgres dbname=postgres" \
ghcr.io/kukymbr/goose-docker:3.22.1

Docker compose example:

services:
# ... Add your DB service

migrations:
  image: ghcr.io/kukymbr/goose-docker:3.22.1
  environment:
    - GOOSE_DRIVER=postgres
    - GOOSE_DBSTRING=host=postgres port=5432 user=postgres password=postgres dbname=postgres
  volumes:
    - ./migrations:/migrations

UPD: also added the goose-docker-cmd image as a goose wrapper without any sugar, just an entrypoint:

docker run --rm -v ./migrations:/migrations \
     -e GOOSE_MIGRATION_DIR="/migrations" \
     -e GOOSE_DRIVER="postgres" \
     ghcr.io/kukymbr/goose-docker-cmd:latest \
     create my_new_feature sql