Chapter 5 · Ship It
Docker Compose: API, PostgreSQL and Redis Together
- Page 18 of 22
- 26 min read
A real AI backend is rarely one process. The review analyser from the last page needs a database to remember what it analysed and a cache for counters and repeated work. Started by hand, that is three docker run commands with a network, a volume, a password and a start order to get right, every time, on every laptop. Docker Compose turns all of it into one file, compose.yaml, and one command: docker compose up.
This page builds a three-service stack: the FastAPI API, PostgreSQL and Redis, plus a one-shot migration job. You will write every file and check the parts that do not need Docker by running them: the settings code, the readiness endpoint, and a script that reads compose.yaml and lists what it really says. The docker compose commands themselves are shown without output here; the text describes exactly what you will see when you run them on your own machine.
What you will learn
- How a
compose.yamldescribes services, networks, volumes and secrets, and how services find each other by name. - Health checks and
depends_onwithcondition: service_healthy, and the difference between liveness and readiness. - Configuration:
.envversusenv_file, and Compose secrets mounted at/run/secretsand read from Python. - Running migrations as a one-shot service, and dev versus production overrides.
- The daily commands:
up,down,ps,logs,exec.
Why Compose
By hand, the database alone is already a long command, and you must also remember to create a network so the API can reach it, and a volume so the data survives:
docker network create review-net
docker volume create review-pgdata
docker run -d --name review-db --network review-net -v review-pgdata:/var/lib/postgresql \
-e POSTGRES_PASSWORD=change-me postgres:18-alpineNow multiply by Redis, the API, its ports and variables, and a start order. Compose keeps all of that in a file you can read, review in a pull request and commit to Git. Use docker compose with a space: it is part of Docker itself (Compose v5 in 2026; the old Python docker-compose command is dead). The file is named compose.yaml, and the old top-level version: key is ignored, so leave it out.
| Compose concept | What it is |
|---|---|
| Project | One running copy of the stack. Named after the folder or the top-level name:; it prefixes container, network and volume names. |
| Service | One kind of container in the stack (api, db, cache). It can run as several containers. |
| Network | A private virtual network. Services on it reach each other by service name. |
| Volume | Storage managed by Docker that outlives containers. |
| Secret | A file mounted read-only at /run/secrets/<name> inside the services that ask for it. |
The project
Create a folder and a virtual environment. The environment is only for the checks you run on your own machine; the containers get their own Python:
mkdir -p ~/projects/review-stack
cd ~/projects/review-stack
python3 -m venv .venv
source .venv/bin/activate
pip install -q "fastapi[standard]" "psycopg[binary]" redis pydantic-settings pyyaml httpx2Settings and secrets in Python
The API gets its configuration from the environment: where the database is, where Redis is, which version it is. The one thing that must not travel in a plain environment variable is the database password, so the code reads it from a secret file when one exists. Compose mounts each secret as a file named after the secret, in /run/secrets. Any platform that gives you a whole DATABASE_URL works too, and wins when set:
# file: settings.py
"""Settings: plain environment variables plus Docker/Compose secret files."""
import os
from pathlib import Path
from urllib.parse import quote
from pydantic_settings import BaseSettings
def read_secret(name: str) -> str | None:
"""Return /run/secrets/<name> (or $SECRETS_DIR/<name>), or None if there is no such file."""
path = Path(os.environ.get("SECRETS_DIR", "/run/secrets")) / name
return path.read_text().strip() if path.is_file() else None
class Settings(BaseSettings):
app_version: str = "dev"
database_url: str | None = None # a platform may hand you the whole URL
db_host: str = "localhost"
db_port: int = 5432
db_name: str = "reviews"
db_user: str = "app"
redis_url: str = "redis://localhost:6379/0"
@property
def postgres_dsn(self) -> str:
if self.database_url:
return self.database_url
password = read_secret("db_password") or os.environ.get("DB_PASSWORD", "")
return f"postgresql://{self.db_user}:{quote(password, safe='')}@{self.db_host}:{self.db_port}/{self.db_name}"Try it with a secret file on your own disk. A folder named secrets next to the project will be exactly what Compose mounts later, and SECRETS_DIR points the code at it for now. The password contains a character that is special in URLs, so you can see quote at work:
mkdir -p secrets
printf 'dev-only-pw!' > secrets/db_password
chmod 644 secrets/db_password
ls secretsdb_passwordimport os
os.environ.update(SECRETS_DIR="secrets", DB_HOST="db", APP_VERSION="1.0.0")
from settings import Settings
s = Settings()
print(s.app_version, s.db_host, s.db_port)
print(s.postgres_dsn)1.0.0 db 5432
postgresql://app:dev-only-pw%21@db:5432/reviewsThe host is db, not localhost; the next section explains why. (That is a throwaway development password. Never print or log a real connection string.) Notice that chmod 644 makes the file readable by other users. A container's user is not you: the API runs as UID 10001, so a file only you can read may be unreadable inside the container. This is fine for a development password; production secrets come from your platform's secret store, not from a file in the project.
The API
The API has two health endpoints with different jobs. /health is liveness: "the process is running", no dependencies checked. /ready is readiness: "I can really serve requests", so it checks PostgreSQL and Redis and answers 503 if either is down. A load balancer should stop sending traffic to an instance that is alive but not ready, instead of restarting it. The analysis endpoint stores each review in PostgreSQL and counts it in Redis. (For brevity it opens one connection per request; Databases with SQLAlchemy and Alembic showed the engine and session-per-request pattern for real services.)
# file: main.py
"""Review API for Compose: FastAPI + PostgreSQL + Redis."""
import logging
from typing import Annotated
import psycopg
import redis
from fastapi import FastAPI, Response
from pydantic import BaseModel, Field
from settings import Settings
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("review-api")
settings = Settings()
cache = redis.Redis.from_url(settings.redis_url, socket_connect_timeout=2, socket_timeout=2)
app = FastAPI(title="Review Analyser", version=settings.app_version)
POSITIVE = {"good", "great", "excellent", "clear", "helpful", "love"}
NEGATIVE = {"bad", "boring", "confusing", "slow", "hate", "broken"}
class Review(BaseModel):
text: Annotated[str, Field(min_length=3, max_length=2000)]
@app.get("/health")
def health() -> dict[str, str]:
"""Liveness: the process is up. Checks nothing else."""
return {"status": "ok", "version": settings.app_version}
@app.get("/ready")
def ready(response: Response) -> dict[str, str]:
"""Readiness: can this instance serve real requests right now?"""
checks = {}
try:
with psycopg.connect(settings.postgres_dsn, connect_timeout=2) as conn:
conn.execute("SELECT 1")
checks["postgres"] = "up"
except psycopg.Error:
checks["postgres"] = "down"
try:
cache.ping()
checks["redis"] = "up"
except redis.RedisError:
checks["redis"] = "down"
ok = all(state == "up" for state in checks.values())
if not ok:
response.status_code = 503
return {"status": "ready" if ok else "unavailable", **checks}
@app.post("/v1/analyse")
def analyse(review: Review) -> dict[str, str | float]:
words = [w.strip(".,!?").lower() for w in review.text.split()]
score = round((sum(w in POSITIVE for w in words) - sum(w in NEGATIVE for w in words)) / max(len(words), 1), 2)
sentiment = "positive" if score > 0 else "negative" if score < 0 else "neutral"
with psycopg.connect(settings.postgres_dsn) as conn:
conn.execute("INSERT INTO reviews (text, sentiment) VALUES (%s, %s)", (review.text, sentiment))
cache.incr("analysed_total")
log.info("stored a %s review", sentiment)
return {"sentiment": sentiment, "score": score}
@app.get("/v1/stats")
def stats() -> dict[str, int]:
with psycopg.connect(settings.postgres_dsn) as conn:
stored = conn.execute("SELECT count(*) FROM reviews").fetchone()[0]
return {"stored": stored, "analysed_total": int(cache.get("analysed_total") or 0)}Both databases are down on your laptop right now, which makes a good test of readiness. Point the settings at a closed port (nothing listens on port 1), and ask the app through FastAPI's TestClient:
import os
os.environ.update(DB_HOST="127.0.0.1", DB_PORT="1", REDIS_URL="redis://127.0.0.1:1/0")
from fastapi.testclient import TestClient
from main import app
client = TestClient(app)
print(client.get("/health").json())
r = client.get("/ready")
print(r.status_code, r.json()){'status': 'ok', 'version': 'dev'}
503 {'status': 'unavailable', 'postgres': 'down', 'redis': 'down'}Liveness stays green while readiness reports exactly which dependency is missing. Inside the Compose stack, the same endpoint will report "postgres": "up" and "redis": "up".
The migration script
Tables must exist before the API stores anything. A small idempotent script creates them (IF NOT EXISTS, so it is safe to run on every start). For a real project this slot is where alembic upgrade head goes, exactly as in Databases with SQLAlchemy and Alembic:
# file: migrate.py
"""Create the schema. Safe to run on every deploy."""
import psycopg
from settings import Settings
SCHEMA = """
CREATE TABLE IF NOT EXISTS reviews (
id SERIAL PRIMARY KEY,
text TEXT NOT NULL,
sentiment TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
"""
if __name__ == "__main__":
with psycopg.connect(Settings().postgres_dsn) as conn:
conn.execute(SCHEMA)
print("schema ready")The image
This is the single-stage pip Dockerfile from Docker: Packaging Your API with three code files instead of one. (The multi-stage uv Dockerfile from that page works unchanged apart from its COPY lines; use whichever your team prefers.) The dependencies live in a requirements.txt; in a real project pin it with uv export, as the last page showed:
cat > requirements.txt <<'EOF'
fastapi[standard]>=0.143.0
psycopg[binary]>=3.3
redis>=8.1
pydantic-settings>=2.15
EOF# file: Dockerfile
FROM python:3.14-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PORT=8000
RUN useradd --create-home --uid 10001 app
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY main.py settings.py migrate.py ./
USER app
EXPOSE 8000
CMD ["fastapi", "run", "main.py"]# file: .dockerignore
.venv/
__pycache__/
.git/
.env
secrets/
*.logWriting compose.yaml
First the two small env files. app.env holds non-secret settings for the containers; it is safe to commit. .env is something else (see below) and stays out of Git:
# file: app.env
DB_HOST=db
DB_NAME=reviews
DB_USER=app
REDIS_URL=redis://cache:6379/0# file: .env
APP_VERSION=1.0.0# file: .gitignore
.venv/
__pycache__/
.env
secrets/And the stack itself. Read it top to bottom; every line is explained after it:
# file: compose.yaml
name: review-stack
# Settings shared by the API and the migration job (a YAML anchor).
x-app: &app
build: .
image: bitbyte-p18-api:${APP_VERSION:-dev}
env_file: [app.env]
environment:
APP_VERSION: ${APP_VERSION:-dev}
secrets: [db_password]
networks: [backend]
services:
api:
<<: *app
ports:
- "8000:8000"
networks: [public, backend]
depends_on:
db: {condition: service_healthy}
cache: {condition: service_healthy}
migrate: {condition: service_completed_successfully}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=3)"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
restart: unless-stopped
migrate:
<<: *app
command: ["python", "migrate.py"]
depends_on:
db: {condition: service_healthy}
restart: "no"
db:
image: postgres:18-alpine
environment:
POSTGRES_USER: app
POSTGRES_DB: reviews
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets: [db_password]
volumes:
- pgdata:/var/lib/postgresql
networks: [backend]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app -d reviews"]
interval: 5s
timeout: 3s
retries: 10
start_period: 10s
restart: unless-stopped
cache:
image: redis:8-alpine
command: ["redis-server", "--appendonly", "yes"]
volumes:
- redisdata:/data
networks: [backend]
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
public: {}
backend:
internal: true
volumes:
pgdata:
redisdata:
secrets:
db_password:
file: ./secrets/db_passwordServices, images and build
Each key under services: is a service name. A service has either an image to pull (postgres:18-alpine, redis:8-alpine) or a build context with a Dockerfile (our API). When a service has both, as api does, Compose builds the image and tags it with that name. ${APP_VERSION:-dev} is interpolation: use APP_VERSION from the environment or from the .env file, and dev if it is unset. The x-app block and the &app / <<: *app pair are plain YAML anchors: keys starting with x- are ignored by Compose, so they are a free place to share settings between api and migrate. Keys written next to <<: *app, such as the API's own networks, win over the anchor's.
Networks: how services find each other
Services on the same network reach each other by service name. Compose runs a tiny DNS server, so the API connects to db:5432 and cache:6379, never to an IP address (those change on every restart). This is why app.env says DB_HOST=db. And localhost inside a container means that container, not your laptop and not another service: pointing the API at localhost:5432 is the classic first Compose error.
| Where the code runs | Reach PostgreSQL at |
|---|---|
Inside the api container | db:5432 (service name and the container's own port) |
On your laptop, e.g. psql or a test run | localhost:5432, only if the db service publishes ports |
The file defines two networks on purpose. public carries the published port. backend is marked internal: true: its members can talk to each other but have no route to the outside world. The database and cache sit only on backend, so even a compromised container there cannot call out. The API is on both, because it must also reach the internet (an LLM provider, say). Notice what is not there: no ports for db and cache. Only the API is published; the datastores are reachable from other services only. Publishing a database port to the whole internet is a classic breach.
Volumes
A container's own filesystem is thrown away when the container is removed. A named volume (pgdata, redisdata) is stored by Docker outside any container, so docker compose down followed by up keeps your reviews. The mount point matters: PostgreSQL 18 images keep data under /var/lib/postgresql (older images used /var/lib/postgresql/data), so that is where the volume goes; Redis writes to /data, and --appendonly yes makes it persist writes. A bind mount (./folder:/path) maps a folder from your machine instead; you will use one for development below.
Health checks and start order
A container is "started" the moment its process begins; PostgreSQL then needs several more seconds to initialise and accept connections. A healthcheck is a command Docker runs inside the container every interval; exit code 0 means healthy. start_period gives slow starters a grace period in which failures do not count; after retries consecutive failures the container is marked unhealthy. The checks here use tools already inside each image: pg_isready for PostgreSQL, redis-cli ping for Redis, and the Python standard library for the API (the slim image has no curl). Docker's check uses /health, the liveness endpoint, because a restart cannot fix a missing database; /ready is for load balancers and your own monitoring.
depends_on with a condition turns that into start order:
| Condition | Waits until the dependency… |
|---|---|
service_started (the default) | has started. Says nothing about being ready, so the API would often crash on its first query. |
service_healthy | passes its health check. Needs a healthcheck on that service. |
service_completed_successfully | has exited with code 0. For one-shot jobs like migrate. |
So the order is: db and cache become healthy, migrate creates the table and exits, and only then does api start. restart: "no" on the job (quotes required, or YAML reads it as false) keeps it from re-running forever; unless-stopped on the long-running services brings them back after a crash or a reboot. Remember that start order is not a guarantee for later: the database can restart while the API is running. Real code should also retry, and report /ready as down meanwhile.
Reading the file you wrote
Because the compose.yaml is only YAML, you can inspect it with a few lines of Python. This script is a handy review tool, and it confirms the file says what you meant (anchors included):
import yaml
cfg = yaml.safe_load(open("compose.yaml"))
for name, svc in cfg["services"].items():
deps = {d: c["condition"] for d, c in svc.get("depends_on", {}).items()}
print(f"{name:8} image={svc['image']}")
print(f" ports={svc.get('ports', [])} healthcheck={'healthcheck' in svc} networks={svc['networks']}")
print(f" depends_on={deps}")
print("volumes:", list(cfg["volumes"]), "| networks:", list(cfg["networks"]), "| secrets:", list(cfg["secrets"]))api image=bitbyte-p18-api:${APP_VERSION:-dev}
ports=['8000:8000'] healthcheck=True networks=['public', 'backend']
depends_on={'db': 'service_healthy', 'cache': 'service_healthy', 'migrate': 'service_completed_successfully'}
migrate image=bitbyte-p18-api:${APP_VERSION:-dev}
ports=[] healthcheck=False networks=['backend']
depends_on={'db': 'service_healthy'}
db image=postgres:18-alpine
ports=[] healthcheck=True networks=['backend']
depends_on={}
cache image=redis:8-alpine
ports=[] healthcheck=True networks=['backend']
depends_on={}
volumes: ['pgdata', 'redisdata'] | networks: ['public', 'backend'] | secrets: ['db_password']Only api has ports; db and cache have none. migrate has no health check because it is a job, not a server.
Environment files and secrets
Compose has two different env-file features, and mixing them up causes many bugs:
.env file | env_file: key | |
|---|---|---|
| Read by | Compose itself, automatically, from the project folder | Compose, for the services that list it |
| Used for | Filling ${VARIABLES} inside compose.yaml | Setting environment variables inside the container |
| In this project | APP_VERSION (the image tag and the version) | app.env (DB_HOST, REDIS_URL…) |
A variable in .env does not appear in the container unless compose.yaml passes it on, as environment: APP_VERSION: ${APP_VERSION:-dev} does. Precedence for the same variable, highest first: your shell's environment, then .env, then the default after :-. You can read an env file the same way Compose does with python-dotenv, which Environment Variables, .env Files and Virtual Environments in the Beginner tutorial introduced; here the standard library is enough:
def read_env_file(path):
pairs = (line.split("=", 1) for line in open(path).read().splitlines() if "=" in line and not line.startswith("#"))
return {key: value for key, value in pairs}
print(read_env_file(".env"))
print(read_env_file("app.env")){'APP_VERSION': '1.0.0'}
{'DB_HOST': 'db', 'DB_NAME': 'reviews', 'DB_USER': 'app', 'REDIS_URL': 'redis://cache:6379/0'}Secrets
Passwords in environment: are easy to leak: they show up in docker inspect, in crash reports and in child processes. A Compose secret is different. The top-level secrets: section says where the value lives on the host (file: ./secrets/db_password); each service lists the secrets it may see; and inside that container the value appears as a file, /run/secrets/db_password. Two consumers in our stack use it two ways:
- PostgreSQL supports the
_FILEconvention:POSTGRES_PASSWORD_FILE=/run/secrets/db_passwordmakes the image read the password from the file instead of fromPOSTGRES_PASSWORD. Many official images do the same for their variables. - Our API calls
read_secret("db_password"), which you tested above withSECRETS_DIR. In the container no variable is needed, because/run/secretsis the default.
Be honest about what this buys you. A Compose file secret is not encrypted: it is a read-only file mount of a file on your host, and it keeps the value out of environment listings, image layers and compose.yaml itself. Keep secrets/ in .gitignore and .dockerignore (both done above). In production, feed the same file mechanism from your platform's secret manager. Also note that PostgreSQL applies POSTGRES_PASSWORD_FILE only when it first initialises an empty volume; changing the file later does not change the database's password, so either change it with SQL or delete the volume (which deletes the data).
Running the stack
With the files in place, build and start everything in the background (-d). --build rebuilds the API image first:
docker compose up -d --build
docker compose psThe first run pulls postgres and redis and builds the API, so it takes a minute or two. Compose creates the networks and volumes, starts db and cache, waits for them, runs migrate, then starts api. docker compose ps prints a table with the columns NAME, IMAGE, COMMAND, SERVICE, CREATED, STATUS and PORTS. You should see three running containers named like review-stack-api-1, review-stack-db-1 and review-stack-cache-1, each with (healthy) in its STATUS, and 0.0.0.0:8000->8000/tcp in the PORTS of the API only. The finished migrate container is not listed, because ps shows running containers; docker compose ps -a shows it as Exited (0). The API process needs a second or two to start listening, even after up has returned, which is why the first curl retries. Now use the API from your host:
curl -s --retry 30 --retry-all-errors --retry-delay 1 http://localhost:8000/ready
echo
curl -s -X POST http://localhost:8000/v1/analyse -H "Content-Type: application/json" -d '{"text": "Great course, very clear and helpful!"}'
echo
curl -s http://localhost:8000/v1/statsThe first call returns {"status":"ready","postgres":"up","redis":"up"}: the readiness check from earlier, now with real services behind it. The second returns {"sentiment":"positive","score":0.5}, and the third {"stored":1,"analysed_total":1}, one row in PostgreSQL and one count in Redis. Open http://localhost:8000/docs in a browser for the interactive docs.
| Command | What it does |
|---|---|
docker compose up -d | Create and start everything in the background. Add --build to rebuild images. |
docker compose ps / ps -a | Show the services and their state and health (-a includes stopped ones). |
docker compose logs -f api | Follow one service's logs (omit the name for all). --tail 50 limits it. |
docker compose exec db psql -U app reviews | Run a command inside a running container, here an interactive SQL shell. |
docker compose run --rm migrate | Start a new one-off container of a service, then remove it. |
docker compose restart api | Restart one service. |
docker compose stop / start | Stop or start containers without deleting them. |
docker compose down | Stop and remove containers and networks. Volumes (your data) are kept. |
docker compose down -v | Also remove the volumes: this deletes the database. |
docker compose config | Print the final file after interpolation and merging: the first thing to run when something looks wrong. |
To prove the volume works, run this and then call /v1/stats again. It still reports your stored reviews, because containers were replaced but pgdata was not:
docker compose down
docker compose up -d
curl -s --retry 30 --retry-all-errors --retry-delay 1 http://localhost:8000/ready > /dev/null
curl -s http://localhost:8000/v1/statsMigrations run on every up here because the job is idempotent. If you change the schema, run the job again explicitly (docker compose run --rm migrate) before the new API version serves traffic; on a real platform the same job runs as a release or pre-deploy step. The rule that keeps deploys safe: make each migration compatible with the previous API version, so old and new containers can run side by side for a moment.
Development and production overrides
The same stack should behave differently on your laptop and on a server. Instead of two copies of the file, keep one base compose.yaml and put the differences in small override files. When you run plain docker compose up, Compose automatically merges compose.override.yaml on top of compose.yaml. That is the place for development conveniences: auto-reload with your source code mounted into the container, and a database you can reach from your laptop's tools. Because backend is an internal network with no route to the outside, a port published from a service that sits only there would not be reachable from your laptop, so the override also puts db on public:
# file: compose.override.yaml
services:
api:
# fastapi dev reloads on every save; --host 0.0.0.0 is needed inside a container
command: ["fastapi", "dev", "main.py", "--host", "0.0.0.0"]
volumes:
- .:/app
db:
networks: [public, backend]
ports:
- "127.0.0.1:5432:5432"Production is selected explicitly with -f. Passing -f turns the automatic override off, so a production command never accidentally includes development settings:
# file: compose.prod.yaml
services:
api:
build: !reset null
image: ghcr.io/nadia-rahman/review-api:${APP_VERSION:?set APP_VERSION}
pull_policy: always
ports: !override
- "127.0.0.1:8000:8000"
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
logging:
driver: json-file
options: {max-size: "10m", max-file: "3"}
migrate:
build: !reset null
image: ghcr.io/nadia-rahman/review-api:${APP_VERSION:?set APP_VERSION}How merging works, in short: mappings are merged key by key and the later file wins for a single value (command, image); for lists, such as ports and volumes, entries are added rather than replaced. That is why the production file uses the !override tag, to replace the list outright, and !reset null to remove the build section, since production runs an image that CI already built and pushed (the next page does that). ${APP_VERSION:?message} stops with that message if the variable is unset, which is far better than silently deploying :dev. The loopback address 127.0.0.1 in ports means "only this machine": a reverse proxy or the platform's router sits in front, and the API is never exposed directly. The deploy.resources.limits keys cap the container's CPU and memory, so one runaway model call cannot starve the database. The two commands, and the one that shows the merged result:
docker compose up -d --build
APP_VERSION=1.0.0 docker compose -f compose.yaml -f compose.prod.yaml configThe first uses the development override automatically. The second only prints: docker compose config shows the final YAML after interpolation and merging, with the production image name and a single loopback port mapping; check it before every production deploy. Both files are still plain YAML, so you can at least confirm that they parse and see which keys each one touches. The ! tags are Compose-specific, so this script tells PyYAML to read them as ordinary values:
import yaml
def any_tag(loader, suffix, node):
if isinstance(node, yaml.SequenceNode):
return loader.construct_sequence(node)
return loader.construct_scalar(node)
yaml.add_multi_constructor("!", any_tag, Loader=yaml.SafeLoader)
for filename in ("compose.override.yaml", "compose.prod.yaml"):
for service, body in yaml.safe_load(open(filename))["services"].items():
print(f"{filename:22} {service:8} changes {sorted(body)}")compose.override.yaml api changes ['command', 'volumes']
compose.override.yaml db changes ['networks', 'ports']
compose.prod.yaml api changes ['build', 'deploy', 'image', 'logging', 'ports', 'pull_policy']
compose.prod.yaml migrate changes ['build', 'image']When something goes wrong
Start with docker compose ps -a: it shows which container exited, restarts or is unhealthy. Then read that service's logs with docker compose logs --tail 50 <service>. Typical findings: migrate exited with 1 (it could not connect or the SQL failed; its log says which); the API restarts in a loop (a missing setting raised at import time, visible in its logs); db is unhealthy (wrong user or database name in the check, or a corrupted volume after a major version upgrade). To start again from nothing in development, run docker compose down -v, which removes containers, networks and volumes, then up again.
Where this shows up
- Local development of AI services. One command gives every teammate the same PostgreSQL, Redis (or Valkey) and API, with seeded data and no installs on their laptops. Vector databases (pgvector inside PostgreSQL, Qdrant, Chroma) are just another service in the same file.
- CI. The pipeline in the next page can start the same databases for integration tests, so a test failure on CI can be reproduced on your machine.
- Small production deployments. A single VPS running
docker composebehind a reverse proxy is a legitimate, cheap production setup for small products. When you need several machines, automatic failover or rolling deploys, Compose's job passes to a platform (Kubernetes, Cloud Run, ECS), and the images you built stay the same.
Common mistakes
- Using
localhostto reach another service. Inside a container it points at the container itself, so the connection is refused. Use the service name (db,cache) and the container's own port. - Plain
depends_onwithout a condition. It waits only for the process to start, not to be ready, so the API races the database and sometimes crashes. Add ahealthcheckandcondition: service_healthy. - Running
docker compose down -vout of habit. The-vdeletes your named volumes, which means your database. Plaindownkeeps the data; save-vfor a deliberate reset. - Passwords in
environment:or in a committed.env. They leak throughdocker inspect, logs and Git history. Use secrets, keep.envandsecrets/in.gitignore, and ship an example file with fake values instead. - Expecting
.envto reach the container, or a changed password to reach an existing database..envonly fills${…}in the YAML, and the database image reads its password variable only when it first initialises an empty volume. - Publishing database ports to the world.
"5432:5432"listens on every network interface. Do not publish datastores in production; in development bind them to127.0.0.1.
Try it yourself
- Easy: with Python and PyYAML, print the names of the services in
compose.yamlthat have nohealthcheck. - Medium: Redis-compatible Valkey (the BSD-licensed Linux Foundation fork) works with the same Python client. Write a script that makes a copy of
compose.yamlcalledcompose.valkey.yamlthat uses thevalkey/valkey:9-alpineimage withvalkey-serverandvalkey-cli, then print the cache service's image and health check from the copy. - Hard: write
lint_compose.py, a functionproblems(cfg)that reports (a) a service that waits for a dependency withservice_healthywhen that dependency has no health check, and (b) any service other thanapithat publishes ports. Show that it finds nothing in your file, and that it catches both problems in a broken copy.
Answers
1. Read the file and filter on the key:
import yaml
services = yaml.safe_load(open("compose.yaml"))["services"]
print([name for name, svc in services.items() if "healthcheck" not in svc])['migrate']2. Text replacement keeps the rest of the file untouched; the volume name stays, and REDIS_URL stays too, because Valkey speaks the Redis protocol. Then parse the copy to check it:
import yaml
text = open("compose.yaml").read()
text = text.replace("redis:8-alpine", "valkey/valkey:9-alpine")
text = text.replace("redis-server", "valkey-server").replace("redis-cli", "valkey-cli")
open("compose.valkey.yaml", "w").write(text)
cache = yaml.safe_load(text)["services"]["cache"]
print(cache["image"], cache["healthcheck"]["test"])valkey/valkey:9-alpine ['CMD', 'valkey-cli', 'ping']3. The linter is a plain function over the parsed dictionary, so it can run in CI as a pre-merge check (and be tested like any code):
# file: lint_compose.py
import copy
import yaml
def problems(cfg: dict) -> list[str]:
services = cfg["services"]
found = []
for name, svc in services.items():
for dep, rule in svc.get("depends_on", {}).items():
if rule["condition"] == "service_healthy" and "healthcheck" not in services[dep]:
found.append(f"{name} waits for {dep} to be healthy, but {dep} has no healthcheck")
if name != "api" and svc.get("ports"):
found.append(f"{name} publishes ports")
return found
if __name__ == "__main__":
cfg = yaml.safe_load(open("compose.yaml"))
print(problems(cfg))
broken = copy.deepcopy(cfg)
del broken["services"]["db"]["healthcheck"]
broken["services"]["cache"]["ports"] = ["6379:6379"]
for line in problems(broken):
print(line)python lint_compose.py[]
api waits for db to be healthy, but db has no healthcheck
migrate waits for db to be healthy, but db has no healthcheck
cache publishes ports
Finally, clean up the stack when you are done experimenting. Containers and networks go; add -v only if you also want to delete the data:
docker compose downSummary
- One
compose.yamldescribes services, networks, volumes and secrets;docker compose up -dbuilds and starts the whole stack,downremoves it (anddown -valso deletes the data). - Services reach each other by service name on a shared network (
db:5432,cache:6379);localhostmeans the container itself. Publish only the API, and keep datastores on an internal network. - Give every dependency a
healthcheckand usedepends_onwithservice_healthy; run migrations as a one-shot service withservice_completed_successfully. Keep liveness (/health) and readiness (/ready) separate. .envfills${…}in the YAML,env_filesets variables inside containers, and Compose secrets arrive as files at/run/secrets/<name>, read from Python or through_FILEvariables.- Keep one base file and add
compose.override.yamlfor development and-f compose.prod.yamlfor production; check the result withdocker compose config.
Next: CI/CD with GitHub Actions runs your tests on every push, starts a PostgreSQL service for them, and builds and publishes the image that the production file above pulls.