Testing Your API with pytest

You can check an API by hand: start the server, open /docs, click "Try it out", read the answer. That works for one endpoint on one afternoon. It does not work for twenty endpoints, a teammate's change on Friday, or a model upgrade next month. Tests are small programs that run your code and check the answers for you, in a second, every time.

On this page you build a small text tools API (counting tokens and cutting text into chunks, which is what RAG pipelines do before an LLM sees a document) and test it with pytest and FastAPI's TestClient. You will test success, validation errors (422), error codes and edge cases, learn to read a failing test, and organise tests in a tests/ folder.

What you will learn

• Why automated tests beat manual checking, and what a test really is.
• pytest basics: assert, test discovery, and the flags -q, -v and -k.
• How to test an API without starting a server, using TestClient (and why you install httpx2 next to pytest).
• How to test success, 422 validation errors, 404 errors and edge cases.
• How to read a failing test and a Python traceback, and how to use breakpoint().
• How to organise tests: the tests/ folder and a first fixture in conftest.py.

Set up the project

A fresh uv project with the app code in an app package, as on Pydantic Models, Validation and Error Responses. Besides FastAPI you need two test packages: pytest (the test runner) and httpx2 (needed by TestClient, explained below). They are only for development, so uv keeps them in a separate group with --dev.

mkdir -p ~/projects
cd ~/projects
uv init -q --no-package bitbyte-text-tools
cd bitbyte-text-tools
uv add -q "fastapi[standard]"
uv add -q --dev pytest httpx2
rm main.py
mkdir app tests
touch app/__init__.py
ls -A

.git
.gitignore
.python-version
.venv
README.md
app
pyproject.toml
tests
uv.lock

With pip instead: create and activate a virtual environment, then pip install "fastapi[standard]" pytest httpx2. Everywhere below, uv run pytest becomes just pytest once the environment is active.

Why httpx2?

TestClient sends requests to your app inside the same Python process, using an HTTP client library. It used to need httpx. Starlette 1.7 (the library under FastAPI) now warns when TestClient runs on httpx and prefers httpx2, the maintained fork of httpx published by the Pydantic team (you would import it as httpx2, but you never have to: TestClient finds it itself). So the rule for new projects is simple: pip install pytest httpx2. If an older tutorial tells you to install httpx, it still mostly works but may show a deprecation warning. For calling other APIs in your own code this course keeps using requests; the advanced tutorial compares the HTTP clients.

The code to test

Two files. The first holds plain functions with no web code at all: they are the easiest things to test, and a good habit is to keep logic in functions like these and keep endpoints thin.

# file: app/chunking.py
"""Split text into pieces, the way RAG pipelines prepare documents."""


def estimate_tokens(text: str) -> int:
    """A rough rule of thumb: about 4 characters per token."""
    return (len(text) + 3) // 4


def chunk_text(text: str, size: int) -> list[str]:
    """Cut text into chunks of at most `size` characters."""
    if size < 1:
        raise ValueError("size must be at least 1")
    return [text[i:i + size] for i in range(0, len(text), size)]

The second is the API. It has a model for each request, limits on the text size, and a 404 for unknown model names, all things that can go wrong and therefore deserve a test:

# file: app/main.py
"""BitByte Text Tools: a tiny API for preparing text for LLMs."""
from typing import Annotated

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

from app.chunking import chunk_text, estimate_tokens

app = FastAPI(title="BitByte Text Tools")

CONTEXT_WINDOWS = {"bitbyte-mini": 8000, "bitbyte-pro": 32000}


class CountRequest(BaseModel):
    text: Annotated[str, Field(min_length=1, max_length=1000)]


class ChunkRequest(BaseModel):
    text: Annotated[str, Field(min_length=1, max_length=5000)]
    size: Annotated[int, Field(ge=10, le=500)] = 100


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/count")
def count(body: CountRequest):
    return {
        "characters": len(body.text),
        "words": len(body.text.split()),
        "estimated_tokens": estimate_tokens(body.text),
    }


@app.post("/chunk")
def chunk(body: ChunkRequest):
    chunks = chunk_text(body.text, body.size)
    return {"count": len(chunks), "chunks": chunks}


@app.get("/models/{name}")
def model_info(name: str):
    if name not in CONTEXT_WINDOWS:
        raise HTTPException(status_code=404, detail=f"Unknown model {name!r}")
    return {"name": name, "context_window": CONTEXT_WINDOWS[name]}

Why test, and why automate

Checking by hand has real value: you see the app the way a user does. But compare the two:

For AI products this matters more than usual. Models, prompts and data change often, and every change can break the plain code around them: the JSON shape, the limits, the error codes. Tests cover that deterministic part, so that when something goes wrong you know whether it is the model or your plumbing. (Testing the quality of model answers is a different job, called evaluation, covered later on the AI path.) Use both ways: hand-check while you explore, then write a test for everything you want to keep working.

pytest basics

A test is a function whose name starts with test_. Inside, you run some code and use a plain assert: if the condition is false, the test fails. First tests for the two functions, in tests/test_chunking.py:

# file: tests/test_chunking.py
import pytest

from app.chunking import chunk_text, estimate_tokens


def test_estimate_tokens():
    assert estimate_tokens("hello world") == 3


def test_chunk_text_splits_evenly():
    assert chunk_text("abcdef", 3) == ["abc", "def"]


def test_chunk_text_keeps_the_remainder():
    assert chunk_text("abcdefg", 3) == ["abc", "def", "g"]


def test_chunk_text_rejects_size_zero():
    with pytest.raises(ValueError):
        chunk_text("abc", 0)

Two details. pytest.raises(ValueError) passes only if the code inside raises that error, which is how you test "this must fail". And imports like from app.chunking import ... need pytest to know where your project root is. A little configuration in pyproject.toml does it (pythonpath); the addopts line adds options to every run, here hiding the version header and turning off the .pytest_cache folder to keep output tidy. Append both and run pytest:

cat >> pyproject.toml <<'EOF'

[tool.pytest.ini_options]
pythonpath = ["."]
addopts = "--no-header -p no:cacheprovider"
EOF
uv run pytest

======================================= test session starts ========================================
collected 4 items

tests/test_chunking.py ....                                                                  [100%]

======================================== 4 passed in 0.01s =========================================

That is a test session. pytest discovered the tests by itself: it looked for files named test_*.py (or *_test.py), then for functions named test_*. Nothing needs registering. Each dot is one passing test. The flags you will use every day:

uv run pytest -q
uv run pytest -v
uv run pytest -q -k remainder

....                                                                                         [100%]
4 passed in 0.01s
======================================= test session starts ========================================
collecting ... collected 4 items

tests/test_chunking.py::test_estimate_tokens PASSED                                          [ 25%]
tests/test_chunking.py::test_chunk_text_splits_evenly PASSED                                 [ 50%]
tests/test_chunking.py::test_chunk_text_keeps_the_remainder PASSED                           [ 75%]
tests/test_chunking.py::test_chunk_text_rejects_size_zero PASSED                             [100%]

======================================== 4 passed in 0.01s =========================================
.                                                                                            [100%]
1 passed, 3 deselected in 0.01s

Testing the API with TestClient

TestClient wraps your FastAPI app. You call client.get(...) or client.post(...) just like requests, and the request goes straight into the app, with no server, no port and no network. That is why the tests are fast and why you start nothing. A first test file, covering success:

# file: tests/test_api.py
from fastapi.testclient import TestClient

from app.main import app

client = TestClient(app)


def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    assert response.json() == {"status": "ok"}


def test_count_returns_characters_words_and_tokens():
    response = client.post("/count", json={"text": "Tokens are small pieces of text"})
    assert response.status_code == 200
    assert response.json() == {"characters": 31, "words": 6, "estimated_tokens": 8}


def test_known_model_has_a_context_window():
    response = client.get("/models/bitbyte-mini")
    assert response.status_code == 200
    assert response.json()["context_window"] == 8000

uv run pytest -q

.......                                                                                      [100%]
7 passed in 0.50s

A good API test has three steps: arrange (build the input), act (call the endpoint), assert (check the status code and the body). Check the status first; many bugs show up there. Compare the body as a whole when you can (== {...}), which also catches fields that should not be there.

Testing errors and edge cases

The happy path is the easy half. The tests that save you are the ones for bad input, because users, scripts and LLMs all send bad input. A second file for the error cases:

# file: tests/test_api_errors.py
import pytest
from fastapi.testclient import TestClient

from app.main import app

client = TestClient(app)


def test_unknown_model_is_404():
    response = client.get("/models/gpt-nothing")
    assert response.status_code == 404
    assert response.json() == {"detail": "Unknown model 'gpt-nothing'"}


def test_count_without_text_is_422():
    response = client.post("/count", json={})
    assert response.status_code == 422
    error = response.json()["detail"][0]
    assert error["type"] == "missing"
    assert error["loc"] == ["body", "text"]


def test_count_with_empty_text_is_422():
    response = client.post("/count", json={"text": ""})
    assert response.status_code == 422
    assert response.json()["detail"][0]["type"] == "string_too_short"


def test_count_accepts_exactly_the_maximum_length():
    assert client.post("/count", json={"text": "a" * 1000}).status_code == 200
    assert client.post("/count", json={"text": "a" * 1001}).status_code == 422


def test_chunk_uses_the_default_size():
    response = client.post("/chunk", json={"text": "x" * 250})
    assert response.status_code == 200
    assert response.json()["count"] == 3


@pytest.mark.parametrize("size", [0, 9, 501, -5])
def test_chunk_rejects_sizes_outside_the_range(size):
    response = client.post("/chunk", json={"text": "some text", "size": size})
    assert response.status_code == 422
    assert response.json()["detail"][0]["loc"] == ["body", "size"]


def test_chunk_rejects_a_text_that_is_not_a_string():
    response = client.post("/chunk", json={"text": 12345})
    assert response.status_code == 422

uv run pytest -q

.................                                                                            [100%]
17 passed in 0.54s

What these tests teach:

• 422 tests check the type and loc from the error body, not the long msg text, because those are the stable parts (see the last page).
• Boundaries are where bugs live. The test with 1000 and 1001 characters checks both sides of the limit: the largest value that must work and the smallest that must not. Do the same for every ge, le, min_length and max_length you write.
• @pytest.mark.parametrize runs one test function once for each value in the list, so four bad sizes need one function instead of four. The report counts them as four tests.
• A 404 test should check the body too, so that a wrong message cannot slip through.

Reading a failing test

A suite that always passes teaches you nothing; the moment of truth is a failure. Create a file with two broken tests on purpose. In the first, the expectation is wrong (the text has 31 characters, so the answer is 8 tokens, not 7). In the second, the code raises an error. The || true only stops the shell block from stopping on the failure; pytest still reports it:

# file: tests/test_demo.py
from app.chunking import chunk_text, estimate_tokens


def test_estimate_tokens_for_a_sentence():
    text = "Tokens are small pieces of text"
    assert estimate_tokens(text) == 7


def test_chunk_with_zero_size_is_not_handled():
    assert chunk_text("abc", 0) == []

uv run pytest -q -k sentence || true

F                                                                                            [100%]
============================================= FAILURES =============================================
_______________________________ test_estimate_tokens_for_a_sentence ________________________________

    def test_estimate_tokens_for_a_sentence():
        text = "Tokens are small pieces of text"
>       assert estimate_tokens(text) == 7
E       AssertionError: assert 8 == 7
E        +  where 8 = estimate_tokens('Tokens are small pieces of text')

tests/test_demo.py:7: AssertionError
===================================== short test summary info ======================================
FAILED tests/test_demo.py::test_estimate_tokens_for_a_sentence - AssertionError: assert 8 == 7
1 failed, 18 deselected in 0.58s

Read a failure from the bottom up:

1. The last lines (FAILED tests/test_demo.py::...) tell you which test failed and why, in one line.
2. The line marked > is the statement that failed, inside the test function.
3. The line marked E is the explanation. pytest rewrites your assert to show the actual values: assert 8 == 7 and, below it, that 8 came from estimate_tokens('Tokens are small pieces of text'). You rarely need a print.

Now the second test, where the code itself raises an error. Ask for the shorter traceback style with --tb=short:

uv run pytest -q -k not_handled --tb=short || true

F                                                                                            [100%]
============================================= FAILURES =============================================
_____________________________ test_chunk_with_zero_size_is_not_handled _____________________________
tests/test_demo.py:11: in test_chunk_with_zero_size_is_not_handled
    assert chunk_text("abc", 0) == []
           ^^^^^^^^^^^^^^^^^^^^
app/chunking.py:13: in chunk_text
    raise ValueError("size must be at least 1")
E   ValueError: size must be at least 1
===================================== short test summary info ======================================
FAILED tests/test_demo.py::test_chunk_with_zero_size_is_not_handled - ValueError: size must be at...
1 failed, 18 deselected in 0.74s

Here nothing compared wrong values; the function raised an exception. The block under the heading is a traceback, and the rule for tracebacks is always the same: read the last line first (the error type and message: ValueError: size must be at least 1), then go up the stack to the lines in your files (tests/test_demo.py called chunk_text in app/chunking.py). Frames from libraries are rarely the cause. The fix here is not to change the code, because rejecting size 0 is right; the test was wrong, and the correct version is the pytest.raises test you already have. Remove the demo file:

rm tests/test_demo.py
uv run pytest -q

.................                                                                            [100%]
17 passed in 0.56s

When a test fails, ask in this order: is the test wrong, is the code wrong, or is the requirement unclear? Never make a test pass just by editing it until it is green.

Debugging exceptions

A traceback is Python's own report for a crashed program, and you meet it outside pytest too. Here is a small script with a classic bug:

cat > average.py <<'EOF'
def average_words(texts):
    total = sum(len(t.split()) for t in texts)
    return total / len(texts)


print(average_words(["one two", "three four five"]))
print(average_words([]))
EOF
uv run python average.py || true

2.5
Traceback (most recent call last):
  File "/home/nadia/projects/bitbyte-text-tools/average.py", line 7, in <module>
    print(average_words([]))
          ^^^^^^^^^^^^^^^^^
  File "/home/nadia/projects/bitbyte-text-tools/average.py", line 3, in average_words
    return total / len(texts)
           ~~~~~~^~~~~~~~~~~~
ZeroDivisionError: division by zero

The last line names the problem (ZeroDivisionError), the line above it points at the exact code, and the first File line shows the call that started it all. An empty list is exactly the kind of edge case a test should cover. When the cause is still not obvious, stop the program inside the problem and look around. Python's built-in way is breakpoint(): put it in the code, run the program, and you get a (Pdb) prompt where you can print variables (p total), go to the next line (n), continue (c) or quit (q):

cat > stop_here.py <<'EOF'
texts = ["one two", "three four five"]
total = sum(len(t.split()) for t in texts)
breakpoint()
print(total / len(texts))
EOF
printf 'p total\np len(texts)\nc\n' | uv run python stop_here.py

> /home/nadia/projects/bitbyte-text-tools/stop_here.py(4)<module>()
-> print(total / len(texts))
(Pdb) 5
(Pdb) 2
(Pdb) 2.5

Here the commands were fed from printf so the page is repeatable; in your own terminal you type them at the prompt. With pytest, pytest --pdb opens the same prompt at the moment a test fails. Remember to delete every breakpoint() before you commit. Editors such as VS Code and PyCharm offer the same thing with a click in the margin.

Organising tests

As the suite grows, three habits keep it manageable:

• Keep tests in a tests/ folder that mirrors your code: tests/test_chunking.py for app/chunking.py, tests/test_api.py for app/main.py.
• One behaviour per test, with a name that reads like a sentence (test_unknown_model_is_404), so a failure tells you what broke before you open the file.
• Do not repeat set-up. This is what fixtures are for.

A fixture is a function marked @pytest.fixture that provides something tests need. A test asks for it by writing its name as a parameter, and pytest calls the function and passes in the result. Fixtures defined in a file named conftest.py are available to every test in the folder without any import. Both API test files above build the same client; move it there:

# file: tests/conftest.py
import pytest
from fastapi.testclient import TestClient

from app.main import app


@pytest.fixture
def client():
    """A fresh TestClient for each test."""
    return TestClient(app)

New test files no longer build their own client; they just ask for one:

# file: tests/test_models.py
def test_pro_model_has_a_bigger_context_window(client):
    small = client.get("/models/bitbyte-mini").json()["context_window"]
    large = client.get("/models/bitbyte-pro").json()["context_window"]
    assert large > small


def test_model_names_are_case_sensitive(client):
    assert client.get("/models/BitByte-Mini").status_code == 404

uv run pytest -q
uv run pytest -q tests/test_models.py
ls -A tests

...................                                                                          [100%]
19 passed in 0.07s
..                                                                                           [100%]
2 passed in 0.01s
conftest.py
test_api.py
test_api_errors.py
test_chunking.py
test_models.py

Fixtures can also create data, open a test database or set environment variables, and clean up afterwards; the advanced tutorial's testing page covers scopes, factories, overriding dependencies and mocking external APIs. The project layout is now:

bitbyte-text-tools/
├── app/
│   ├── __init__.py
│   ├── chunking.py       plain functions: easy to test
│   └── main.py           the API
├── tests/
│   ├── conftest.py       shared fixtures (client)
│   ├── test_api.py
│   ├── test_api_errors.py
│   ├── test_chunking.py
│   └── test_models.py
├── pyproject.toml        dependencies and the [tool.pytest.ini_options] settings
└── uv.lock

Where this shows up in real AI work

• Pull requests and CI: a team merges only when the tests pass; the advanced tutorial runs pytest in GitHub Actions on every push.
• Prediction endpoints: tests send known feature rows and check the response shape, ranges and error codes, with a tiny fake model so the test runs in milliseconds.
• LLM backends: tests replace the real provider with a fake one, so they cost nothing, never hit rate limits and always return the same answer. Only the plumbing is tested, which is exactly what you want in CI.
• Data pipelines: chunking, cleaning and parsing functions like chunk_text get dozens of edge-case tests (empty text, text shorter than a chunk, non-English text) because one silent bug there poisons every later step.

Common mistakes

1. Naming a file or function wrongly. check_api.py or def api_test() is never collected, so nothing runs (pytest prints "no tests ran" and exits with code 5, which CI treats as a failure, but a person skimming the output may miss it). Fix: test_*.py files, test_* functions, and look at the count in the summary line.
2. Only testing the happy path. The 200 is the case that already worked when you tried it by hand. Fix: for every endpoint, add at least one 404 or 422 test and the boundary values.
3. Asserting nothing useful. A test that only calls the endpoint, or checks status_code == 200 but never the body, can pass while the answer is wrong. Fix: assert the status and the content.
4. Tests that depend on each other or on the order. A test that needs another test to have created data first breaks as soon as you run it alone with -k. Fix: each test builds what it needs (a fixture helps).
5. Editing the test until it is green. That hides a real bug. Fix: first decide which of test, code or requirement is wrong.
6. ModuleNotFoundError: No module named 'app'. pytest cannot find your project root. Fix: pythonpath = ["."] in pyproject.toml, and run pytest from the project folder.

Try it yourself

1. Easy: Write tests/test_exercise_easy.py with a test for estimate_tokens("") (what should it return?) and one for a 4-character and a 5-character text. Run only these tests with -k.
2. Medium: Add tests (using the client fixture) that POST /chunk with {"text": "abcdefghijkl", "size": 10} answers 200 with the chunks ["abcdefghij", "kl"] and a count of 2, and a parametrized test that sends three invalid bodies to /count (a missing key, a number instead of text, and null) and checks that each answers 422.
3. Hard: The function below should return the last n words of a text, but it has a bug. Write tests that catch it, including the edge cases n=0 and n larger than the number of words, then fix the function.
   PythonCopy

   def last_words(text, n):
       return text.split()[-n:]

Answers

# file: tests/test_exercise_easy.py
from app.chunking import estimate_tokens


def test_empty_text_has_zero_tokens():
    assert estimate_tokens("") == 0


def test_four_and_five_characters():
    assert estimate_tokens("abcd") == 1
    assert estimate_tokens("abcde") == 2

uv run pytest -q -k "empty or characters"

....                                                                                         [100%]
4 passed, 17 deselected in 0.02s

# file: tests/test_exercise_medium.py
import pytest


def test_chunk_returns_chunks_and_count(client):
    response = client.post("/chunk", json={"text": "abcdefghijkl", "size": 10})
    assert response.status_code == 200
    assert response.json() == {"count": 2, "chunks": ["abcdefghij", "kl"]}


@pytest.mark.parametrize("body", [{}, {"text": 42}, {"text": None}])
def test_count_rejects_invalid_bodies(client, body):
    assert client.post("/count", json=body).status_code == 422

uv run pytest -q tests/test_exercise_medium.py

....                                                                                         [100%]
4 passed in 0.02s

For the hard exercise: text.split()[-0:] is the same as text.split()[0:], so n=0 returns all the words instead of none. (A larger n is fine: slicing never raises.) The tests below expose it; the fixed function handles the edge case. With the original function, test_last_words_zero_returns_nothing fails with assert ['one', 'two', 'three'] == []. Writing the edge-case test first, watching it fail, and then fixing the code is the habit behind "test-driven development".

# file: last_words.py
def last_words(text, n):
    if n <= 0:
        return []
    return text.split()[-n:]

# file: tests/test_exercise_hard.py
from last_words import last_words


def test_last_words_normal():
    assert last_words("one two three four", 2) == ["three", "four"]


def test_last_words_zero_returns_nothing():
    assert last_words("one two three", 0) == []


def test_last_words_more_than_available_returns_all():
    assert last_words("one two", 5) == ["one", "two"]

uv run pytest -q tests/test_exercise_hard.py

...                                                                                          [100%]
3 passed in 0.01s

The import from last_words import last_words works because pythonpath = ["."] puts the project folder on the path.

Summary

• Automated tests check your code in seconds, every time; write one for each behaviour you want to keep, and for each bug you fix.
• pytest finds test_*.py files and test_* functions, uses plain assert, and has handy flags: -q, -v, -k, -x.
• TestClient(app) calls your API without a server. Install pytest and httpx2 (Starlette 1.7 warns about plain httpx).
• Test the success path, the 422 and 404 errors (check type and loc) and the boundaries on both sides of every limit.
• Read failures from the bottom: the E line says what differed, the > line is where, and a traceback's last line names the error. Use breakpoint() or --pdb when you need to look inside.
• Keep tests in tests/ and share set-up with fixtures in conftest.py.

Next: Project, Part 1: Build a Data Explorer API. You will combine everything so far, Git, requests, FastAPI, Pydantic and pytest, into one small project.