Git for AI Projects: Data, Models, LFS, DVC and Leaked Secrets

Git was built for source code: many small text files that change a few lines at a time. An AI project also has datasets of hundreds of megabytes, model files that change completely every time you retrain, notebooks full of output, an environment that has to be rebuilt exactly, and API keys for model providers that must never be shared. Put all of that into Git the naive way and the repository becomes slow, huge, unreproducible or, worst of all, a public copy of your keys.

This page sets up an AI repository the way experienced teams do: what goes into Git, what goes next to it (Git LFS, DVC), how a lockfile makes the environment reproducible, and what to do, in the right order, when a secret has been committed.

What you will learn

• A clean layout and a .gitignore for AI projects, checked with git check-ignore
• Reproducible environments with uv.lock, and why the lockfile is committed
• Why large files hurt Git, and how Git LFS and DVC solve it differently (with a mini-DVC you build yourself)
• Choosing between plain Git, LFS, DVC and other storage
• Finding a secret anywhere in history, removing it, and the step that matters more: rotating it

An AI repository layout

There is no single standard, but most AI repositories that age well separate the same things:

sentiment-lab/
├── pyproject.toml        dependencies (Git)
├── uv.lock               exact versions of everything (Git)
├── .gitignore  .gitattributes  .env.example   (Git)
├── src/sentiment_lab/    importable code: features, training, serving (Git)
├── notebooks/            exploration; outputs stripped before commit (Git)
├── prompts/              prompt templates, versioned like code (Git)
├── configs/              training and model settings, YAML (Git)
├── evals/                evaluation sets and scripts (Git, small sets)
├── tests/                (Git)
├── data/
│   ├── raw/              original data, never edited (DVC or a bucket, not Git)
│   └── processed/        produced by code, can be rebuilt (DVC or ignored)
├── models/               trained artefacts (LFS, DVC or a model registry)
└── .env                  real keys (never committed)

The rule behind it: Git stores what people write (code, configs, prompts, small eval sets, documentation). Things that machines produce or that are big are either rebuilt from code, or stored somewhere built for large files, with Git keeping only a small reference to them.

A .gitignore for AI projects

Nadia starts the project in a new repository:

mkdir -p ~/projects/sentiment-lab
cd ~/projects/sentiment-lab
git init -q
mkdir -p src/sentiment_lab notebooks prompts data/raw data/processed models

# file: .gitignore
# Python
__pycache__/
*.py[cod]
.venv/
.pytest_cache/
.ruff_cache/

# Secrets: commit .env.example, never .env
.env
.env.*
!.env.example

# Notebooks
.ipynb_checkpoints/

# Data and models: tracked with DVC or LFS, or rebuilt by code.
# Keep the folders (.gitkeep), DVC pointers (*.dvc) and DVC's own .gitignore files.
/data/raw/*
/data/processed/*
!/data/**/.gitkeep
!/data/**/*.dvc
!/data/**/.gitignore
/models/*.pkl
/models/*.joblib
/models/*.safetensors
/models/*.onnx

# Experiment tracking and caches
mlruns/
wandb/
.cache/

# Editors and OS
.vscode/
.idea/
.DS_Store

Patterns are easy to get subtly wrong, so test them. git check-ignore -v prints the file, line and pattern that ignores a path; -n also lists paths that are not ignored (with :: in front):

touch .env .env.example .env.production data/raw/.gitkeep data/raw/reviews.csv models/model.joblib notebooks/explore.ipynb
git check-ignore -v -n .env .env.example .env.production data/raw/.gitkeep data/raw/reviews.csv models/model.joblib notebooks/explore.ipynb

.gitignore:10:.env	.env
.gitignore:12:!.env.example	.env.example
.gitignore:11:.env.*	.env.production
.gitignore:21:!/data/**/.gitkeep	data/raw/.gitkeep
.gitignore:19:/data/raw/*	data/raw/reviews.csv
.gitignore:25:/models/*.joblib	models/model.joblib
::	notebooks/explore.ipynb

Notice the order of the rules. .env.* would also ignore .env.example; the later !.env.example takes it back. /data/raw/* ignores the folder's contents rather than the folder itself, because Git cannot un-ignore a file inside an ignored folder; that is why the empty .gitkeep file can keep the folder in the repository.

Notebooks are text (JSON), but they store every output, image and error in the file, which makes diffs unreadable and can leak data you printed. Most teams strip outputs before committing (with nbstripout or a pre-commit hook) or keep notebooks as .py files with Jupytext.

A reproducible environment: uv.lock

"Works on my machine" in AI usually means a different version of NumPy, scikit-learn or a tokenizer. pyproject.toml says what you depend on (scikit-learn>=1.9); uv.lock records the exact version and file hash of every package, including the dependencies of your dependencies, for every platform:

uv init -q --bare
uv add -q scikit-learn joblib
cat pyproject.toml
grep -c '^\[\[package\]\]' uv.lock

[project]
name = "sentiment-lab"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
    "joblib>=1.6.0",
    "scikit-learn>=1.9.1",
]
8

Two direct dependencies, eight locked entries: the project itself, scikit-learn and joblib, and the five packages they pull in (NumPy, SciPy and others), each pinned. Commit both files. A teammate, a CI job or a Docker build then runs uv sync --locked, which installs exactly what the lockfile says and fails instead of silently updating if pyproject.toml and uv.lock disagree:

uv sync -q --locked
git add .gitignore pyproject.toml uv.lock data/raw/.gitkeep
git commit -q -m "Start sentiment-lab: layout, .gitignore, locked dependencies"
git ls-files

.gitignore
data/raw/.gitkeep
pyproject.toml
uv.lock

.env, the CSV, the model, .venv/: none of it is in Git. With pip, the equivalent is a requirements.txt with pinned versions and hashes (pip install --require-hashes -r requirements.txt, generated by uv pip compile --generate-hashes or pip-tools). Lock the Python version too (requires-python, or a .python-version file); this course standardises on 3.14.

Why big files hurt Git

Git keeps every version of every file forever, and every clone downloads all of them. For a 5 MB model that is retrained three times and then deleted:

for v in 1 2 3; do
  python -c "import random; random.seed($v); open('models/sentiment.bin', 'wb').write(random.randbytes(5_000_000))"
  git add models/sentiment.bin
  git commit -q -m "Model v$v"
done
git rm -q models/sentiment.bin
git commit -q -m "Remove the model from Git"
git gc -q
git count-objects -vH | grep size-pack

size-pack: 14.31 MiB

The model is gone from the working folder, yet the repository holds all three versions, 14 MB that every git clone will download forever. Model weights barely compress and two versions share almost nothing, so delta compression cannot help. Now picture a 400 MB model retrained every week. GitHub warns about files over 50 MB, rejects files over 100 MB, and recommends keeping repositories well under a few gigabytes. Deleting the file does not help; only rewriting history does (as with secrets, below).

We undo those four commits so they don't follow us around for the rest of the page (this is our own unpublished history, so rewriting it is fine):

git reset -q --hard HEAD~4
git reflog expire --expire=now --all
git gc -q --prune=now
git count-objects -vH | grep size-pack

size-pack: 2.67 KiB

Git LFS: big files outside, pointers inside

Git LFS (Large File Storage, version 3.x) is a Git extension. You tell it which patterns are large; when you git add such a file, a clean filter stores the real content in .git/lfs/objects and puts a tiny pointer file into Git instead. On push, the content goes to a separate LFS server (GitHub has one built in); on checkout, a smudge filter downloads the content and swaps it back in. Your working folder looks normal; Git's history only contains pointers.

Git LFS is a separate program (git lfs version shows whether you have it; install it from git-lfs.com or your package manager; Git for Windows includes it). It was not installed where this page's code was checked, so the git lfs commands below are shown with the output Git LFS 3.x prints but were not run here.

git lfs install                 # once per machine: registers the LFS filters in ~/.gitconfig

Then, once per repository, say which files LFS should handle:

git lfs track "models/*.bin"
cat .gitattributes

Tracking "models/*.bin"
models/*.bin filter=lfs diff=lfs merge=lfs -text

.gitattributes is the important part: it is committed, so every clone applies the same rule. -text stops Git from changing line endings in binary files. After that, nothing changes in your workflow: git add models/sentiment.bin, git commit, git push. What Git actually stores is a pointer like this. We can compute the exact pointer for our model with plain Python, because it is just the SHA-256 and size of the content:

import hashlib
import random

random.seed(3)
content = random.randbytes(5_000_000)   # model v3 from above
pointer = (
    "version https://git-lfs.github.com/spec/v1\n"
    f"oid sha256:{hashlib.sha256(content).hexdigest()}\n"
    f"size {len(content)}\n"
)
print(pointer, end="")
print(len(pointer.encode()), "bytes in Git instead of", len(content))

version https://git-lfs.github.com/spec/v1
oid sha256:8e220a7fd1cc035ec59dfdf85ee3f995c70c62f28ee6f66bd6ed45877f36ea65
size 5000000
132 bytes in Git instead of 5000000

Useful commands, all real Git LFS subcommands:

Costs: GitHub's free plan includes 10 GiB of LFS storage and 10 GiB of bandwidth per month (older tutorials say 1 GB; that is out of date). Bandwidth is counted for every clone and CI run that downloads the files, which is where teams get surprised: a CI job that checks out 2 GB of models on every push uses the monthly quota in five runs. Check current limits on GitHub's billing pages, and in CI fetch LFS files only in the jobs that need them (lfs: false is the default in actions/checkout).

DVC: data versioning beside Git

DVC (Data Version Control, 3.x) takes the same pointer idea further. Content goes into a cache and to your own storage (S3, Google Cloud Storage, Azure, SSH, a shared drive), not GitHub; Git stores small .dvc files. On top, DVC describes pipelines (prepare → train → evaluate) so it knows which outputs to rebuild when data or code changes. The project moved from Iterative to lakeFS (Treeverse) in November 2025; it is still maintained and widely used, but development has slowed, so treat it as a solid tool rather than a fast-moving one.

The idea, in 30 lines of Python

Before using DVC, build its core yourself. add() copies a file into a cache named by its MD5 hash (the same files/md5/ab/cdef… layout DVC 3 uses), writes a .dvc pointer in DVC's format, and makes Git ignore the real file. checkout() restores the content a pointer names:

import hashlib
import shutil
from pathlib import Path

CACHE = Path(".minidvc/cache/files/md5")


def md5_of(path):
    h = hashlib.md5()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()


def add(path):
    """Cache the file by content hash, write a pointer for Git, ignore the real file."""
    path = Path(path)
    digest = md5_of(path)
    cached = CACHE / digest[:2] / digest[2:]
    cached.parent.mkdir(parents=True, exist_ok=True)
    shutil.copyfile(path, cached)
    pointer = path.with_name(path.name + ".dvc")
    pointer.write_text(f"outs:\n- md5: {digest}\n  size: {path.stat().st_size}\n"
                       f"  hash: md5\n  path: {path.name}\n")
    ignore = path.parent / ".gitignore"
    rule = f"/{path.name}\n"
    if rule not in (ignore.read_text() if ignore.exists() else ""):
        with open(ignore, "a") as f:
            f.write(rule)


def checkout(pointer):
    """Put back the content that a .dvc pointer names."""
    pointer = Path(pointer)
    digest = pointer.read_text().split("md5: ")[1].split()[0]
    shutil.copyfile(CACHE / digest[:2] / digest[2:], pointer.with_name(pointer.name[:-4]))

The dataset: 1,000 labelled reviews. Version 1 is added and its pointer committed:

import csv

def write_reviews(n):
    with open("data/raw/reviews.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["id", "text", "label"])
        for i in range(n):
            w.writerow([i, f"review number {i}", "positive" if i % 3 else "negative"])

write_reviews(1000)
add("data/raw/reviews.csv")
print(Path("data/raw/reviews.csv.dvc").read_text(), end="")
print(Path("data/raw/.gitignore").read_text(), end="")

outs:
- md5: 10bbe7d39438b5716cf45052a9fb0e39
  size: 31795
  hash: md5
  path: reviews.csv
/reviews.csv

echo ".minidvc/" >> .gitignore
git add .gitignore data/raw/reviews.csv.dvc data/raw/.gitignore
git commit -q -m "Track reviews.csv v1 (1000 rows)"
git status --short --ignored data

!! data/raw/reviews.csv

Git tracks a pointer of under a hundred bytes; the 31 KB CSV itself is ignored. Now the data team adds 500 reviews, and Nadia commits version 2:

write_reviews(1500)
add("data/raw/reviews.csv")

git commit -q -am "Track reviews.csv v2 (1500 rows)"
git log --oneline -2
git diff HEAD~1 -- data/raw/reviews.csv.dvc

3483dd4 Track reviews.csv v2 (1500 rows)
389fbfc Track reviews.csv v1 (1000 rows)
diff --git a/data/raw/reviews.csv.dvc b/data/raw/reviews.csv.dvc
index 8c09b27..c188e9f 100644
--- a/data/raw/reviews.csv.dvc
+++ b/data/raw/reviews.csv.dvc
@@ -1,5 +1,5 @@
 outs:
-- md5: 10bbe7d39438b5716cf45052a9fb0e39
-  size: 31795
+- md5: bc72a9bc77a761b3cf4e81c669e04329
+  size: 48795
   hash: md5
   path: reviews.csv

That diff is the whole history of the dataset in Git: one hash and one size per version. To reproduce last week's model, go back to the old pointer and restore the data it names:

git checkout -q HEAD~1 -- data/raw/reviews.csv.dvc

checkout("data/raw/reviews.csv.dvc")
print(sum(1 for _ in open("data/raw/reviews.csv")) - 1, "rows")
print(sorted(p.parent.name + p.name for p in CACHE.rglob("*") if p.is_file()))

1000 rows
['10bbe7d39438b5716cf45052a9fb0e39', 'bc72a9bc77a761b3cf4e81c669e04329']

Both versions sit in the cache; the pointer picks one. Real DVC adds what this toy lacks: pushing the cache to remote storage and pulling it on another machine, links instead of copies for big files, directories, pipelines and metrics. Put the newest pointer and its data back before moving on:

git checkout -q HEAD -- data/raw/reviews.csv.dvc

checkout("data/raw/reviews.csv.dvc")
print(sum(1 for _ in open("data/raw/reviews.csv")) - 1, "rows")

1500 rows

The real DVC commands

DVC is a Python package (uv add --dev dvc or pip install dvc, plus an extra like dvc[s3] for cloud storage). It was not available in the offline environment where this page was checked, so these commands are shown, not run.

pip install dvc
dvc init                                    # creates .dvc/ and .dvcignore and stages them in Git
dvc add data/raw/reviews.csv                # cache + data/raw/reviews.csv.dvc + data/raw/.gitignore
git add data/raw/reviews.csv.dvc data/raw/.gitignore
git commit -m "Track reviews.csv with DVC"

dvc remote add -d storage s3://bitbyte-ml/dvc   # or a folder: dvc remote add -d storage /mnt/shared/dvc
git commit -am "Configure DVC remote"
dvc push                                    # upload cached data to the remote

# a teammate, or CI:
git clone https://github.com/bitbyte-ai/sentiment-lab.git
cd sentiment-lab
dvc pull                                    # download the data the pointers name

# go back to an old dataset version:
git checkout v1.0 -- data/raw/reviews.csv.dvc
dvc checkout data/raw/reviews.csv.dvc

The .dvc file real DVC 3 writes has exactly the shape of ours (outs, md5, size, hash: md5, path). Pipelines live in dvc.yaml: each stage lists its command, dependencies and outputs, and dvc repro re-runs only the stages whose inputs changed, while dvc.lock records the hashes of what was used.

stages:
  prepare:
    cmd: python src/sentiment_lab/prepare.py
    deps:
      - src/sentiment_lab/prepare.py
      - data/raw/reviews.csv
    outs:
      - data/processed/train.csv
  train:
    cmd: python src/sentiment_lab/train.py
    deps:
      - src/sentiment_lab/train.py
      - data/processed/train.csv
    params:
      - train.C
    outs:
      - models/model.joblib
    metrics:
      - metrics.json:
          cache: false

Git, LFS, DVC or something else?

At larger scale, teams also use: Hugging Face Hub repositories for models and datasets (Git-based, with its Xet storage backend for large files), lakeFS (Git-like branches over a data lake), plain object storage with versioning switched on plus a manifest file in Git, and model registries (MLflow, Weights & Biases artifacts, cloud ML platforms) that version models together with their metrics and lineage. The principle is the same everywhere: Git holds a small, reviewable reference; the bytes live elsewhere.

Leaked secrets

An AI project has more keys than most: model providers, vector databases, Hugging Face, cloud storage. Bots scan public GitHub continuously and use leaked LLM keys within minutes, usually to run up a bill. Here is the classic mistake, step by step:

cat > src/sentiment_lab/config.py <<'EOF'
MODEL = "gpt-5-mini"
OPENAI_API_KEY = "sk-proj-FAKEa1b2c3d4e5f6g7h8i9j0"
EOF
git add src/sentiment_lab/config.py
git commit -q -m "Add LLM config"
# ... pushed to GitHub; the next day a teammate spots the key ...
cat > src/sentiment_lab/config.py <<'EOF'
import os

MODEL = "gpt-5-mini"
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
EOF
git commit -q -am "Read the API key from the environment"
grep -c "sk-proj" src/sentiment_lab/config.py

0

The current file is clean. The history is not.

Finding a secret anywhere in history

Three Git tools search history rather than the current files:

git log --oneline -S "sk-proj-"
git log --oneline -G "sk-[A-Za-z0-9_-]{20,}"
git grep -n "sk-proj-" $(git rev-list --all)

6337f52 Read the API key from the environment
b0525ca Add LLM config
6337f52 Read the API key from the environment
b0525ca Add LLM config
b0525ca83816a1883d43688dedd9c31fd5626396:src/sentiment_lab/config.py:2:OPENAI_API_KEY = "sk-proj-FAKEa1b2c3d4e5f6g7h8i9j0"

• -S "text" (the "pickaxe") lists commits where the number of occurrences of the text changed: the commit that added it and the one that removed it. Add -p to see the diffs.
• -G regex lists commits whose diff has an added or removed line matching a regular expression: use it when you don't know the exact secret, only its shape.
• git grep text $(git rev-list --all) searches the full content of every commit and prints commit:file:line:text.

Dedicated scanners do this with hundreds of patterns for known key formats, plus entropy checks: gitleaks (gitleaks git -v scans the whole history), TruffleHog (which can also check whether a found key is still live), and GitHub's own secret scanning, which is on and free for public repositories.

Step 1, always: rotate the key

Once a key has been pushed to a remote anyone else can read, assume it is compromised. Cleaning history does not un-leak it: forks, clones, CI logs, caches and the bots already have it. So the order is fixed:

1. Revoke or rotate the key in the provider's dashboard, and put the new key in .env or your secret manager. This alone makes the leak harmless.
2. Check the damage: the provider's usage and billing pages, audit logs; set or lower spending limits.
3. Clean the history, if the repository is public or the secret is sensitive beyond the key itself (passwords, personal data).
4. Prevent the next one: .env in .gitignore, push protection, a pre-commit scanner.

Not pushed yet: fix it locally

If the commit with the secret has not left your machine, fixing it is cheap. Say Nadia commits a Hugging Face token, notices immediately, and undoes the commit:

echo 'HF_TOKEN = "hf_FAKEabcdefghijklmnopqrstuvwxyz"' >> src/sentiment_lab/config.py
git commit -q -am "Add HF token"
git reset -q --hard HEAD~1
git log --oneline -1
git reflog -3

6337f52 Read the API key from the environment
6337f52 HEAD@{0}: reset: moving to HEAD~1
c61a854 HEAD@{1}: commit: Add HF token
6337f52 HEAD@{2}: commit: Read the API key from the environment

The commit has left the branch, but it still exists: the reflog remembers it, and its objects stay in .git until garbage collection removes unreachable objects, normally after weeks. Anyone who copies the .git folder (a zip sent to a colleague, a Docker image built with COPY . .) copies the token:

git show HEAD@{1}:src/sentiment_lab/config.py | grep HF_TOKEN
git reflog expire --expire=now --all
git gc -q --prune=now
git show HEAD@{1}:src/sentiment_lab/config.py
git cat-file -t c61a854

HF_TOKEN = "hf_FAKEabcdefghijklmnopqrstuvwxyz"
fatal: log for HEAD is empty
fatal: Not a valid object name c61a854

The reflog is empty and garbage collection has deleted the commit's objects: Git no longer knows the hash. Now it is really gone from this repository. For the OpenAI key above, which "was pushed", the same reset would not be enough: the remote and every clone still have it.

Pushed: rewrite history with git filter-repo

git filter-repo is the tool GitHub and the Git project recommend for removing data from history (the old git filter-branch is slow and easy to misuse; Git itself warns against it). It is a single Python script, not part of Git, and was not installed where this page was checked, so these commands are shown, not run. Work in a fresh clone; filter-repo refuses to run in a repository that is not one, so you cannot destroy your working copy by accident:

pip install git-filter-repo
git clone https://github.com/bitbyte-ai/sentiment-lab.git sentiment-lab-clean
cd sentiment-lab-clean
# Each line: text to find ==> replacement. (A line without ==> is replaced by ***REMOVED***.)
echo 'sk-proj-FAKEa1b2c3d4e5f6g7h8i9j0==>REMOVED-OPENAI-KEY' > ../replacements.txt
git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
git log --oneline -S "sk-proj-"            # prints nothing now
git push --force --mirror origin

To remove a whole file from every commit instead (a committed .env, a dataset with personal data), use git filter-repo --sensitive-data-removal --invert-paths --path .env. Every commit from the first changed one onwards gets a new hash, so:

• Coordinate first: everyone pushes their work, then re-clones after the rewrite. An old clone that pushes or merges brings the secret straight back.
• On GitHub, the old commits can stay reachable through cached views and pull-request refs; GitHub's guide "Removing sensitive data from a repository" explains how to ask GitHub Support to purge them. Forks keep their copy.
• BFG Repo-Cleaner (java -jar bfg.jar --replace-text replacements.txt on a mirror clone, then git reflog expire --expire=now --all && git gc --prune=now --aggressive) is an older, faster alternative; by default it does not touch the latest commit, so fix the current files first.

Prevention

On GitHub, push protection checks pushes for known key formats and blocks them; it is on by default for your account when you push to public repositories. A blocked push fails with GH013: Repository rule violations found and "Push cannot contain secrets", names the file, line and commit, and gives a link to bypass it if it is a false positive. Locally, a pre-commit hook running a scanner catches the secret even earlier, before the commit exists (you write one in the Hard exercise). Better still, never have a key in a file Git can see: read it from the environment, as Environment Variables, .env Files and Virtual Environments taught.

Real-world use

• Reproducing a model. "Which data and code produced the model in production?" is answered by one Git commit: the code, uv.lock, the config and the .dvc pointers together. Teams tag that commit when they release a model.
• Hugging Face model repositories are Git repositories with large-file storage; git clone of a model repo shows the same pointer mechanism you saw here.
• Incident response. Real leaks are handled exactly in the order above. The first question in the incident channel is "has the key been rotated?", not "has the commit been removed?".
• CI caches. Lockfiles are also cache keys: CI restores the virtual environment when uv.lock is unchanged, which makes builds fast and identical.

Common mistakes

1. Deleting a big file or a secret in a new commit and thinking it is gone. History keeps it. For size, rewrite history (git lfs migrate, filter-repo); for secrets, rotate first.
2. Not committing the lockfile (or adding uv.lock to .gitignore). Every install then resolves fresh, and two machines get different versions. Commit it, install with --locked.
3. Ignoring a folder, then trying to keep one file in it. data/ plus !data/.gitkeep does not work; ignore data/* instead.
4. Tracking with LFS after the fact without migrating. git lfs track only affects files added from then on; old big blobs stay in history until git lfs migrate import.
5. Cleaning history before rotating the key. The key stays valid in every clone, fork and bot's database while you work on the rewrite. Rotate first, always.

Try it yourself

1. Easy: Add rules to .gitignore so that everything in notebooks/outputs/ is ignored and *.parquet files are ignored anywhere, then prove both with git check-ignore -v.
2. Medium: Using the mini-DVC functions, write status(pointer) that prints unchanged or modified by comparing the file's current MD5 with the pointer (what dvc status does). Test it before and after appending a row.
3. Hard: Write a pre-commit hook that refuses a commit if any staged line being added matches sk-[A-Za-z0-9_-]{20,} or hf_[A-Za-z0-9]{30,}, and prove it blocks a commit with a fake key. (Hint: git diff --cached -U0 shows only the staged changes.)

Answers

# Easy
printf '/notebooks/outputs/\n*.parquet\n' >> .gitignore
mkdir -p notebooks/outputs data/processed
touch notebooks/outputs/plot.png data/processed/features.parquet
git check-ignore -v notebooks/outputs/plot.png data/processed/features.parquet

.gitignore:39:/notebooks/outputs/	notebooks/outputs/plot.png
.gitignore:40:*.parquet	data/processed/features.parquet

data/processed/features.parquet matches two rules, /data/processed/* and *.parquet. When several patterns match, the last one decides, and -v reports that one.

# Medium
def status(pointer):
    pointer = Path(pointer)
    recorded = pointer.read_text().split("md5: ")[1].split()[0]
    current = md5_of(pointer.with_name(pointer.name[:-4]))
    print(pointer.name[:-4], "unchanged" if current == recorded else "modified")

status("data/raw/reviews.csv.dvc")
with open("data/raw/reviews.csv", "a") as f:
    f.write("1500,one more review,positive\n")
status("data/raw/reviews.csv.dvc")

reviews.csv unchanged
reviews.csv modified

# Hard
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
if git diff --cached -U0 | grep '^+' | grep -Eq 'sk-[A-Za-z0-9_-]{20,}|hf_[A-Za-z0-9]{30,}'; then
  echo "pre-commit: this commit adds something that looks like an API key." >&2
  echo "Put it in .env and read it with os.environ instead." >&2
  exit 1
fi
EOF
chmod +x .git/hooks/pre-commit
echo 'KEY = "sk-proj-FAKEzzzzzzzzzzzzzzzzzzzzzz"' > src/sentiment_lab/keys.py
git add src/sentiment_lab/keys.py
git commit -q -m "Add keys"
echo "exit code: $?"

pre-commit: this commit adds something that looks like an API key.
Put it in .env and read it with os.environ instead.
exit code: 1

Hooks in .git/hooks are not committed or shared, so teams install them with a tool such as the pre-commit framework (with gitleaks' ready-made hook) and still rely on server-side push protection as the safety net.

Summary

• Git stores what people write; data and models go to LFS, DVC or other storage, with small pointers in Git.
• Test .gitignore with git check-ignore -v; ignore folder contents when you need exceptions; never commit .env.
• Commit uv.lock and install with uv sync --locked for identical environments everywhere.
• Git LFS swaps big files for pointers on your Git host (10 GiB free on GitHub); DVC versions data in your own storage and adds pipelines.
• Find secrets with git log -S/-G and git grep $(git rev-list --all); rotate first, then rewrite with git filter-repo, then prevent with push protection and pre-commit scanning.

Next: HTTP in Depth: Idempotency, Caching and Auth moves from repositories to the APIs your AI services expose: which requests are safe to retry, how caching with ETags works, and how OAuth really logs users in.