Git for AI
Use version control to track AI code, prompts, configs, and deployment changes.
AI projects contain more than model code. They include notebooks, feature logic, prompts, evaluation suites, configurations, API services, infrastructure, documentation, and references to data and model artifacts. Git provides a reviewable history for the text-based parts of that system and connects changes to MLOps pipelines.
What Is Git?
Git is a distributed version-control system. It records snapshots of a repository, lets developers create parallel branches, compares changes, supports collaborative review, and preserves history that can be inspected or restored.
Instead of folders named Final, Final-New, and Final-Latest, Git identifies each committed state with a unique hash and records its authorship, timestamp, parent history, and message.
Why Git Matters in AI and MLOps
- Tracks training, evaluation, inference, API, and data-processing code.
- Versions prompts, schemas, tests, deployment manifests, and configuration templates.
- Enables peer review and collaboration without overwriting another person's work.
- Triggers continuous integration, model-training, packaging, and deployment pipelines.
- Connects experiments and model artifacts to the code revision that produced them.
- Supports investigation, rollback, audit history, and reproducible releases.
- Makes documentation and operational runbooks evolve beside the implementation.
What Belongs in Git?
| Usually Track | Usually Store Elsewhere |
|---|---|
| Source code and tests | Large raw or processed datasets |
| Prompt templates and output schemas | Trained model weights and checkpoints |
| Small configuration templates | Secrets, tokens, passwords, and private keys |
| Environment and dependency definitions | Large generated reports and experiment artifacts |
| Infrastructure and deployment files | Caches, logs, virtual environments, and build output |
| Documentation and runbooks | Personal, confidential, licensed, or restricted data |
| Dataset and model metadata references | Binary artifacts that change frequently |
Store large datasets and models in versioned object storage, artifact repositories, dataset-versioning systems, or model registries. Commit a stable identifier, checksum, manifest, or URI so code can locate the exact approved artifact.
The Basic Git Model
| Area | Meaning |
|---|---|
| Working tree | Files currently checked out and being edited |
| Staging area | Selected changes prepared for the next commit |
| Local repository | Commit history stored on the developer's machine |
| Remote repository | Shared repository used for collaboration and automation |
| Branch | A movable name pointing to a line of commits |
| Tag | A named reference commonly used to mark a release |
Basic Workflow
git clone <repository-url>
cd <repository-directory>
git switch -c feature/add-drift-check
# Edit files and run tests
git status
git diff
git add src/drift.py tests/test_drift.py
git diff --staged
git commit -m "Add input drift validation"
git push -u origin feature/add-drift-checkA pull or merge request then lets teammates review the change and automated checks before it enters the protected main branch.
Common Commands
| Command | Purpose |
|---|---|
| git init | Create a repository in the current directory |
| git clone URL | Copy an existing repository and its history |
| git status | Show branch, staged, modified, and untracked files |
| git diff | Inspect unstaged changes |
| git add FILE | Stage selected changes |
| git commit -m MESSAGE | Create a commit from staged changes |
| git log | Inspect commit history |
| git switch BRANCH | Change branches |
| git fetch | Download remote references without merging |
| git pull | Fetch and integrate remote changes |
| git push | Publish local commits to a remote |
| git restore FILE | Restore working-tree content carefully |
Review Before You Commit
git status
git diff
git add path/to/intended-file.py
git diff --staged
git commit -m "Validate classification output schema"Stage specific files or hunks rather than automatically adding everything. This reduces accidental inclusion of secrets, large artifacts, temporary files, or unrelated changes.
Good Commits
- Represent one coherent change that can be understood and reviewed.
- Include relevant tests, documentation, and configuration updates.
- Avoid mixing formatting, refactoring, dependency changes, and new behavior without reason.
- Use an imperative summary that states the result, such as Add model health endpoint.
- Explain motivation, trade-offs, migration, or risk in the commit body when useful.
- Do not commit generated outputs merely to make the working tree look clean.
| Weak Message | Stronger Message |
|---|---|
| updates | Add schema validation for prediction requests |
| fix bug | Handle empty feature vectors in batch inference |
| model changes | Switch fraud threshold to calibrated validation value |
| config | Set canary traffic to five percent |
| final version | Release inference API v1.4.0 |
Branches and Collaboration
Branches isolate work until it is ready to integrate. Keep branches short-lived and focused, synchronize regularly, and avoid allowing long-lived branches to diverge from the main line for weeks.
- Protect the main branch from direct, unreviewed changes.
- Require passing checks and appropriate reviewers before merge.
- Use descriptive branch names such as feature/batch-inference or fix/token-redaction.
- Delete merged branches to reduce clutter.
- Choose merge, squash, or rebase conventions consistently across the team.
- Do not rewrite history other people may already depend on without coordination.
Pull Requests and Code Review
| Pull Request Section | What to Include |
|---|---|
| Problem | Why the change is necessary |
| Solution | What changed and important design choices |
| Evidence | Tests, evaluation results, screenshots, or benchmark links |
| AI impact | Model, prompt, data, metric, and behavior changes |
| Operational impact | Deployment, migration, monitoring, cost, and rollback |
| Risk | Security, privacy, fairness, compatibility, and known limitations |
| Review guidance | Files or decisions requiring special attention |
Review AI changes for more than syntax. Ask whether the evaluation is valid, data use is authorized, leakage is prevented, metrics reflect the decision, model artifacts are traceable, and production safeguards are sufficient.
Handling Merge Conflicts
A conflict occurs when Git cannot combine changes automatically. Read both versions, understand the intended final behavior, edit the file, run tests and evaluations, then mark the resolution. Do not choose one side blindly in model configuration, prompt, dependency, or data-schema conflicts.
git status
# Edit conflicted files and remove conflict markers
git add path/to/resolved-file
# Complete the merge or rebase according to the team's workflow
# Run tests and AI evaluations before pushing.gitignore for AI Projects
# Python
__pycache__/
*.py[cod]
.venv/
# Local configuration and secrets
.env
.env.*
!.env.example
*.pem
# Notebooks
.ipynb_checkpoints/
# Data and model artifacts
data/raw/
data/processed/
artifacts/
checkpoints/
*.pkl
*.joblib
*.onnx
*.pt
# Logs and local tools
logs/
.cache/
.DS_StoreAdapt ignore rules to the repository. A file already committed remains tracked after it is added to .gitignore; remove it from tracking deliberately and verify no required shared artifact is lost.
Secrets Must Never Enter Git
- Use environment variables only as runtime inputs, not as a substitute for secret storage.
- Use an approved secret manager or CI/CD encrypted-secret facility.
- Commit a documented .env.example containing names and safe placeholders only.
- Run secret scanning before commits and in continuous integration.
- Limit repository and pipeline permissions using least privilege.
- If a secret is committed, rotate or revoke it immediately; deleting the latest file does not remove it from history.
Notebooks in Git
Jupyter notebooks are JSON documents containing code, text, metadata, and possibly large outputs. They can be difficult to review and merge. Clear unnecessary output, keep cells ordered and reproducible, avoid hidden state, and move reusable logic into tested Python modules.
- Use notebooks for exploration and communication, not as the only production implementation.
- Restart and run all cells before review to expose order dependencies.
- Avoid embedding secrets or private data in cells and outputs.
- Use notebook-aware diff or cleaning tools when adopted by the team.
- Export critical figures or reports through reproducible scripts rather than manual editing.
Large Data and Model Files
Git stores file history and performs poorly when large binaries change repeatedly. Use an artifact or object store, model registry, dataset-versioning tool, or Git Large File Storage where appropriate. The repository should retain a stable reference and integrity information.
dataset:
name: support_tickets
version: 2026-07-20
uri: s3://example-ml-data/support/2026-07-20/
manifest_sha256: <approved-checksum>
model:
registry_name: ticket_classifier
version: 17Repository Structure
ai-project/
├── README.md
├── pyproject.toml
├── .gitignore
├── .env.example
├── src/
│ ├── data/
│ ├── features/
│ ├── training/
│ └── serving/
├── tests/
├── notebooks/
├── prompts/
├── evals/
├── configs/
├── deploy/
├── docs/
└── scripts/The exact structure depends on the project. The important goal is to separate reusable code, experiments, tests, configuration, deployment, and documentation so responsibilities remain clear.
Configuration Management
- Commit safe configuration defaults and schemas.
- Inject environment-specific values during deployment.
- Keep secrets outside the repository.
- Version feature definitions, thresholds, prompt templates, and model-selection rules.
- Validate configuration during CI and application startup.
- Record the resolved production configuration or its immutable reference for traceability.
Dependency and Environment Reproducibility
Commit dependency declarations and lock files appropriate to the ecosystem, plus container or environment definitions when used. Review dependency updates, scan for vulnerabilities, and avoid unbounded versions that can change a build unexpectedly.
Git and Experiment Tracking
An experiment tracker should record the Git commit hash together with dataset version, configuration, environment, metrics, and artifacts. This connects a result to the source that produced it without storing large artifacts in Git.
import subprocess
def current_git_commit() -> str:
return subprocess.check_output(
["git", "rev-parse", "HEAD"], text=True
).strip()
run_metadata = {
"git_commit": current_git_commit(),
"dataset_version": "support-2026-07-20",
"config": "configs/ticket_classifier.yaml",
}Training on an uncommitted or dirty working tree reduces reproducibility. Record that state or require reviewed commits for official candidate models.
Prompts and Evaluations in Git
- Store prompt templates as dedicated files rather than scattered string literals.
- Version system instructions, examples, tool definitions, and output schemas together.
- Keep evaluation inputs, expected behavior, rubrics, and scoring code under review.
- Use synthetic or properly sanitized cases instead of committing sensitive production conversations.
- Require evaluation results when a pull request changes AI behavior.
- Tag releases so production outputs can be traced to a prompt and evaluation version.
Git in CI/CD and MLOps
Pull request opened
-> Lint and unit tests
-> Data-contract and configuration checks
-> Security and secret scans
-> AI evaluation suite
-> Peer approval
-> Merge protected branch
-> Build immutable package or container
-> Train or register model candidate
-> Deploy to staging
-> Validate and approve
-> Progressive production releaseThe commit should identify the source of a build. Promote the same immutable artifact through environments instead of rebuilding different binaries from the same branch name.
Tags and Releases
Tags mark significant repository states such as application releases, model-serving changes, or prompt versions. A release should include notes about behavior, compatibility, migration, evaluation evidence, deployment, monitoring, and rollback.
git tag -a v1.4.0 -m "Release inference API v1.4.0"
git push origin v1.4.0Reverting and Rolling Back
A Git revert creates a new commit that reverses an earlier change while preserving shared history. A production rollback may also require restoring a model, prompt, feature schema, container, infrastructure configuration, and data migration. Git rollback alone is not automatically an operational rollback.
Repository Security
- Require strong authentication and least-privilege repository roles.
- Protect important branches and require reviews and status checks.
- Use signed commits or tags when provenance requirements justify them.
- Review third-party actions, hooks, dependencies, and CI permissions.
- Pin or verify external automation dependencies where possible.
- Prevent untrusted pull-request code from accessing production secrets.
- Audit sensitive changes to deployment, permissions, data, and model-release configuration.
Common Mistakes
| Mistake | Better Practice |
|---|---|
| git add . without review | Inspect status and staged diff; add intended changes |
| Commit secrets, data, or model binaries | Use secret and artifact stores with safe references |
| Huge mixed commits | Create small coherent commits with tests |
| Vague messages | Describe the behavior or result changed |
| Long-lived branches | Integrate small reviewed changes frequently |
| Notebook-only production logic | Move reusable behavior into tested modules |
| Force-pushing shared history | Coordinate and preserve public history |
| Assuming Git equals full reproducibility | Version data, artifacts, configuration, and environment too |
Practical Exercise
mkdir ai-git-practice
cd ai-git-practice
git init
# Create README.md, .gitignore, src/train.py, and tests/test_train.py
git status
git add README.md .gitignore src/train.py tests/test_train.py
git diff --staged
git commit -m "Create reproducible training project structure"
git switch -c feature/add-evaluation
# Add evaluation code and tests
git add src/evaluate.py tests/test_evaluate.py
git commit -m "Add classification evaluation metrics"
git log --oneline --decorate --graph --allExtend the exercise with a remote repository, pull request, automated tests, a safe dataset manifest, and experiment metadata that records the commit hash.
Best Practices
- Track text-based source, tests, prompts, schemas, configuration templates, deployment files, and documentation.
- Store secrets, sensitive data, large datasets, and models in systems designed for them.
- Review diffs and create small, meaningful commits.
- Use short-lived branches, pull requests, protected main branches, and automated checks.
- Connect every experiment, model, build, and deployment to an immutable commit and artifact version.
- Keep notebooks reproducible and move production logic into tested modules.
- Treat repository and CI/CD security as part of the ML supply chain.
- Document releases and remember that model and data rollback extends beyond Git.