• Data Version Control (DVC) is an open source tool that brings Git like versioning to datasets, models, and pipelines, letting you track changes to large files without storing them directly in Git.

  • DVC works by storing lightweight metafiles (.dvc files) in your Git repo that point to the actual data, while the real files live in remote storage (S3, GCS, Azure Blob, etc.); this way Git tracks the pointers and DVC handles pulling/pushing the actual large files.

    • DVC also supports local remotes, you can point it to a folder on your own machine or an external drive as the storage backend instead of the cloud, useful for offline work or testing, though it won't be as durable or accessible as cloud storage.

  • DVC is cheap and durable because it relies on remote storage backends (S3, GCS, Azure Blob, etc.) for the actual data, so you pay standard cloud storage rates instead of bloating your Git host, and your data stays safe and versioned even if your local machine or repo is lost.

  • DVC's workflow mirrors Git closely: you init once per repo, then add files to start tracking them, and every time the data changes you run add again to create a new version; the actual commands stay almost identical to Git's muscle memory.

    # initialize DVC in your repo, generates a .dvc/ folder (similar to .git/)
dvc init

# start tracking a file/folder, generates a data.csv.dvc metafile (a pointer file)
dvc add data/data.csv

# push the actual data to the configured remote storage
dvc push

# pull the actual data down from remote storage (e.g. after cloning the repo)
dvc pull

# check status of tracked files (like git status)
dvc status

  • The .dvc metafile (not the actual data) is what gets committed to Git, this keeps the repo lightweight while still tracking exactly which data version belongs to which code commit.

  • To use an S3 bucket as remote storage, you configure it once and DVC handles push/pull to it from then on and push the config file to the git repo.

        # add an S3 bucket as the default remote storage
    dvc remote add -d myremote s3://my-bucket/dvc-storage

  • Typical team flow: make changes to the dataset, run dvc add to version it, dvc push to upload the data to remote storage, then git add/git commit/git push the updated .dvc file so colleagues can git pull and dvc pull to sync both code and data together.