by dvc.ai
Get Started
by dvc.ai
Get startedGet Enterprise
HomeInstallGet StartedUse CasesUser Guide
Project StructureData Management
Remote StorageCloud VersioningDiscovering and Accessing DataExternal DataLarge Dataset OptimizationModifying Large Datasets
PipelinesExperiment ManagementHow ToIntegrationsTroubleshootingAnonymized Usage AnalyticsGlossaryUpgrading to DVC 3.0
Command ReferencePython API ReferenceContributingChangelog VS Code Extension StudioDVCLiveGTO
Edit on GitHub

Cloud Versioning

When cloud versioning is enabled, DVC will store files in the remote according to their original directory location and filenames. Different versions of a file will then be stored as separate versions of the corresponding object in cloud storage. This is useful for cases where users prefer to retain their original filenames and directory hierarchy in remote storage (instead of using DVC's usual content-addressable storage format).

Note that not all DVC functionality is supported when using cloud versioned remotes, and using cloud versioning comes with the tradeoff of losing certain benefits of content-addressable storage.

dvc remote storage normally uses content-addressable storage to organize versioned data. Different versions of files are stored in the remote according to hash of their data content instead of according to their original filenames and directory location. This allows DVC to optimize certain remote storage lookup and data sync operations, and provides data de-duplication at the file level. However, this comes with the drawback of losing human-readable filenames without the use of the DVC CLI (dvc get --show-url) or API (dvc.api.get_url()).

When using cloud versioning, DVC does not provide de-duplication, and certain remote storage performance optimizations will be unavailable.

Supported storage providers

Cloud versioning features are only avaible for certain storage providers. Currently, it is supported on the following dvc remote types:

Lifecycle management policies may delete object versions, in which case DVC will be unable to recover those versions. For more information about lifecycle management, see:

Version-aware remotes

When the version_aware option is enabled on a dvc remote:

  • dvc push will utilize cloud versioning when storing data in the remote. Data will retain its original directory structure and filenames, and each version of a file tracked by DVC will be stored as a new version of the corresponding object in cloud storage.
  • dvc fetch and dvc pull will download the corresponding version of an object from cloud storage.

With version_aware enabled, dvc push will modify dvc files. Always dvc push before git commit so that the updated cloud version info is available in Git.

Note that when version_aware is in use, DVC does not delete current versions or restore noncurrent versions of objects in cloud storage. So the current version of an object in cloud storage may not match the version of a file in your DVC repository.

Importing versioned data

DVC supports importing cloud-versioned data from supported storage providers. Refer to dvc import-url (--version-aware) and dvc update --rev for more information.

PrevNext
Content

🐛 Found an issue? Let us know! Or fix it:

Edit on GitHub

❓ Have a question? Join our chat, we will help you:

Discord Chat