usage: dvc commit [-h] [-q | -v] [-f] [-d] [-R] [targets [targets ...]] positional arguments: targets Limit command scope to these stages or .dvc files. Using -R, directories to search for stages or .dvc files can also be given.
dvc commit command is useful for several scenarios, when data already
tracked by DVC changes: when a stage or
pipeline is in development/experimentation; when
manually editing or generating DVC outputs; or to force update the
.dvc files without reproducing stages or pipelines. These
scenarios are further detailed below.
Code or data for a stage is under active development, with multiple iterations
(experiments) in code, configuration, or data. Use the
--no-commit option of
DVC commands (
dvc repro) to avoid caching unnecessary
data repeatedly. Use
dvc commit when the DVC-tracked data is final.
dvc unprotect). Once a desirable result is reached, use
dvc commitas appropriate to update the
.dvcfiles and store changed data to the cache.
dvc committo force update the
.dvcfiles and cache.
Let's take a look at what is happening in the first scenario closely. Normally
DVC commands like
dvc repro or
dvc run commit the data to the
cache after creating or updating a
.dvc file. What
commit means is that DVC:
.gitignore). (Note that if the project was initialized with no Git support (
dvc init --no-scm), this does not happen.)
There are many cases where the last step is not desirable (for example rapid
iterations on an experiment). The
--no-commit option prevents the last step
from occurring (on the commands where it's available), saving time and space by
not storing unwanted data artifacts. The file hash is still
computed and added to the
.dvc file, but the actual data file is
not saved in the cache. This is where the
dvc commit command comes into play.
It performs that last step (saving the data in cache).
Note that it's best to avoid the last two scenarios. They essentially
.dvc files and save data to cache. They are
still useful, but keep in mind that DVC can't guarantee reproducibility in those
--with-deps- determines files to commit by tracking dependencies to the target stages or
.dvcfiles. If no
targetsare provided, this option is ignored. By traversing all stage dependencies, DVC searches backward from the target stages in the corresponding pipelines. This means DVC will not commit files referenced in later stages than the
--recursive- determines the files to commit by searching each target directory and its subdirectories for stages or
.dvcfiles to inspect. If there are no directories among the
targets, this option is ignored.
--force- commit data even if hash values for dependencies or outputs did not change.
--help- prints the usage/help message, and exit.
--quiet- do not write anything to standard output. Exit with 0 if no problems arise, otherwise 1.
--verbose- displays detailed tracing information from executing the
Let's employ a simple workspace with some data, code, ML models,
pipeline stages, such as the DVC project created for the
Get Started. Then we can see what happens with
git commit and
dvc commit in different situations.
Sometimes we want to iterate through multiple changes to configuration, code, or
data, trying different ways to improve the output of a stage. To avoid filling
the cache with undesired intermediate results, we can run a single
dvc run --no-commit, or reproduce an entire pipeline using
dvc repro --no-commit. This prevents data from being pushed to cache. When
development of the stage is finished,
dvc commit can be used to store data
files in the cache.
src/featurization.py is executed. A useful change to
make is adjusting the parameters for that script. The parameters are defined in
params.yaml file. Updating the value of the
max_features param to 6000
changes the resulting model:
featurize: max_features: 6000 ngrams: 2
This edit introduces a change that would cause the
evaluate stages to execute if we ran
dvc repro. But if we want to try
several values for
max_features and save only the best result to the cache, we
can run it like this:
$ dvc repro --no-commit
We can run this command as many times as we like, editing
params.yaml any way
we like, and so long as we use
--no-commit, the data does not get saved to the
cache. Let's verify that's the case:
$ dvc status featurize: changed outs: not in cache: data/features train: changed outs: not in cache: model.pkl
Now we can look in the cache directory to see if the new version of
is not in cache indeed. Let's look at the latest state of
train: cmd: python src/train.py data/features model.pkl deps: - path: data/features md5: de03a7e34e003e54dde0d40582c6acf4.dir - path: src/train.py md5: ad8e71b2cca4334a7d3bb6495645068c params: params.yaml: train.n_estimators: 100 train.seed: 20170428 outs: - path: model.pkl md5: 9aba000ba83b341a423a81eed8ff9238
To verify this instance of
model.pkl is not in the cache, we must know the
path to the cached file. In the cache directory, the first two characters of the
hash value are used as a subdirectory name, and the remaining characters are the
file name. Therefore, had the file been committed to the cache, it would appear
in the directory
.dvc/cache/9a. Let's check:
$ ls .dvc/cache/9a ls: .dvc/cache/9a: No such file or directory
If we've determined the changes to
params.yaml were successful, we can execute
this set of commands:
$ dvc commit $ dvc status Data and pipelines are up to date. $ ls .dvc/cache/70 ba000ba83b341a423a81eed8ff9238
We've verified that
dvc commit has saved the changes into the cache, and that
the new instance of
model.pkl is there.
It is also possible to execute the commands that are executed by
dvc repro by
hand. You won't have DVC helping you, but you have the freedom to run any
command you like, even ones not defined in
dvc.yaml stages. For example:
$ python src/featurization.py data/prepared data/features $ python src/train.py data/features model.pkl $ python src/evaluate.py model.pkl data/features auc.metric
Sometimes we want to clean up a code or configuration file in a way that doesn't cause a change in its results. We might write in-line documentation with comments, change indentation, remove some debugging printouts, or any other change that doesn't produce different output of pipeline stages.
$ git status -s M src/train.py $ dvc status train.dvc: changed deps: modified: src/train.py
Let's edit one of the source code files. It doesn't matter which one. You'll see that both Git and DVC recognize a change was made.
If we ran
dvc repro at this point, this pipeline would be reproduced. But
since the change was inconsequential, that would be a waste of time and CPU.
That's especially critical if the corresponding stages take lots of resources to
$ git add src/train.py $ git commit -m "CHANGED" [master 72327bd] CHANGED 1 file changed, 2 insertions(+) $ dvc commit dependencies ['src/train.py'] of 'train.dvc' changed. Are you sure you commit it? [y/n] y $ dvc status Data and pipelines are up to date.
Instead of reproducing the pipeline for changes that do not produce different
results, just use
commit on both Git and DVC.