Edit on GitHub

exp gc

Remove unnecessary dvc experiments from the project.

Synopsis

usage: dvc exp gc [-h] [-q | -v] [-w]
                  [-a] [-T] [-A] [--queued] [-f]

Description

This command can delete (garbage collect) experiments that exist in the project but are no longer needed.

See How does DVC track experiments? in dvc exp run to learn more about DVC experiment storage.

To avoid accidentally deleting work, dvc exp gc doesn't do anything unless one or a combination of scope options are provided (--workspace, --all-branches, --all-tags, --all-commits). Use these to indicate which experiments are still needed. See the Options section for more details.

Note that after removing experiments, the corresponding data which may no longer be needed still remains in the cache. You can use dvc gc separately to delete it.

Options

  • -w, --workspace - keep only experiments derived from the last commit (HEAD, default base for dvc experiments). This option is enabled automatically with the other scope options (below).
  • -a, --all-branches - keep experiments derived from all Git branches, as well as from the last commit (implies -w). Note that this can be combined with -T below, for example using the -aT flags.
  • -T, --all-tags - keep experiments derived from all Git tags, as well as from the last commit (implies -w). Note that this can be combined with -a above, for example using the -aT flags.
  • -A, --all-commits - keep experiments derived from all Git commits, as well as from the last commit (implies -w). This is mainly needed when clearing --queued experiments (below).
  • --queued - keep also experiments that haven't been run yet (defined via dvc exp run --queue). Another scope option (-w, -a, etc.) is required along with this. The experiment run queue will be cleared if this option isn't used.
  • -f, --force - force garbage collection. Skip confirmation prompt.
  • -h, --help - shows the help message and exit.
  • -q, --quiet - do not write anything to standard output. Exit with 0 if no problems arise, otherwise 1.
  • -v, --verbose - displays detailed tracing information from executing the dvc pull command.

Examples

This example is based on our Get Started, where you can find the actual source code.

Let's say we have the following project, and have just applied and committed exp-1dad0 (current HEAD of master):

$ dvc exp show --all-commits --include-params=featurize
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Experiment            โ”ƒ Created      โ”ƒ     auc โ”ƒ featurize.max_features โ”ƒ featurize.ngrams โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ workspace             โ”‚ -            โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ master                โ”‚ 05:39 PM     โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ 10-bigrams-experiment โ”‚ Jun 20, 2020 โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ โ”œโ”€โ”€ exp-e6c97         โ”‚ Oct 21, 2020 โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ โ”œโ”€โ”€ exp-1dad0         โ”‚ Oct 09, 2020 โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ โ””โ”€โ”€ exp-1df77         โ”‚ Oct 09, 2020 โ”‚ 0.51676 โ”‚ 500                    โ”‚ 2                โ”‚
โ”‚ 9-bigrams-model       โ”‚ Jun 20, 2020 โ”‚ 0.54175 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ โ””โ”€โ”€ exp-069d9         โ”‚ Sep 24, 2020 โ”‚ 0.51076 โ”‚ 2500                   โ”‚ 2                โ”‚
โ”‚ 8-evaluation          โ”‚ Jun 20, 2020 โ”‚ 0.54175 โ”‚ 500                    โ”‚ 1                โ”‚
โ”‚ 7-ml-pipeline         โ”‚ Jun 20, 2020 โ”‚       - โ”‚ 500                    โ”‚ 1                โ”‚
  ...
โ”‚ 0-git-init            โ”‚ Jun 20, 2020 โ”‚       - โ”‚ 1500                   โ”‚ 2                โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

If we consider all the other experiments unnecessary, we can delete them like this:

$ dvc exp gc -w
WARNING: This will remove all experiments except ...
Are you sure you want to proceed? [y/n] y
Removed 4 experiments. To remove unused cache files use 'dvc gc'.

We can confirm that all the previous experiments are gone:

$ dvc exp show --all-commits --include-params=featurize
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Experiment            โ”ƒ Created      โ”ƒ     auc โ”ƒ featurize.max_features โ”ƒ featurize.ngrams โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ workspace             โ”‚ -            โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ master                โ”‚ 05:39 PM     โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ 10-bigrams-experiment โ”‚ Jun 20, 2020 โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ 9-bigrams-model       โ”‚ Jun 20, 2020 โ”‚ 0.54175 โ”‚ 1500                   โ”‚ 2                โ”‚
  ...
โ”‚ 0-git-init            โ”‚ Jun 20, 2020 โ”‚       - โ”‚ 2000                   โ”‚ 2                โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

To remove any cached data associated to the deleted experiments and which are no longer needed in the project, we can use regular dvc gc (with the appropriate options):

$ dvc dvc gc --all-commits
WARNING: This will remove all cache except ...
Are you sure you want to proceed? [y/n] y
...

Note the use of --all-commits to ensure that we do not garbage collect files or directories referenced in remaining commits in the repo.

Content

๐Ÿ› Found an issue? Let us know! Or fix it:

Edit on GitHub

โ“ Have a question? Join our chat, we will help you:

Discord Chat