Reviewing and Comparing Experiments
DVC provides commands to list, tabulate and contrast experiments. Let's see how they can help you streamline the experimentation process.
New! You can compare your ML experiments with DVC directly from Visual Studio Code.
List experiments in the project
You can get a list of existing experiments in the repository with
dvc exp list
. Without any options, this command lists the experiments based on
the latest commit of the current branch (Git HEAD
).
$ dvc exp list
refs/tags/baseline-experiment:
2399f24 [cnn-128]
f1edf21 [cnn-64]
If you want to list all the experiments in the repo regardless of their parent
commit, use the --all-commits
(-A
) flag.
$ dvc exp list -A
refs/tags/baseline-experiment:
2399f24 [cnn-128]
f1edf21 [cnn-64]
main:
1f7e42f [toric-chiv]
List experiments saved remotely
Experiments can be shared (with dvc exp push
) from another location. To
review experiments uploaded to a remote repository (which you may
not have locally), provide a Git remote name to dvc exp list
.
$ dvc exp list origin
refs/tags/baseline-experiment:
cnn-32
cnn-64
This command lists remote experiments based on that repo's HEAD
. You can use
--all-commits
(-A
) to list all experiments, or add any other supported
option to the remote dvc exp list
command.
List experiment names to use in scripts
dvc exp list
may be printing too much information when it comes to feed its
output to other commands. You can get only the names of the experiments via the
--name-only
flag. For example, to download the model files from all
experiments in a remote:
$ for name in $(dvc exp list origin --name-only -A); do
dvc get --rev ${name} \
git@github.com:iterative/example-dvc-experiments.git \
models/model.h5 -o exps/${name}.h5
done
List experiments from a specific project version
The --rev
option allows to specify a Git commit, tag or branch name to list
the experiments that are based on it. For example:
# from a commit:
$ dvc exp list origin --rev 23ceb4a
23ceb4a:
cnn-32
cnn-96
# from a tag:
$ dvc exp list origin --rev baseline-experiment
refs/tags/baseline-experiment:
cnn-64
cnn-128
# from a fully specified Git reference:
$ dvc exp list origin --rev refs/tags/baseline-experiment
refs/tags/baseline-experiment:
cnn-64
cnn-128
Show a table of experiments
Experimentation is about generating many possibilities before selecting a few of
them. You can get a table of experiments with dvc exp show
, which displays all
the metrics (yellow), parameters (blue) and dependencies (violet)
in a nicely formatted table.
$ dvc exp show
────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Experiment Created loss acc train.epochs model.conv_units src data
────────────────────────────────────────────────────────────────────────────────────────────────────────────────
workspace - 0.03332 0.9888 10 16 695e061 6875529
baseline-experiment Jan 14, 2022 0.03332 0.9888 10 16 695e061 6875529
├── 38d6c53 [cnn-64] Jan 19, 2022 0.038246 0.988 10 64 c77a505 6875529
└── bc0faf5 [cnn-128] Jan 19, 2022 0.038325 0.989 10 128 bc75d6a 6875529
────────────────────────────────────────────────────────────────────────────────────────────────────────────────
dvc exp show
only tabulates experiments in the workspace and in HEAD
. You
can use --all-commits
(-A
) flag to show all the experiments in the project
instead.
Note that queued experiments will be marked with an asterisk *
.
Customize the table of experiments
The table output may become cluttered if you have a large number of metrics,
parameters and dependencies. dvc exp show
provides several options to select
the columns to be shown in the table.
As a quick way of reducing noise, dvc exp show --only-changed
will drop any
column with values that do not change across experiments:
$ dvc exp show --only-changed
───────────────────────────────────────────────────────────────────────────────
Experiment Created loss acc model.conv_units
───────────────────────────────────────────────────────────────────────────────
workspace - 0.03332 0.9888 16
baseline-experiment Jan 14, 2022 0.03332 0.9888 16
├── 38d6c53 [cnn-64] Jan 19, 2022 0.038246 0.988 64
└── bc0faf5 [cnn-128] Jan 19, 2022 0.038325 0.989 128
───────────────────────────────────────────────────────────────────────────────
You can also exclude unwanted columns from the table by using the --drop
option. Or make sure that the table includes important columns with --keep
.
See more examples in the
dvc exp show
reference.
By default dvc exp show
sorts the experiments by their timestamp. You can sort
the metrics or parameters columns by the option --sort-by
and --sort-order
.
--sort-by
takes a metric or parameter name, and --sort-order
takes either
asc
or desc
.
$ dvc exp show --sort-by auc --sort-order desc
───────────────────────────────────────────────────
Experiment acc model.conv_units
───────────────────────────────────────────────────
workspace 0.9127 16
baseline-experiment 0.9127 16
├── 6d13f33 [cnn-64] 0.9153 64
└── 69503c6 [cnn-128] 0.916 128
───────────────────────────────────────────────────
Get experiments table in CSV
dvc exp show
can also output the table in CSV, with --csv
. It includes all
the data found in the table.
$ dvc exp show --csv
Experiment,rev,typ,Created,parent,loss,acc,train.epochs,model.conv_units
,workspace,baseline,,,0.236574187874794,0.9126999974250793,10,16
baseline-experiment,23ceb4a,baseline,2021-09-06T23:38:07,,0.236574187874794,0.9126999974250793,10,16
cnn-64,6d13f33,branch_commit,2021-09-09T13:06:05,,0.2338544875383377,0.9153000116348267,10,64
cnn-128,69503c6,branch_commit,2021-09-09T12:53:51,,0.2324332743883133,0.9160000085830688,10,128
For example, let's parse the CSV output with csvkit to get a statistical summary about the experiments:
$ dvc exp show --csv | csvstat
...
7. "acc"
Type of data: Number
Contains null values: False
Unique values: 5
Smallest value: 0.9127
Largest value: 0.9167
Sum: 5.4895
Mean: 0.914917
Median: 0.91565
StDev: 0.001774
Most common values: 0.9127 (2x)
0.9167 (1x)
0.9153 (1x)
0.9161 (1x)
0.916 (1x)
...
Get table data in JSON
It's also possible to output the table of experiments in a machine-readable
format, for example to parse in scripts. To do so, use the --json
or --csv
options.
$ dvc exp show --json
{
"workspace": {
"baseline": {
"data": {
"timestamp": null,
"params": {
"params.yaml": {
"data": {
"train": {
"epochs": 10
},
"model": {
"conv_units": 16
}
}
}
},
"queued": false,
"running": false,
"executor": null,
"metrics": {
"metrics.json": {
"data": {
"loss": 0.236574187874794,
"acc": 0.9126999974250793
}
}
}
}
}
},
"23ceb4a6623d46a5c906f265b5846ef2f332f756": {
"baseline": {...}
"ca55d7d4763d74bfeac2ca08369489c2881c03ac": {...}
"6d13f334bbdd6ea40412d4ce9b5d81d6abb731ee": {...}
"69503c61df98752b07772415e0715655dc4ccaa0": {...}
"49779aa9c863a5503008eae08b03a6e707f8ddad": {...}
}
}
The result is a JSON dictionary with the keys workspace
and the Git commit IDs
in SHA-256 digest. These commit IDs are those that the experiments are
originated from. These commits contain a baseline
key that denotes the
baseline experiment of that particular commit and experiment IDs in SHA-256
digest. The example output above is shortened to show this structure.
Each experiment entry has the following structure:
"49779aa9c863a5503008eae08b03a6e707f8ddad": {
"data": {
"timestamp": "2021-09-09T12:53:51",
"params": {
"params.yaml": {
"data": {
"train": {
"epochs": 10
},
"model": {
"conv_units": 96
}
}
}
},
"queued": false,
"running": false,
"executor": null,
"metrics": {
"metrics.json": {
"data": {
"loss": 0.2309877723455429,
"acc": 0.916100025177002
}
}
},
"name": "cnn-64"
}
}
As an example, let's feed our experiments to JSON-parsing tool jq and filter through only metrics:
$ dvc exp show --json | jq '.[].baseline.data.metrics'
{
"metrics.json": {
"data": {
"loss": 0.236574187874794,
"acc": 0.9126999974250793
}
}
}
{
"metrics.json": {
"data": {
"loss": 0.236574187874794,
"acc": 0.9126999974250793
}
}
}
Compare specific experiments
In addition to showing a summary table of experiments, DVC provides the
dvc exp diff
command to compare pairs of experiments by the difference in
their metrics and params.
$ dvc exp diff
Path Metric HEAD workspace Change
metrics.json acc 0.9127 0.9151 0.0024
metrics.json loss 0.23657 0.23867 0.0020977
Path Param HEAD workspace Change
params.yaml model.conv_units 16 256 240
Without experiment names, dvc exp diff
shows the change between the last
experiment (workspace
) and the last commit (HEAD
). So if the command doesn't
print an output, there might be no experiment since the previous commit or it
didn't produce changes in results. If you want to see all the parameters and
metrics regardless of whether they have changed, you can use --all
flag.
$ dvc exp diff --all
Path Metric HEAD workspace Change
metrics.json acc 0.9127 0.9151 0.0024
metrics.json loss 0.23657 0.23867 0.0020977
Path Param HEAD workspace Change
params.yaml model.conv_units 16 256 240
params.yaml train.epochs 10 10 0
Compare an experiment with the workspace
When you want to compare two experiments, either the baseline experiment in a
commit, branch, or tag; or an attached experiment by name, you can supply any of
these references to dvc exp diff
.
$ dvc exp diff cnn-128 cnn-64
Path Metric cnn-128 cnn-64 Change
metrics.json acc 0.9151 0.9153 0.00020003
metrics.json loss 0.23867 0.23385 -0.0048174
Path Param cnn-128 cnn-64 Change
params.yaml model.conv_units 128 64 -64
Customize the comparison output
You can hide the file path of metrics and parameter files the --no-path
option
— produces cleaner output for single params/metrics files, for example. You can
also customize the amount of significant digits shown for numeric values with
the --precision
option (5 by default).
$ dvc exp diff puffy-daks cnn-64 --no-path --precision 2
Metric puffy-daks cnn-64 Change
acc 0.92 0.92 0.0002
loss 0.23 0.23 -0.0048
Param puffy-daks Value Change
model.conv_units 256 64 -192
Get the comparison in JSON
Parsing the output of dvc exp diff
may not be easy when you want to use it in
other commands. dvc exp diff
can output in JSON with --json
flag.
$ dvc exp diff puffy-daks cnn-64 --json
{
"metrics": {
"metrics.json": {
"loss": {
"old": 0.2386719286441803,
"new": 0.2338544875383377,
"diff": -0.00481744110584259
},
"acc": {
"old": 0.9150999784469604,
"new": 0.9153000116348267,
"diff": 0.00020003318786621094
}
}
},
"params": {
"params.yaml": {
"model.conv_units": {
"old": 256,
"new": 64,
"diff": -192
}
}
}
}
The output is a JSON dictionary with two keys, metrics
and params
, which
have dictionaries as values. metrics
and params
dictionaries have keys for
each of the metrics or parameters files, and for each file metrics and
parameters are listed as keys.
As an example, we can get only a specific metric with jq:
$ dvc exp diff puffy-daks cnn-64 --json | jq '.metrics."metrics.json".acc'
{
"old": 0.9150999784469604,
"new": 0.9153000116348267,
"diff": 0.00020003318786621094
}
Get the comparison in Markdown table format
dvc exp diff
has also an option to output a Github-Flavored Markdown
table to embed in the reports directly.
$ dvc exp diff puffy-daks cnn-64 --md
| Path | Metric | puffy-daks | cnn-64 | Change |
| ------------ | ------ | ---------- | ------- | ---------- |
| metrics.json | acc | 0.9151 | 0.9153 | 0.00020003 |
| metrics.json | loss | 0.23867 | 0.23385 | -0.0048174 |
| Path | Param | puffy-daks | cnn-64 | Change |
| ----------- | ---------------- | ---------- | ------ | ------ |
| params.yaml | model.conv_units | 256 | 64 | -192 |
You can use this output to automatically update the documents with a command like:
$ dvc exp diff puffy-daks cnn-64 --md | xargs --replace=DIFFTABLE -- sed -i -e 's/EXPERIMENT_RESULT/DIFFTABLE/' my-template.md