Edit on GitHub

exp diff

Show changes in metrics and parameters between dvc experiments.

Synopsis

usage: dvc exp diff [-h] [-q | -v] [--all] [--param-deps]
                    [--show-json] [--show-md] [--old]
                    [--no-path] [--precision <n>]
                    [a_rev] [b_rev]
positional arguments:
  a_rev            Old experiment to compare (defaults to HEAD)
  b_rev            New experiment to compare (defaults to the current
                   workspace)

Description

Provides a quick way to compare dvc params and dvc metrics between two dvc experiments by printing a table of differences. By default, it includes the params/metrics file "Path", "Param" or "Metric" name, the new "Value", and the difference ("Change") for numeric values. Example:

$ dvc exp diff
Path         Metric    Value    Change
scores.json  roc_auc   0.93345  0.018087
scores.json  grade     B-       โ€”

Path         Param         Value    Change
params.yaml  max_features  3000     1500

This is similar to combining the reports from dvc params diff and dvc metrics diff together, for the experiments in question.

Without arguments, this command compares all the params and metrics referenced in dvc.yaml files present in the workspace with the latest committed versions (required). Only params/metrics that changed are listed, by default (show everything with --all).

a_rev and b_rev are optional experiments to compare. They accept experiment names or hashes (see dvc exp run for details). Providing a single one results in comparing it to the workspace.

Another way to display experiments is the dvc exp show command, which lists all the current experiments (without comparisons).

Options

  • --all - list all parameters and metrics, including those without changes.
  • --param-deps - include only parameters that are stage dependencies.
  • --show-json - prints the command's output in easily parsable JSON format, instead of a human-readable table.
  • --show-md - prints the command's output in the Markdown table format (GFM).
  • --old - include the "Old" value column in addition to the new "Value" (and "Change"). Useful when the values are not numeric, for example
  • --no-path - hide the "Path" column that lists the param/metrics file location. Useful when only one metrics or params file exists, for example
  • --precision <n> - round decimal values to n digits of precision (5 by default). Applies to metrics only.
  • -h, --help - prints the usage/help message, and exit.
  • -q, --quiet - do not write anything to standard output. Exit with 0 if no problems arise, otherwise 1.
  • -v, --verbose - displays detailed tracing information.

Examples

This example is based on our Get Started, where you can find the actual source code.

Let's say we have run 3 experiments in our project:

$ dvc exp show --include-params=featurize
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Experiment            โ”ƒ Created      โ”ƒ     auc โ”ƒ featurize.max_features โ”ƒ featurize.ngrams โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ workspace             โ”‚ -            โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ 10-bigrams-experiment โ”‚ Jun 20, 2020 โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ โ”œโ”€โ”€ exp-e6c97         โ”‚ Oct 21, 2020 โ”‚ 0.61314 โ”‚ 1500                   โ”‚ 2                โ”‚
โ”‚ โ”œโ”€โ”€ exp-1dad0         โ”‚ Oct 09, 2020 โ”‚ 0.57756 โ”‚ 2000                   โ”‚ 2                โ”‚
โ”‚ โ””โ”€โ”€ exp-1df77         โ”‚ Oct 09, 2020 โ”‚ 0.51676 โ”‚ 500                    โ”‚ 2                โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Since we haven't made any changes to the workspace, we can compare exp-1dad0 to its baseline (10-bigrams-experiment, current HEAD) like this:

$ dvc exp diff exp-1dad0
Path         Metric    Value    Change
scores.json  auc       0.61314  0.035575
Path         Param                   Value    Change
params.yaml  featurize.max_features  1500     -500

To compare two specific experiments (values are shown for the second one by default):

$ dvc exp diff exp-1dad0 exp-1df77
Path         Metric    Value    Change
scores.json  auc       0.51676  -0.060799
Path         Param                   Value    Change
params.yaml  featurize.max_features  500      -1500

To compare an experiment to the 7-ml-pipeline tag (or any other revision):

$ dvc exp diff exp-1dad0 7-ml-pipeline
Path         Metric    Value    Change
scores.json  auc       None     diff not supported
Path         Param                   Value    Change
params.yaml  featurize.max_features  500      -1500
params.yaml  featurize.ngrams        1        -1
Content

๐Ÿ› Found an issue? Let us know! Or fix it:

Edit on GitHub

โ“ Have a question? Join our chat, we will help you:

Discord Chat