params

Contains a command to show changes in parameters: diff.

Synopsis

usage: dvc params [-h] [-q | -v] {diff} ...

positional arguments:
  COMMAND
    diff         Show changes in params between commits in the
                 DVC repository, or between a commit and the workspace.

Description

Parameters can be any values used inside your code to influence the results (e.g. machine learning hyperparameters). DVC can track these as key/value pairs from structured YAML 1.2, JSON, TOML 1.0, or Python files (params.yaml by default). Params usually have simple names like epochs, learning-rate, batch_size, etc. Example:

epochs: 900
tuning:
  - learning-rate: 0.945
  - max_depth: 7
paths:
  - labels: 'materials/labels'
  - truth: 'materials/ground'

To start tracking parameters, list their names under the params field of dvc.yaml (manually or with the the -p/--params option of dvc stage add). For example:

stages:
  learn:
    cmd: python deep.py # reads params.yaml internally
    params:
      - epochs # specific param from params.yaml
      - tuning.learning-rate # nested param from params.yaml
      - paths # entire group from params.yaml
      - myparams.toml:
          - batch_size # param from custom file
      - config.json: # all params in this file

See more details about this syntax.

Multiple stages of a pipeline can use the same params file as dependency, but only certain values will affect each stage.

Parameters can also be used for templating dvc.yaml itself (see also Dict Unpacking), which means you can pass them to your stage commands as command-line arguments. You can also load them in Python code with dvc.api.params_show().

The dvc params diff command is available to show parameter changes, displaying their current and previous values.

DVC saves parameter names and values to dvc.lock in order to track them over time. They will be compared to the latest params files to determine if the stage is outdated upon dvc repro (or dvc status).

Options

-h, --help - prints the usage/help message, and exit.
-q, --quiet - do not write anything to standard output.
-v, --verbose - displays detailed tracing information.

Examples

First, let's create a simple parameters file in YAML format, using the default file name params.yaml:

lr: 0.0041

train:
  epochs: 70
  layers: 9

process:
  thresh: 0.98
  bow: 15000

Using dvc stage add, define a stage that depends on params lr, layers, and epochs from the params file above. Full paths should be used to specify layers and epochs from the train group:

$ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
                -p lr,train.epochs,train.layers \
                python train.py

Note that we could use the same parameter addressing with JSON, TOML, or Python parameters files.

The train.py script will have some code to parse and load the needed parameters. You can use dvc.api.params_show() for this:

import dvc.api

params = dvc.api.params_show()

lr = params['lr']
epochs = params['train']['epochs']
layers = params['train']['layers']

We use ruamel.yaml which supports YAML 1.2 (unlike the more popular PyYAML).

You can find that each parameter was defined in dvc.yaml, as well as saved to dvc.lock along with the values. These are compared to the params files when dvc repro is used, to determine if the parameter dependency has changed.

# dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
      - users.csv
    params:
      - lr
      - train.epochs
      - train.layers
    outs:
      - model.pkl

Alternatively, the entire group of parameters train can be referenced, instead of specifying each of the params separately:

$ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
                -p lr,train \
                python train.py

# in dvc.yaml
params:
  - lr
  - train

In the examples above, the default parameters file name params.yaml was used. Note that this file name can be redefined using a prefix in the -p argument of dvc stage add. In our case:

$ dvc stage add -n train -d train.py -d logs/ -o users.csv -f \
                -p parse_params.yaml:threshold,classes_num \
                python train.py

Examples: Print all parameters

Following the previous example, we can use dvc params diff to list all of the param values available in the workspace:

$ dvc params diff
Path         Param           HEAD  workspace
params.yaml  lr              —     0.0041
params.yaml  process.bow     —     15000
params.yaml  process.thresh  —     0.98
params.yaml  train.epochs    —     70
params.yaml  train.layers    —     9

This command shows the difference in parameters between the workspace and the last committed version of the params.yaml file. In our example there's no previous version, which is why all Old values are —.

Examples: Python parameters file

See Note that complex expressions (unsupported by ast.literal_eval) won't be parsed as DVC parameters.

Consider this Python parameters file named params.py:

# All standard variable types are supported.
BOOL = True
INT = 5
FLOAT = 0.001
STR = 'abc'
DICT = {'a': 1, 'b': 2}
LIST = [1, 2, 3]
SET = {4, 5, 6}
TUPLE = (10, 100)
NONE = None
UNARY_OP = -1

# Complex expressions will be ignored.
DICT_EXP = dict(a=1, b=2)

# DVC can retrieve class constants and variables defined in __init__
class TrainConfig:

    EPOCHS = 70

    def __init__(self):
        self.layers = 5
        self.layers = 9  # TrainConfig.layers param will be 9
        self.sum = 1 + 2  # Will NOT be found due to the expression
        bar = 3  # Will NOT be found since it's locally scoped


class TestConfig:

    TEST_DIR = 'path'
    METRICS = ['metric']

The following stage depends on params BOOL, INT, as well as TrainConfig's EPOCHS and layers:

$ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
                -p params.py:BOOL,INT,TrainConfig.EPOCHS,TrainConfig.layers \
                python train.py

Resulting dvc.yaml and dvc.lock files (notice the params lists):

stages:
  train:
    cmd: python train.py
    deps:
      - users.csv
    params:
      - params.py:
          - BOOL
          - INT
          - TrainConfig.EPOCHS
          - TrainConfig.layers
    outs:
      - model.pkl

schema: '2.0'
stages:
  train:
    cmd: python train.py
    deps:
      - path: users.csv
        md5: 23be4307b23dcd740763d5fc67993f11
    params:
      params.py:
        INT: 5
        BOOL: true
        TrainConfig.EPOCHS: 70
        TrainConfig.layers: 9
    outs:
      - path: model.pkl
        md5: 1c06b4756f08203cc496e4061b1e7d67

Alternatively, the entire TestConfig params group (class) can be referenced (dictionaries are also supported), instead of the parameters in it:

$ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
                -p params.py:BOOL,INT,TestConfig \
                python train.py