repro

Reproduce complete or partial pipelines by running their stages as needed in the correct order.

Synopsis

usage: dvc repro [-h] [-q | -v] [-f] [-i]
                 [-s] [-p] [-P] [-R] [-m]
                 [--downstream] [--force-downstream]
                 [--pull] [--dry]
                 [--glob] [--no-commit] [--no-run-cache]
                 [targets [<target> ...]]

positional arguments:
  targets       Stages to reproduce. 'dvc.yaml' by default.

See targets for more details.

Description

Provides a way to regenerate data pipeline results, by restoring the dependency graph implicitly defined by the stages listed in dvc.yaml. The commands defined in these stages are then executed in the correct order.

For stages with multiple commands (having a list in the cmd field), commands are run one after the other in the order they are defined. The failure of any command will halt the remaining stage execution, and raises an error.

Pipeline stages are defined in dvc.yaml (either manually or by using dvc stage add) while initial data dependencies can be registered with dvc add.

dvc repro is similar to Make in software build automation, but DVC captures build requirements (dependencies and outputs) and caches the pipeline's outputs along the way.

💡 For convenience, a Git hook is available to remind you to dvc repro when needed after a git commit. See dvc install for more details.

Keep in mind that one dvc.yaml file does not necessarily equal one pipeline (although that is typical). DVC evaluates all the dvc.yaml files in the workspace to rebuild and validate pipeline(s). Then it executes the corresponding commands (cmd field of dvc.yaml).

There are a few ways to restrict what will be regenerated by this command: by specifying specific reproduction targets, or by using certain command options, such as --single-item or --all-pipelines.

Note that stages without dependencies nor outputs are considered always changed, so dvc repro always executes them.

Stage outputs are deleted from the workspace before executing the stage commands that produce them (unless persist: true is used in dvc.yaml).

dvc repro does not run dvc fetch, dvc pull or dvc checkout to get data files, intermediate or final results (except if the --pull option is used).

It stores all the data files, intermediate or final results in the cache (unless the --no-commit option is used), and updates the hash values of changed dependencies and outputs in the dvc.lock and .dvc files.

Parallel stage execution

Currently, dvc repro is not able to parallelize stage execution automatically. If you need to do this, you can launch dvc repro multiple times manually. For example, let's say a pipelines graph looks something like this:

$ dvc dag
+--------+          +--------+
|   A1   |          |   B1   |
+--------+          +--------+
     *                   *
     *                   *
     *                   *
+--------+          +--------+
|   A2   |          |   B2   |
+--------+          +--------+
          *         *
           **     **
             *   *
        +------------+
        |    train   |
        +------------+

This pipeline consists of two parallel branches (A and B), and the final train stage, where the branches merge. If you run dvc repro at this point, it would reproduce each branch sequentially before train. To reproduce both branches simultaneously, you could run dvc repro A2 and dvc repro B2 at the same time (e.g. in separate terminals). After both finish successfully, you can then run dvc repro train: DVC will know that both branches are already up-to-date and only execute the final stage.

Options

• targets (optional command argument) - what to reproduce (all pipeline(s) in ./dvc.yaml by default). Different things can be provided as targets depending on the flags used (more details in each option). Examples:

  • dvc repro linear/dvc.yaml: A dvc.yaml file
  • dvc repro -R pipelines/: Directory to explore recursively for pipelines
  • dvc repro train-model: Specific stage(s) from ./dvc.yaml
  • dvc repro modeling/dvc.yaml:prepare: Stage(s) from a specific dvc.yaml file
  • dvc repro train-model@1: Foreach stage(s) from ./dvc.yaml
  • dvc repro --glob train-*: Pattern to match groups of stages

• -R, --recursive - looks for dvc.yaml files to reproduce in any directories given as targets, and in their subdirectories. If there are no directories among the targets, this option has no effect.

• --glob - causes the targets to be interpreted as wildcard patterns to match for stage names. For example: train-* (certain stage names) or models/dvc.yaml:train-* (stages in specific dvc.yaml file). Note that it does not match patterns with the path, only to the stages present in the specified file.

• -s, --single-item - reproduce only a single stage by turning off the recursive search for changed dependencies. Multiple stages are executed (non-recursively) if multiple stage names are given as targets.

• -f, --force - reproduce pipelines, regenerating its results, even if no changes were found. This executes all of the stages by default, but it can be limited with the targets argument, or the -s, -p options.

• --no-commit - do not store the outputs of this execution in the cache (dvc.yaml and dvc.lock are still created or updated); useful to avoid caching unnecessary data when exploring different data or stages. Use dvc commit to finish the operation.

• -m, --metrics - show metrics after reproduction. The target pipelines must have at least one metrics file defined in dvc.yaml.

• --dry - only print the commands that would be executed without actually executing the commands.

• -i, --interactive - ask for confirmation before reproducing each stage. The stage is only executed if the user types "y".

• -p, --pipeline - reproduce the entire pipelines that the targets belong to. Use dvc dag <target> to show the parent pipeline of a target.

• -P, --all-pipelines - reproduce all pipelines for all dvc.yaml files present in the DVC project. Specifying targets has no effects with this option, as all possible targets are already included.

• --no-run-cache - execute stage command(s) even if they have already been run with the same dependencies and outputs (see the details). Useful for example if the stage command/s is/are non-deterministic (not recommended).

• --force-downstream - in cases like ... -> A (changed) -> B -> C it will reproduce A first and then B, even if B was previously executed with the same inputs from A (cached). To be precise, it reproduces all descendants of a changed stage or the stages following the changed stage, even if their direct dependencies did not change.

  It can be useful when we have a common dependency among all stages, and want to specify it only once (for stage A here). For example, if we know that all stages (A and below) depend on requirements.txt, we can specify it in A, and omit it in B and C.

  This is a way to force-execute stages without changes. This can also be useful for pipelines containing stages that produce non-deterministic (semi-random) outputs, where outputs can vary on each execution, meaning the cache cannot be trusted for such stages.

• --downstream - only execute the stages after the given targets in their corresponding pipelines, including the target stages themselves. This option has no effect if targets are not provided.

• --pull - attempts to download outputs of stages found in the run-cache during reproduction. Uses the default remote storage. See also dvc pull

• -h, --help - prints the usage/help message, and exit.

• -q, --quiet - do not write anything to standard output. Exit with 0 if all stages are up to date or if all stages are successfully executed, otherwise exit with 1. The command defined in the stage is free to write output regardless of this flag.

• -v, --verbose - displays detailed tracing information.

Examples

To get hands-on experience with data science and machine learning pipelines, see Get Started: Data Pipelines.

Let's build and reproduce a simple pipeline. It takes this text.txt file:

dvc
1231
is
3
the
best

And runs a few simple transformations to filter and count numbers:

$ dvc stage add -n filter -d text.txt -o numbers.txt \
           "cat text.txt | egrep '[0-9]+' > numbers.txt"

$ dvc stage add -n count -d numbers.txt -d process.py -M count.txt \
           "python process.py numbers.txt > count.txt"

Where process.py is a script that, for simplicity, just prints the number of lines:

import sys
num_lines = 0
with open(sys.argv[1], 'r') as f:
    for line in f:
        num_lines += 1
print(num_lines)

The result of executing dvc repro should look like this (cat shows the contents of a file and tree shows the contents of the working directory):

$ dvc repro
Running stage 'filter':
> cat text.txt | egrep '[0-9]+' > numbers.txt
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'

Running stage 'count':
> python process.py numbers.txt > count.txt
Updating lock file 'dvc.lock'
Use `dvc push` to send your updates to remote storage.

$ cat count.txt
2

$ tree
.
├── count.txt      <---- result: "2"
├── dvc.lock       <---- file to record pipeline state
├── dvc.yaml       <---- file containing list of stages.
├── numbers.txt    <---- intermediate result of the first stage
├── process.py     <---- code that implements data transformation
└── text.txt       <---- text file to process

You may want to check the contents of dvc.lock and count.txt for later reference.

Now, let's imagine we want to print a description and we add this line to the process.py:

...
print('Number of lines:')
print(num_lines)

If we now run dvc repro, we should see this:

$ dvc repro
Stage 'filter' didn't change, skipping
Running stage 'count' with command:
        python process.py numbers.txt > count.txt
Updating lock file 'dvc.lock'

You can now check that dvc.lock and count.txt have been updated with the new information: updated dependency/output file hash values, and a new result, respectively.

Example: Downstream from a target stage

This example continues the previous one.

The --downstream option, when used with a target stage, allows us to only reproduce results from commands after that specific stage in a pipeline. To demonstrate how it works, let's make a change in text.txt (the input of our first stage, created in the previous example):

...
The answer to universe is 42
- The Hitchhiker's Guide to the  Galaxy

Let's say we also want to print the file name in the description, and so we update the process.py as:

print(f'Number of lines in {sys.argv[1]}:')
print(num_lines)

Now, using the --downstream option with dvc repro results in the execution of only the target (count) and following stages (none in this case):

$ dvc repro --downstream count
Running stage 'count' with command:
        python process.py numbers.txt > count.txt
Updating lock file 'dvc.lock'

The change in text.txt is ignored because that file is a dependency in the filter stage, which wasn't executed by the dvc repro above. This is because filter happens before the target (count) in the pipeline (see dvc dag), as shown below:

$ dvc dag

  +--------+
  | filter |
  +--------+
      *
      *
      *
  +-------+
  | count |
  +-------+

Note that using dvc repro without --downstream in the above example results in the execution of the target (count), and the preceding stages (only 'filter' in this case).