Data science and machine learning are iterative processes that require a large number of attempts to reach a certain level of a metric. Experimentation is part of the development of data features, hyperspace exploration, deep learning optimization, etc.
Some of DVC's base features already help you codify and analyze experiments. Parameters are simple values in a formatted text file which you can tweak and use in your code. On the other end, metrics (and plots) let you define, visualize, and compare quantitative measures of your results.
New in DVC 2.0 (see
DVC experiment management features conform a comprehensive framework to organize, execute, manage, and share ML experiments. They support these main approaches:
Compare parameters and metrics of existing project versions (for example Git
branches) against each other or against new results in the workspace. See for
dvc exp diff.
Run and capture multiple experiments (derived from any project version as baseline) without polluting your Git history. DVC tracks them for you, letting you compare and share them. 📖 More info in the Experiments Overview.
👨💻 See Get Started: Experiments for a hands-on introduction to DVC experiments.
It's up to you to decide how to organize completed experiments. These are the main alternatives:
Git tags and branches - use the repo's "time dimension" to distribute your experiments. This makes the most sense for experiments that build on each other. Git-based experiment structures are especially helpful along with Git history exploration tools like GitHub.
Directories - the project's "space dimension" can be structured with directories (folders) to organize experiments. Useful when you want to see all your experiments at the same time (without switching versions) by just exploring the file system.
Hybrid - combining an intuitive directory structure with a good repo
branching strategy tends to be the best option for complex projects.
Completely independent experiments live in separate directories (and can be
foreach stages, for example), while their progress can be
found in different branches.
Labels - in general, you can record experiments in a separate system and structure them using custom labeling. This is typical in dedicated experiment tracking tools. A possible problem with this approach is that it's easy to lose the connection between your project history and the experiments logged.
Every time you reproduce a pipeline with DVC, it
logs the unique signature of each stage run (in
.dvc/cache/runs by default).
If it never happened before, its command(s) are executed normally. Every
subsequent time a stage runs under the same
conditions, the previous results can be restored instantly, without wasting time
or computing resources.
✅ This built-in feature is called run-cache and it can dramatically improve performance. It's enabled out-of-the-box (can be disabled), which means DVC is already saving all of your tests and experiments behind the scene. But there's no easy way to explore it.