Generate live (real-time) metrics and plots for remote experiments
In your model training script, you can use DVCLive to send live updates to metrics and plots to Iterative Studio, without writing them to your Git repository. This will enable you to view all intermediate results in Iterative Studio while your experiment is still running.
This requires a 3-step process:
Set up an access token
Iterative Studio uses access tokens to authorize DVCLive to send live updates
to the metrics and plots. To set up the access token, open your user profile
page. In the Studio access token
section, click on Generate new token
. You
can also regenerate or delete your access token.
The option to delete the access token is also available when you try to change your password. That is, you can reset all your access credentials (your password and the access token) at once. This is handy if you suspect that your account security may have been compromised.
Configure your model training job
You should provide the following environment variables to your model training job:
-
STUDIO_TOKEN
: The access token must be present in any request that sends data to the Iterative Studio ingestion endpoint. Requests with missing or incorrect access tokens are rejected with an appropriate HTTP error code and error message.If you are running the experiment locally, you can set this environment variable when submitting the training job.
$ STUDIO_TOKEN=**** dvc exp run
If you are running the experiment as part of a CI job, a secure way to provide the access token is to create a GitHub secret containing the value of the token, and use the secret in your CI job (see example below).
... steps: - name: Train model env: STUDIO_TOKEN: ${{ secrets.STUDIO_TOKEN }} ...
-
STUDIO_REPO_URL
: If you are running the experiment locally, or your repository is on github.com, gitlab.com or bitbucket.org, you do not need to set this environment variable. But if you are using some other Git provider, then you should set the repository url in this format:{remote-type}:{namespace}/{repo-name}
. For example, for theexample-get-started
repository in theiterative
namespace,STUDIO_REPO_URL
should be set to the following value:- If you are using a custom (self-hosted) GitLab server, set it to
custom-gitlab:iterative/example-get-started
. - If you are using a GitHub enterprise server, set it to
github:iterative/example-get-started
.
- If you are using a custom (self-hosted) GitLab server, set it to
Send and view live metrics and plots
In the training job (which has been configured as detailed above), whenever you log your metrics or plots using DVCLive, they will be automatically sent to Iterative Studio. Here is an example of how you can use DVCLive in your training code:
from dvclive import Live
with Live() as live:
for i in range(params["epochs"]):
...
live.log_metric("accuracy", accuracy)
live.next_step()
...
DVCLive signals the end of the experiment using live.end()
. Using
with Live() as live:
or one of the DVCLive integrations for
ML Frameworks ensures that
live.end()
is automatically called when the experiment concludes.
Iterative Studio stores the live metrics and plots data in its database.
In the project table, the live metrics are displayed nested under the parent Git commit. Updates to the live metrics are highlighted in orange.
Updates to the live metrics are highlighted in orange in the compare pane as well.
The number of experiments with new updates to the live metrics values are
displayed in the Live
icon, which can also be used to filter and show only
running experiments in the table.
Live plots are displayed in the plots pane; you can see the plots getting populated as Studio receives new updates.
An experiment can have one of the following statuses:
- Running - Iterative Studio expects to receive live metrics and plots for these experiments. Note that if the experiment stops due to any error, Iterative Studio will not be aware of this and it will continue to wait for live updates. In this case, you should delete the live metrics row from the project table in Iterative Studio.
- Completed - Iterative Studio does not expect to receive any more updates for these experiments. Once the experiment concludes, you can delete the live metrics row from the project table. Iterative Studio does not automatically commit and push the final results of your experiment to Git. If you want to save the experiment result, you should persist it using appropriate Git commands.
If there are multiple projects connected to a single Git repository, then live metrics and plots for this repository are displayed in all its connected projects.