GitHub Actions are run on GitHub-hosted runners by default. However, there are many great reasons to use your own runners: to take advantage of GPUs; to orchestrate your team's shared computing resources, or to train in the cloud.
☝️ Tip! Check out the official GitHub documentation to get started setting up your self-hosted runner.
When a workflow requires computational resources (such as GPUs) CML can
automatically allocate cloud instances. For example, the following workflow
deploys a t2.micro
instance on AWS EC2 and trains a model on the instance.
After the instance is idle for 120 seconds, it automatically shuts down.
name: train-my-model
on: [push]
jobs:
deploy-cloud-runner:
runs-on: [ubuntu-latest]
container: docker://dvcorg/cml:latest
steps:
- name: deploy
env:
repo_token: ${{ secrets.REPO_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
echo "Deploying..."
MACHINE="cml$(date +%s)"
docker-machine create \
--driver amazonec2 \
--amazonec2-instance-type t2.micro \
--amazonec2-region us-east-1 \
--amazonec2-zone f \
--amazonec2-vpc-id vpc-06bc773d85a0a04f7 \
--amazonec2-ssh-user ubuntu \
$MACHINE
eval "$(docker-machine env --shell sh $MACHINE)"
(
docker-machine ssh \
$MACHINE "sudo mkdir -p /docker_machine && \
sudo chmod 777 /docker_machine" && \
docker-machine scp -r -q ~/.docker/machine \
$MACHINE:/docker_machine && \
docker run --name runner -d \
-v /docker_machine/machine:/root/.docker/machine \
-e RUNNER_IDLE_TIMEOUT=120 \
-e DOCKER_MACHINE=${MACHINE} \
-e RUNNER_LABELS=cml \
-e repo_token=$repo_token \
-e RUNNER_REPO="https://github.com/${GITHUB_REPOSITORY}" \
dvcorg/cml-py3 && \
sleep 20 && echo "Deployed $MACHINE"
) || (echo y | docker-machine rm $MACHINE && exit 1)
train:
needs: deploy-cloud-runner
runs-on: [self-hosted, cml]
steps:
- uses: actions/checkout@v2
- name: cml_run
env:
repo_token: ${{ secrets.GITHUB_TOKEN }}
run: |
pip install -r requirements.txt
python train.py
cat metrics.txt >> report.md
cml-publish confusion_matrix.png --md >> report.md
cml-send-comment report.md
Please note that for GCP's Compute Engine, deploying the cloud runner involves different steps:
deploy-gce:
runs-on: [ubuntu-latest]
container: docker://dvcorg/cml:latest
steps:
- name: deploy
shell: bash
env:
repo_token: ${{ secrets.REPO_TOKEN }}
GOOGLE_APPLICATION_CREDENTIALS_DATA:
${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}
run: |
echo "Deploying..."
echo '${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}' \
> gce-credentials.json
export GOOGLE_APPLICATION_CREDENTIALS='gce-credentials.json'
RUNNER_LABELS="gce"
RUNNER_REPO="https://github.com/${GITHUB_REPOSITORY}"
MACHINE="cml$(date +%s)"
docker-machine create --driver google \
--google-machine-type "n1-standard-4" \
--google-project "cml-project-279709" \
$MACHINE
eval "$(docker-machine env --shell sh $MACHINE)"
(
docker-machine ssh \
$MACHINE "sudo mkdir -p /docker_machine && \
sudo chmod 777 /docker_machine" && \
docker-machine scp -r -q ~/.docker/machine/ \
$MACHINE:/docker_machine && \
docker-machine scp -q gce-credentials.json \
$MACHINE:/docker_machine/gce-credentials.json && \
eval "$(docker-machine env --shell sh $MACHINE)" && \
docker run --name runner -d \
-v /docker_machine/gce-credentials.json:/gce-credentials.json \
-e GOOGLE_APPLICATION_CREDENTIALS='/gce-credentials.json' \
-v /docker_machine/machine:/root/.docker/machine \
-e DOCKER_MACHINE=$MACHINE \
-e repo_token=$repo_token \
-e RUNNER_LABELS=$RUNNER_LABELS \
-e RUNNER_REPO=$RUNNER_REPO \
-e RUNNER_IDLE_TIMEOUT=120 \
dvcorg/cml-py3 && \
sleep 20 && echo "Deployed $MACHINE"
) || (docker-machine rm -f $MACHINE && exit 1)
You will need to
create a new personal access token,
REPO_TOKEN
, with repository read/write access. REPO_TOKEN
must be added as a
secret in your project repository.
Note that you will also need to provide access credentials for your cloud
compute resources as secrets. In the above example, AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
are required to deploy EC2 instances.
In the above example, we use Docker Machine to provision instances. Please see their documentation for further details.
Note several CML-specific arguments to docker run
:
repo_token
should be set to your repository's personal access tokenRUNNER_REPO
should be set to the URL of your project repositorydvcorg/cml
, dvcorg/cml-py3
,
dvc/org/cml-gpu
, or dvcorg/cml-gpu-py3