Skip to content

Commit 8174d98

Browse files
algattiksudivate
algattik
and
sudivate
authored
Manage AML environments from conda YAML files (microsoft#168)
 * PR review fixes * Update Dockerfile * . * Update deploy_web_service.py * Update deploy_web_service.py * PR review fixes * PR review fixes * PR review fixes * Update training_dependencies.yml * Linting fixes * Fixed merge * Update code_test.py * Simplified environment management, restored deploy task * Simplified environment management, restored deploy task * Fixed doc for new file path * Added comments Co-authored-by: Sushant Divate <[email protected]>
1 parent 9e3b007 commit 8174d98

11 files changed

Lines changed: 110 additions & 81 deletions

‎diabetes_regression/azureml_environment.json‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{
2+
"name": "diabetes_regression_sklearn",
3+
"version": null,
4+
"environmentVariables": {
5+
"EXAMPLE_ENV_VAR": "EXAMPLE_VALUE"
6+
},
7+
"python": {
8+
"userManagedDependencies": false,
9+
"interpreterPath": "python",
10+
"condaDependenciesFile": null,
11+
"baseCondaEnvironment": null
12+
},
13+
"docker": {
14+
"enabled": true,
15+
"baseImage": "mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04",
16+
"baseDockerfile": null,
17+
"sharedVolumes": true,
18+
"shmSize": "2g",
19+
"arguments": [],
20+
"baseImageRegistry": {
21+
"address": null,
22+
"username": null,
23+
"password": null
24+
}
25+
},
26+
"spark": {
27+
"repositories": [],
28+
"packages": [],
29+
"precachePackages": true
30+
},
31+
"databricks": {
32+
"mavenLibraries": [],
33+
"pypiLibraries": [],
34+
"rcranLibraries": [],
35+
"jarLibraries": [],
36+
"eggLibraries": []
37+
},
38+
"inferencingStackVersion": null
39+
}

‎diabetes_regression/ci_dependencies.yml‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,19 @@ dependencies:
55
# The python interpreter version.
66
- python=3.7.5
77

8-
- r=3.6.0
9-
- r-essentials=3.6.0
8+
# dependencies with versions aligned with conda_dependencies.yml.
109
- numpy=1.18.1
1110
- pandas=1.0.0
1211
- scikit-learn=0.22.1
1312

13+
# dependencies for MLOps with R.
14+
- r=3.6.0
15+
- r-essentials=3.6.0
16+
1417
- pip=20.0.2
1518
- pip:
1619

17-
# dependencies shared with other environment .yml files.
20+
# dependencies with versions aligned with conda_dependencies.yml.
1821
- azureml-sdk==1.0.79
1922

2023
# Additional pip dependencies for the CI environment.

‎diabetes_regression/conda_dependencies.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Details about the Conda environment file format:
2+
# https://conda.io/docs/using/envs.html#create-environment-file-by-hand
3+
4+
name: diabetes_regression_sklearn
5+
6+
dependencies:
7+
8+
# The python interpreter version.
9+
- python=3.7.5
10+
11+
# Required by azureml-defaults, installed separately through Conda to
12+
# get a prebuilt version and not require build tools for the install.
13+
- psutil=5.6 #latest
14+
15+
- numpy=1.18.1
16+
- pandas=1.0.0
17+
- scikit-learn=0.22.1
18+
#- r-essentials
19+
#- tensorflow
20+
#- keras
21+
22+
- pip=20.0.2
23+
- pip:
24+
25+
# Dependencies for training environment.
26+
27+
- azureml-core==1.0.79
28+
29+
# Dependencies for scoring environment.
30+
31+
# You must list azureml-defaults as a pip dependency
32+
- azureml-defaults==1.0.79
33+
- inference-schema[numpy-support]==1.0.1

‎diabetes_regression/scoring/deployment_config_aci.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
---
1+
computeType: ACI
22
containerResourceRequirements:
33
cpu: 1
44
memoryInGB: 4
5-
computeType: ACI

‎diabetes_regression/scoring/inference_config.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
entryScript: score.py
22
runtime: python
3-
condaFile: ../scoring_dependencies.yml
3+
condaFile: ../conda_dependencies.yml
44
extraDockerfileSteps:
55
schemaFile:
66
sourceDirectory:

‎diabetes_regression/scoring_dependencies.yml‎

Lines changed: 0 additions & 36 deletions
This file was deleted.

‎diabetes_regression/training_dependencies.yml‎

Lines changed: 0 additions & 17 deletions
This file was deleted.

‎docs/code_description.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@
2727

2828
### Environment Definitions
2929

30-
- `diabetes_regression/training_dependencies.yml` : Conda environment definition for the training environment (Docker image in which train.py is run).
31-
- `diabetes_regression/scoring_dependencies.yml` : Conda environment definition for the scoring environment (Docker image in which score.py is run).
30+
- `diabetes_regression/azureml_environment.json` : Azure ML environment definition for the training environment, including base Docker image and a reference to `conda_dependencies.yml` Conda environment file.
31+
- `diabetes_regression/conda_dependencies.yml` : Conda environment definition for the environment used for both training and scoring (Docker image in which train.py and score.py are run).
3232
- `diabetes_regression/ci_dependencies.yml` : Conda environment definition for the CI environment.
3333

3434
### Code

‎docs/getting_started.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -175,9 +175,9 @@ Great, you now have the build pipeline set up which automatically triggers every
175175
pipelines using R to train a model. This is enabled
176176
by changing the `build-train-script` pipeline variable to either of:
177177
* `diabetes_regression_build_train_pipeline_with_r.py` to train a model
178-
with R on Azure ML Compute. You will also need to add the
179-
`r-essentials` Conda packages into `diabetes_regression/scoring_dependencies.yml`
180-
and `diabetes_regression/training_dependencies.yml`.
178+
with R on Azure ML Compute. You will also need to uncomment (i.e. include) the
179+
`r-essentials` Conda packages in the environment definition
180+
`diabetes_regression/conda_dependencies.yml`.
181181
* `diabetes_regression_build_train_pipeline_with_r_on_dbricks.py`
182182
to train a model with R on Databricks. You will need
183183
to manually create a Databricks cluster and attach it to the ML Workspace as a

‎ml_service/pipelines/diabetes_regression_build_train_pipeline.py‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
from azureml.pipeline.core.graph import PipelineParameter
22
from azureml.pipeline.steps import PythonScriptStep
33
from azureml.pipeline.core import Pipeline
4-
from azureml.core import Workspace
5-
from azureml.core.runconfig import RunConfiguration, CondaDependencies
4+
from azureml.core import Workspace, Environment
5+
from azureml.core.runconfig import RunConfiguration
66
from azureml.core import Dataset, Datastore
77
from ml_service.util.attach_compute import get_compute
88
from ml_service.util.env_variables import Env
@@ -28,17 +28,17 @@ def main():
2828
print("aml_compute:")
2929
print(aml_compute)
3030

31-
# Create a run configuration environment
32-
conda_deps_file = "diabetes_regression/training_dependencies.yml"
33-
conda_deps = CondaDependencies(conda_deps_file)
34-
run_config = RunConfiguration(conda_dependencies=conda_deps)
35-
run_config.environment.docker.enabled = True
36-
config_envvar = {}
31+
# Create a reusable run configuration environment
32+
# Read definition from diabetes_regression/azureml_environment.json
33+
environment = Environment.load_from_directory(e.sources_directory_train)
3734
if (e.collection_uri is not None and e.teamproject_name is not None):
3835
builduri_base = e.collection_uri + e.teamproject_name
3936
builduri_base = builduri_base + "/_build/results?buildId="
40-
config_envvar["BUILDURI_BASE"] = builduri_base
41-
run_config.environment.environment_variables = config_envvar
37+
environment.environment_variables["BUILDURI_BASE"] = builduri_base
38+
environment.register(aml_workspace)
39+
40+
run_config = RunConfiguration()
41+
run_config.environment = environment
4242

4343
model_name_param = PipelineParameter(
4444
name="model_name", default_value=e.model_name)

0 commit comments

Comments
 (0)