Skip to content

Commit ffd0647

Browse files
woopwoop
authored
Housekeeping of Feast CI, docker compose, and notebooks (#916)
* Add caching to core and serving builds * Fix flaky e2e tests * Simplify docker-compose setup * Add kafka broker to docker compose test * Rename batch to historical * Update troubleshooting.md * Fix casing for FEAST_HISTORICAL_SERVING_URL * Remove f-string prefix * Fix missing key in docker compose setup * Fix documentation typos * Add jupyter to load test * Add docker compose back to load-test * Clean up FEAST_VERSION in load test * Fix artifact output for load test * Revert e2e test order
1 parent d3295a2 commit ffd0647

File tree

25 files changed

+268
-200
lines changed

25 files changed

+268
-200
lines changed

‎.github/workflows/complete.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,23 @@ jobs:
77
runs-on: [self-hosted]
88
strategy:
99
matrix:
10-
component: [core, serving]
10+
component: [core, serving, jupyter]
1111
env:
1212
GITHUB_PR_SHA: ${{ github.event.pull_request.head.sha }}
1313
REGISTRY: gcr.io/kf-feast
14+
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2019-10-24.tar
1415
steps:
1516
- uses: actions/checkout@v2
1617
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
1718
with:
1819
version: '290.0.1'
1920
export_default_credentials: true
2021
- run: gcloud auth configure-docker --quiet
22+
- name: Get m2 cache
23+
run: |
24+
infra/scripts/download-maven-cache.sh \
25+
--archive-uri ${MAVEN_CACHE} \
26+
--output-dir .
2127
- name: Build image
2228
run: make build-${{ matrix.component }}-docker REGISTRY=${REGISTRY} VERSION=${GITHUB_SHA}
2329
- name: Push image

‎.github/workflows/master_only.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,20 @@ jobs:
1212
strategy:
1313
matrix:
1414
component: [core, serving, jupyter, ci]
15+
env:
16+
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2019-10-24.tar
1517
steps:
1618
- uses: actions/checkout@v2
1719
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
1820
with:
1921
version: '290.0.1'
2022
export_default_credentials: true
2123
- run: gcloud auth configure-docker --quiet
24+
- name: Get m2 cache
25+
run: |
26+
infra/scripts/download-maven-cache.sh \
27+
--archive-uri ${MAVEN_CACHE} \
28+
--output-dir .
2229
- name: Build image
2330
run: make build-${{ matrix.component }}-docker REGISTRY=gcr.io/kf-feast VERSION=${GITHUB_SHA}
2431
- name: Push image

‎docs/administration/troubleshooting.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ docker ps
1515
```text
1616
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1717
d7447205bced jupyter/datascience-notebook:latest "tini -g -- start-no…" 2 minutes ago Up 2 minutes 0.0.0.0:8888->8888/tcp feast_jupyter_1
18-
8e49dbe81b92 gcr.io/kf-feast/feast-serving:latest "java -Xms1024m -Xmx…" 2 minutes ago Up 5 seconds 0.0.0.0:6567->6567/tcp feast_batch-serving_1
19-
b859494bd33a gcr.io/kf-feast/feast-serving:latest "java -jar /opt/feas…" 2 minutes ago Up About a minute 0.0.0.0:6566->6566/tcp feast_online-serving_1
18+
8e49dbe81b92 gcr.io/kf-feast/feast-serving:latest "java -Xms1024m -Xmx…" 2 minutes ago Up 5 seconds 0.0.0.0:6567->6567/tcp feast_historical_serving_1
19+
b859494bd33a gcr.io/kf-feast/feast-serving:latest "java -jar /opt/feas…" 2 minutes ago Up About a minute 0.0.0.0:6566->6566/tcp feast_online_serving_1
2020
5c4962811767 gcr.io/kf-feast/feast-core:latest "java -jar /opt/feas…" 2 minutes ago Up 2 minutes 0.0.0.0:6565->6565/tcp feast_core_1
2121
1ba7239e0ae0 confluentinc/cp-kafka:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:9094->9094/tcp feast_kafka_1
2222
e2779672735c confluentinc/cp-zookeeper:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 2181/tcp, 2888/tcp, 3888/tcp feast_zookeeper_1
@@ -59,8 +59,8 @@ You will probably need to connect using the hostnames of services and standard F
5959

6060
```bash
6161
export FEAST_CORE_URL=core:6565
62-
export FEAST_ONLINE_SERVING_URL=online-serving:6566
63-
export FEAST_BATCH_SERVING_URL=batch-serving:6567
62+
export FEAST_ONLINE_SERVING_URL=online_serving:6566
63+
export FEAST_HISTORICAL_SERVING_URL=historical_serving:6567
6464
```
6565

6666
### **Docker Compose \(from outside the docker cluster\)**
@@ -70,7 +70,7 @@ You will probably need to connect using `localhost` and standard ports:
7070
```bash
7171
export FEAST_CORE_URL=localhost:6565
7272
export FEAST_ONLINE_SERVING_URL=localhost:6566
73-
export FEAST_BATCH_SERVING_URL=localhost:6567
73+
export FEAST_HISTORICAL_SERVING_URL=localhost:6567
7474
```
7575

7676
### **Google Kubernetes Engine \(GKE\)**
@@ -81,7 +81,7 @@ You will need to find the external IP of one of the nodes as well as the NodePor
8181
export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1)
8282
export FEAST_CORE_URL=${FEAST_IP}:32090
8383
export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091
84-
export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092
84+
export FEAST_HISTORICAL_SERVING_URL=${FEAST_IP}:32092
8585
```
8686

8787
`netcat`, `telnet`, or even `curl` can be used to test whether all services are available and ports are open, but `grpc_cli` is the most powerful. It can be installed from [here](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md).
@@ -107,7 +107,7 @@ ListProjects
107107
### Testing Feast Batch Serving and Online Serving
108108

109109
```bash
110-
grpc_cli ls ${FEAST_BATCH_SERVING_URL} feast.serving.ServingService
110+
grpc_cli ls ${FEAST_HISTORICAL_SERVING_URL} feast.serving.ServingService
111111
```
112112

113113
```text
@@ -145,11 +145,11 @@ In order to print the logs from these services, please run the commands below.
145145
```
146146

147147
```text
148-
docker logs -f feast_batch-serving_1
148+
docker logs -f feast_historical_serving_1
149149
```
150150

151151
```text
152-
docker logs -f feast_online-serving_1
152+
docker logs -f feast_online_serving_1
153153
```
154154

155155
### Google Kubernetes Engine

‎docs/getting-started/deploying-feast/docker-compose.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -80,12 +80,12 @@ Configure the `.env` file based on your environment. At the very least you have
8080
| Parameter | Description |
8181
| :--- | :--- |
8282
| `FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY` | This should be your service account file name, for example `key.json`. |
83-
| `FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY` | This should be your service account file name, for example `key.json` |
83+
| `FEAST_HISTORICAL_SERVING_GCP_SERVICE_ACCOUNT_KEY` | This should be your service account file name, for example `key.json` |
8484
| `FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY` | This should be your service account file name, for example `key.json` |
8585

8686
### 3.3 Configure Historical Serving
8787

88-
We will also need to configure the `batch-serving.yml` file inside `infra/docker-compose/serving/`. This configuration is used to retrieve training datasets from Feast. At a minimum you will need to set:
88+
We will also need to configure the `historical-serving.yml` file inside `infra/docker-compose/serving/`. This configuration is used to retrieve training datasets from Feast. At a minimum you will need to set:
8989

9090
| Parameter | Description |
9191
| :--- | :--- |

‎docs/installation/docker-compose.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ Configure the `.env` file based on your environment. At the very least you have
8585
| Parameter | Description |
8686
| :--- | :--- |
8787
| FEAST\_CORE\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json`. |
88-
| FEAST\_BATCH\_SERVING\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json` |
88+
| FEAST\_HISTORICAL\_SERVING\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json` |
8989
| FEAST\_JUPYTER\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json` |
9090
| FEAST\_JOB\_STAGING\_LOCATION | Google Cloud Storage bucket that Feast will use to stage data exports and batch retrieval requests, for example `gs://your-gcs-bucket/staging` |
9191

‎docs/installation/gke.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ For this guide we will use `NodePort` for exposing Feast services. In order to d
8080
export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1)
8181
export FEAST_CORE_URL=${FEAST_IP}:32090
8282
export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091
83-
export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092
83+
export FEAST_HISTORICAL_SERVING_URL=${FEAST_IP}:32092
8484
```
8585

8686
Add firewall rules to open up ports on your Google Cloud Platform project:

‎docs/user-guide/feature-retrieval.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ The computation of statistics is not enabled by default. To indicate to Feast th
116116
```python
117117
dataset = client.get_historical_features(
118118
feature_refs=features,
119-
entity_rows=entity_df
119+
entity_rows=entity_df,
120120
compute_statistics=True
121121
)
122122

‎examples/basic/basic.ipynb‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@
4343
"FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'localhost:6566')\n",
4444
"\n",
4545
"# Feast Batch Serving allows for the retrieval of historical feature data\n",
46-
"FEAST_BATCH_SERVING_URL = os.getenv('FEAST_BATCH_SERVING_URL', 'localhost:6567')"
46+
"FEAST_HISTORICAL_SERVING_URL = os.getenv('FEAST_HISTORICAL_SERVING_URL', 'localhost:6567')"
4747
]
4848
},
4949
{
@@ -466,8 +466,8 @@
466466
"source": [
467467
"online_features = client.get_online_features(\n",
468468
" feature_refs=[\n",
469-
" f\"daily_transactions\",\n",
470-
" f\"total_transactions\",\n",
469+
" \"daily_transactions\",\n",
470+
" \"total_transactions\",\n",
471471
" ],\n",
472472
" entity_rows=[\n",
473473
" {\n",
@@ -592,7 +592,7 @@
592592
"metadata": {},
593593
"outputs": [],
594594
"source": [
595-
"batch_client = Client(core_url=FEAST_CORE_URL, serving_url=FEAST_BATCH_SERVING_URL)"
595+
"batch_client = Client(core_url=FEAST_CORE_URL, serving_url=FEAST_HISTORICAL_SERVING_URL)"
596596
]
597597
},
598598
{
@@ -614,8 +614,8 @@
614614
"source": [
615615
"job = batch_client.get_historical_features(\n",
616616
" feature_refs=[\n",
617-
" f\"daily_transactions\", \n",
618-
" f\"total_transactions\", \n",
617+
" \"daily_transactions\",\n",
618+
" \"total_transactions\",\n",
619619
" ],\n",
620620
" entity_rows=entity_rows\n",
621621
" )"
@@ -754,4 +754,4 @@
754754
},
755755
"nbformat": 4,
756756
"nbformat_minor": 4
757-
}
757+
}

‎examples/feast-xgboost-churn-prediction-tutorial/Telecom Customer Churn Prediction (with Feast and XGBoost).ipynb‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6752,7 +6752,7 @@
67526752
"FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'localhost:6566')\n",
67536753
"\n",
67546754
"# Feast Batch Serving allows for the retrieval of historical feature data\n",
6755-
"FEAST_BATCH_SERVING_URL = os.getenv('FEAST_BATCH_SERVING_URL', 'localhost:6567')"
6755+
"FEAST_HISTORICAL_SERVING_URL = os.getenv('FEAST_HISTORICAL_SERVING_URL', 'localhost:6567')"
67566756
]
67576757
},
67586758
{
@@ -7187,7 +7187,7 @@
71877187
" def __init__(self, features, target, model_path=None):\n",
71887188
" # Set up Feast clients to retrieve training and online serving data\n",
71897189
" self._feast_online_client = Client(serving_url=os.environ['FEAST_ONLINE_SERVING_URL'])\n",
7190-
" self._feast_batch_client = Client(serving_url=os.environ['FEAST_BATCH_SERVING_URL'],\n",
7190+
" self._feast_batch_client = Client(serving_url=os.environ['FEAST_HISTORICAL_SERVING_URL'],\n",
71917191
" core_url=os.environ['FEAST_CORE_URL'])\n",
71927192
" \n",
71937193
" # Path to either save models after training or load models for serving\n",

‎infra/charts/feast/charts/feast-jupyter/templates/deployment.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ spec:
3838
value: "{{ .Release.Name }}-feast-core:6565"
3939
- name: FEAST_ONLINE_SERVING_URL
4040
value: "{{ .Release.Name }}-feast-online-serving:6566"
41-
- name: FEAST_BATCH_SERVING_URL
41+
- name: FEAST_HISTORICAL_SERVING_URL
4242
value: "{{ .Release.Name }}-feast-batch-serving:6566"
4343
{{- if .Values.gcpServiceAccount.enabled }}
4444
- name: GOOGLE_APPLICATION_CREDENTIALS

0 commit comments

Comments
 (0)