Skip to content

Commit 6e40e27

Browse files
khorshuhengkhorshuheng
authored
ability to whitelist on feature table level (#101)
Signed-off-by: Khor Shu Heng <[email protected]> linting Signed-off-by: Khor Shu Heng <[email protected]> fix parameter Signed-off-by: Khor Shu Heng <[email protected]> fix linting Signed-off-by: Khor Shu Heng <[email protected]> add options to mount config maps add options to mount config maps Signed-off-by: Khor Shu Heng <[email protected]>
1 parent 7825d58 commit 6e40e27

11 files changed

Lines changed: 127 additions & 8 deletions

File tree

‎infra/charts/feast-spark/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
apiVersion: v1
22
description: Feast Extension for running Ingestion on Spark
33
name: feast-spark
4-
version: 0.2.15
4+
version: 0.2.16

‎infra/charts/feast-spark/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# feast-spark
22

3-
Feast Extension for running Ingestion on Spark 0.2.15
3+
Feast Extension for running Ingestion on Spark 0.2.16
44

55
## Installation
66

@@ -10,7 +10,7 @@ https://docs.feast.dev/v/master/getting-started/deploying-feast/kubernetes
1010

1111
| Repository | Name | Version |
1212
|------------|------|---------|
13-
| | feast-jobservice | 0.2.15 |
13+
| | feast-jobservice | 0.2.16 |
1414
| | prometheus-statsd-exporter | 0.1.2 |
1515

1616
## Values

‎infra/charts/feast-spark/charts/feast-jobservice/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
apiVersion: v1
22
description: Feast Job Service manage ingestion jobs.
33
name: feast-jobservice
4-
version: 0.2.15
4+
version: 0.2.16

‎infra/charts/feast-spark/charts/feast-jobservice/README.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# feast-jobservice
22

3-
![Version: 0.2.15](https://img.shields.io/badge/Version-0.2.15-informational?style=flat-square)
3+
![Version: 0.2.16](https://img.shields.io/badge/Version-0.2.16-informational?style=flat-square)
44

55
Feast Job Service manage ingestion jobs.
66

77
## Values
88

99
| Key | Type | Default | Description |
1010
|-----|------|---------|-------------|
11+
| configMaps | list | `[]` | Arbitrary config maps to be mounted on the job service pod, on /etc/configs/<config name> |
1112
| envOverrides | object | `{}` | Extra environment variables to set |
1213
| gcpProjectId | string | `""` | Project ID to use when using Google Cloud services such as BigQuery, Cloud Storage and Dataflow |
1314
| gcpServiceAccount.enabled | bool | `false` | Flag to use [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) JSON key |
@@ -58,8 +59,14 @@ Feast Job Service manage ingestion jobs.
5859
| service.http.port | int | `80` | Service port for HTTP requests |
5960
| service.http.targetPort | int | `8080` | Container port serving HTTP requests and Prometheus metrics |
6061
| service.type | string | `"ClusterIP"` | Kubernetes service type |
62+
| sparkOperator.batchJobTemplate | object | `{}` | |
6163
| sparkOperator.enabled | bool | `false` | Flag to create and mount custom job template on the jobservice deployment as configmap |
64+
| sparkOperator.historicalJobTemplate | object | `{}` | |
6265
| sparkOperator.jobTemplate | object | `{}` | Content of the job template, in yaml format |
66+
| sparkOperator.streamJobTemplate | object | `{}` | |
67+
| whitelist | object | `{"enabled":false,"featureTables":[]}` | If enabled, only <project>:<feature table> in the whitelist can be ingested |
68+
| whitelist.enabled | bool | `false` | Flag to create and mount whitelist as configmap |
69+
| whitelist.featureTables | list | `[]` | Whitelisted feature tables, in the form of <project>:<feature table> |
6370

6471
----------------------------------------------
6572
Autogenerated from chart metadata using [helm-docs v1.5.0](https://github.com/norwoodj/helm-docs/releases/v1.5.0)

‎infra/charts/feast-spark/charts/feast-jobservice/templates/deployment.yaml‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,37 +40,56 @@ spec:
4040
{{- toYaml . | nindent 8 }}
4141
{{- end }}
4242

43-
{{- if or .Values.secrets .Values.sparkOperator.enabled }}
43+
{{- if or .Values.secrets .Values.sparkOperator.enabled .Values.configMaps .Values.whitelist.enabled }}
4444
volumes:
4545
{{- end }}
4646
{{- range $secret := .Values.secrets }}
4747
- name: {{ $secret }}
4848
secret:
4949
secretName: {{ $secret }}
5050
{{- end }}
51+
{{- range $configMap := .Values.configMaps }}
52+
- name: {{ $configMap }}
53+
configMap:
54+
name: {{ $configMap }}
55+
{{- end }}
5156
{{- if .Values.sparkOperator.enabled }}
5257
- name: {{ template "feast-jobservice.fullname" . }}-spark-template
5358
configMap:
5459
name: {{ template "feast-jobservice.fullname" . }}-spark-template
5560
{{- end }}
61+
{{- if .Values.whitelist.enabled }}
62+
- name: {{ template "feast-jobservice.fullname" . }}-whitelist
63+
configMap:
64+
name: {{ template "feast-jobservice.fullname" . }}-whitelist
65+
{{- end }}
5666

5767
containers:
5868
- name: {{ .Chart.Name }}
5969
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
6070
imagePullPolicy: {{ .Values.image.pullPolicy }}
6171

62-
{{- if or .Values.secrets .Values.sparkOperator.enabled }}
72+
{{- if or .Values.secrets .Values.sparkOperator.enabled .Values.configMaps .Values.whitelist.enabled }}
6373
volumeMounts:
6474
{{- end }}
6575
{{- range $secret := .Values.secrets }}
6676
- name: {{ $secret }}
6777
mountPath: "/etc/secrets/{{ $secret }}"
6878
readOnly: true
6979
{{- end }}
80+
{{- range $configMap := .Values.configMaps }}
81+
- name: {{ $configMap }}
82+
mountPath: "/etc/configs/{{ $configMap }}"
83+
readOnly: true
84+
{{- end }}
7085
{{- if .Values.sparkOperator.enabled }}
7186
- name: {{ template "feast-jobservice.fullname" . }}-spark-template
7287
mountPath: "/etc/configs"
7388
{{- end }}
89+
{{- if .Values.whitelist.enabled }}
90+
- name: {{ template "feast-jobservice.fullname" . }}-whitelist
91+
mountPath: "/etc/whitelist"
92+
{{- end }}
7493

7594
env:
7695
{{- if .Values.sparkOperator.enabled }}
@@ -83,6 +102,10 @@ spec:
83102
- name: FEAST_SPARK_K8S_HISTORICAL_RETRIEVAL_TEMPLATE_PATH
84103
value: /etc/configs/historicalJobTemplate.yaml
85104
{{- end }}
105+
{{- if .Values.whitelist.enabled }}
106+
- name: FEAST_WHITELISTED_FEATURE_TABLES_PATH
107+
value: /etc/whitelist/whitelist.txt
108+
{{- end }}
86109
{{- range $key, $value := .Values.envOverrides }}
87110
- name: {{ printf "%s" $key | replace "." "_" | upper | quote }}
88111
{{- if eq (kindOf $value) "map" }}

‎infra/charts/feast-spark/charts/feast-jobservice/templates/whitelist.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{{- if .Values.whitelist.enabled }}
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{ template "feast-jobservice.fullname" . }}-whitelist
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
app: {{ template "feast-jobservice.name" . }}
9+
component: jobservice
10+
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
11+
release: {{ .Release.Name }}
12+
heritage: {{ .Release.Service }}
13+
data:
14+
whitelist.txt: |
15+
{{- range $featureTable := .Values.whitelist.featureTables }}
16+
{{ $featureTable }}
17+
{{- end }}
18+
{{- end }}

‎infra/charts/feast-spark/charts/feast-jobservice/values.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,13 @@ podLabels: {}
137137

138138
# secrets -- Arbitrary secrets to mount on the job service pod, on /etc/secrets/<secret name>
139139
secrets: []
140+
141+
# configMaps -- Arbitrary config maps to be mounted on the job service pod, on /etc/configs/<config name>
142+
configMaps: []
143+
144+
# whitelist -- If enabled, only <project>:<feature table> in the whitelist can be ingested
145+
whitelist:
146+
# whitelist.enabled -- Flag to create and mount whitelist as configmap
147+
enabled: false
148+
# whitelist.featureTables -- Whitelisted feature tables, in the form of <project>:<feature table>
149+
featureTables: []

‎infra/charts/feast-spark/requirements.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
dependencies:
22
- name: feast-jobservice
3-
version: 0.2.15
3+
version: 0.2.16
44
condition: feast-jobservice.enabled
55
- name: prometheus-statsd-exporter
66
version: 0.1.2

‎python/feast_spark/constants.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,10 @@ class ConfigOptions(metaclass=ConfigMeta):
169169
#: Whitelisted Feast projects
170170
WHITELISTED_PROJECTS: Optional[str] = None
171171

172+
#: File path to a whitelist containing all the feature tables allowed for ingestion.
173+
#: Each line in the file should be in the format of <project>:<feature table>
174+
WHITELISTED_FEATURE_TABLES_PATH: Optional[str] = None
175+
172176
#: If set - streaming ingestion job will be consuming incoming rows not continuously,
173177
#: but periodically with configured interval (in seconds).
174178
#: That may help to control amount of write requests to storage

‎python/feast_spark/job_service.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ def _job_to_proto(spark_job: SparkJob) -> JobProto:
104104
class JobServiceServicer(JobService_pb2_grpc.JobServiceServicer):
105105
def __init__(self, client: Client):
106106
self.client = client
107+
self._whitelisted_project_feature_table_pairs_cached: List[Tuple[str, str]] = []
107108

108109
@property
109110
def _whitelisted_projects(self) -> Optional[List[str]]:
@@ -112,12 +113,40 @@ def _whitelisted_projects(self) -> Optional[List[str]]:
112113
return whitelisted_projects.split(",")
113114
return None
114115

116+
@property
117+
def _whitelisted_project_feature_table_pairs(
118+
self,
119+
) -> Optional[List[Tuple[str, str]]]:
120+
if self._whitelisted_project_feature_table_pairs_cached:
121+
return self._whitelisted_project_feature_table_pairs_cached
122+
123+
if self.client.config.exists(opt.WHITELISTED_FEATURE_TABLES_PATH):
124+
_whitelisted_feature_tables = self.client.config.get(
125+
opt.WHITELISTED_FEATURE_TABLES_PATH
126+
)
127+
with open(str(_whitelisted_feature_tables), "r") as whitelist:
128+
whitelist.seek(0)
129+
whitelisted_feature_tables = [
130+
(line.strip().split(":")[0], line.strip().split(":")[-1])
131+
for line in whitelist.readlines()
132+
]
133+
self._whitelisted_project_feature_table_pairs_cached = (
134+
whitelisted_feature_tables
135+
)
136+
return whitelisted_feature_tables
137+
return None
138+
115139
def is_whitelisted(self, project: str):
116140
# Whitelisted projects not specified, allow all projects
117141
if not self._whitelisted_projects:
118142
return True
119143
return project in self._whitelisted_projects
120144

145+
def is_feature_table_whitelisted(self, project: str, feature_table: str):
146+
if not self._whitelisted_project_feature_table_pairs:
147+
return True
148+
return (project, feature_table) in self._whitelisted_project_feature_table_pairs
149+
121150
def StartOfflineToOnlineIngestionJob(
122151
self, request: StartOfflineToOnlineIngestionJobRequest, context
123152
):
@@ -128,6 +157,11 @@ def StartOfflineToOnlineIngestionJob(
128157
f"Project {request.project} is not whitelisted. Please contact your Feast administrator to whitelist it."
129158
)
130159

160+
if not self.is_feature_table_whitelisted(request.project, request.table_name):
161+
raise ValueError(
162+
f"Project {request.project}:{request.table_name} is not whitelisted. Please contact your Feast administrator to whitelist it."
163+
)
164+
131165
feature_table = self.client.feature_store.get_feature_table(
132166
request.table_name, request.project
133167
)

0 commit comments

Comments
 (0)