cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden #53906

boeboe · 2017-10-13T18:25:34Z

Hi all,

/sig area/kubectl
/sig sig/network
@kubernetes/sig-network-bugs

We made the upstep from k8s 1.6 to 1.7.8 on our k8s clusters. As good k8s citizens, we have updated our NetworkPolicy yaml deployment files to use the new networking.k8s.io/v1 apiVersion. It appears however that a retrieval of NetworkPolicies deployed this way, still use the extensions/v1beta1 apiVersion. I suppose this is due to backwards compatibility concerns, which I can perfectly understand.

What is the issue however, is that we deploy our NetworkPolicies (in an automated way, using a cluster applier process within each namespace, that pulls the yaml config from git and applies them periodically) using "kubectl apply -f *.yaml". The kubectl client however refuses to do so, since he detects a false change between what's deployed (extensions/v1beta1) and what's to be deployed (networking.k8s.io/v1). I'd expect the same compatibility check/measures at apply time (not only at fetch/retrieval time).

Does anyone have any advice on how to get around this? We know that we could write a more advanced cluster applier process that circumvents this corner cases in an if-then-else matter, but I would expect the kubectl client to handle those things, or am I overlooking a certain kubectl feature here?

Best regards,
Bart

boeboe · 2017-10-13T19:04:48Z

/sig area/kubectl
/sig sig/network

boeboe · 2017-10-13T19:06:00Z

@kubernetes/sig-network-bugs

k8s-ci-robot · 2017-10-13T19:06:09Z

@boeboe: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

liggitt · 2017-10-13T21:28:08Z

It appears however that a retrieval of NetworkPolicies deployed this way, still use the extensions/v1beta1 apiVersion. I suppose this is due to backwards compatibility concerns, which I can perfectly understand.

For compatibility reasons, without qualifying the resource, the extensions API group takes precedence with doing kubectl get.

You can qualify the API group you want to fetch. This will fetch the policies from the networking.k8s.io API group:

kubectl get networkpolicies.networking.k8s.io

Or even more specifically, the v1 version of the networking.k8s.io API group:

kubectl get networkpolicies.v1.networking.k8s.io

If you are scripting kubectl to apply and expect specific versions of resources, I would strongly recommend fully-qualifying the resources.

…l get gives back beta which make us to run ansible without any issue Refer: kubernetes/kubernetes#53906

boeboe · 2017-10-17T10:44:54Z

@liggitt can you please re-open this issue, since one cannot specify a fully-qualifying resource when applying through a yaml file ... or to rephrase, we were fully qualifying our yaml files using apiVersion: networking.k8s.io/v1, but this brakes kubectl apply -f nwp.yaml expected behaviour.

# cat networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-policy
spec:
  podSelector:
    matchLabels:
      name: whatever
  ingress:
    - ports:
        - protocol: TCP
          port: 9090
# k apply -f networkpolicy.yaml
networkpolicy "test-policy" created
# k get networkpolicy test-policy -o yaml
apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"test-policy","namespace":"es4-acc-mon-prometheus"},"spec":{"ingress":[{"ports":[{"port":9090,"protocol":"TCP"}]}],"podSelector":{"matchLabels":{"name":"whatever"}}}}
  creationTimestamp: 2017-10-17T10:40:37Z
  generation: 1
  name: test-policy
  namespace: es4-acc-mon-prometheus
  resourceVersion: "54940382"
  selfLink: /apis/extensions/v1beta1/namespaces/es4-acc-mon-prometheus/networkpolicies/test-policy
  uid: 9cb5d845-b327-11e7-85ad-005056830d0a
spec:
  ingress:
  - ports:
    - port: 9090
      protocol: TCP
  podSelector:
    matchLabels:
      name: whatever
# k apply -f networkpolicy.yaml
The NetworkPolicy "test-policy" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden.
# k delete networkpolicy test-policy
networkpolicy "test-policy" deleted

When trying to fully qualify as you mention in your reply....

# cat networkpolicy-fullname.yaml
apiVersion: networkpolicies.v1.networking.k8s.io
kind: NetworkPolicy
metadata:
  name: test-policy
spec:
  podSelector:
    matchLabels:
      name: whatever
  ingress:
    - ports:
        - protocol: TCP
          port: 9090
# k apply -f networkpolicy-fullname.yaml
error: unable to recognize "networkpolicy-fullname.yaml": no matches for /, Kind=NetworkPolicy

Bottom line it boils down to this error...

The NetworkPolicy "test-policy" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden.

... which should not happen since no changes where made on the networkpolicy. There is a mismatch between backwards compatible read vs non-forward compatible apply/update.

liggitt · 2017-10-17T12:06:17Z

can you please re-open this issue, since one cannot specify a fully-qualifying resource when applying through a yaml file

Resources in a yaml file are always fully qualified (they include the full api group and version)

That syntax for fully specifying a resource, api group, and api version is specific to kubectl

Your subsequent get is the unqualified bit, so the client defaults to the first API group and version containing that resource (extensions/v1beta1) and the server converts the saved object to that version and returns it to you

the following get would show it in the expected version

k get networkpolicies.v1.networking.k8s.io test-policy -o yaml

liggitt · 2017-10-17T12:12:18Z

The apply issue is likely related to defaulting or validation in the v1 group alone.

The returned extensions/v1beta1 content for an unqualified get is unrelated

cmluciano · 2017-10-17T18:49:01Z

May be related #51955

cmluciano · 2017-10-17T18:49:05Z

/assign

caseydavenport · 2017-10-17T18:50:09Z

The NetworkPolicy "test-policy" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden.

This is expected if running on a k8s < v1.8 cluster, since updates to NP were only introduced in v1.8. If you try to update on a v1.8 cluster is should succeed.

error: unable to recognize "networkpolicy-fullname.yaml": no matches for /, Kind=NetworkPolicy

I think you've got the wrong apiVersion in your manifest - networkpolicies.v1.networking.k8s.io should be networking.k8s.io/v1. The networkpolicies.v1.networking.k8s.io bit is just for the kubectl get command.

boeboe · 2017-10-17T22:00:14Z

@caseydavenport

apply on network policy was working on k8s < 1.7, that is, if the policy applied was exactly the same as the one in the cluster (hence your remark indeed that updates are only supported from 1.8 onwards). My point is that on 1.7 an apply (with a no-update operation, as in yaml == cluster state) does not work since apiVersion issue legacy vs new.

caseydavenport · 2017-10-17T23:40:23Z

@boeboe oh yes, I see what you mean. Thanks for clarifying.

liggitt · 2017-10-17T23:55:35Z

the objects as seen internally by the validation code are:

(networking.NetworkPolicySpec) {                                  (networking.NetworkPolicySpec) {
 PodSelector: (v1.LabelSelector) {                                 PodSelector: (v1.LabelSelector) {
  MatchLabels: (map[string]string) (len=1) {                        MatchLabels: (map[string]string) (len=1) {
   (string) (len=4) "name": (string) (len=8) "whatever"              (string) (len=4) "name": (string) (len=8) "whatever"
  },                                                                },
  MatchExpressions: ([]v1.LabelSelectorRequirement) <nil>           MatchExpressions: ([]v1.LabelSelectorRequirement) <nil>
 },                                                                },
 Ingress: ([]networking.NetworkPolicyIngressRule) (len=1 cap=1) {  Ingress: ([]networking.NetworkPolicyIngressRule) (len=1 cap=1) {
  (networking.NetworkPolicyIngressRule) {                           (networking.NetworkPolicyIngressRule) {
   Ports: ([]networking.NetworkPolicyPort) (len=1 cap=1) {           Ports: ([]networking.NetworkPolicyPort) (len=1 cap=1) {
    (networking.NetworkPolicyPort) {                                  (networking.NetworkPolicyPort) {
     Protocol: (*api.Protocol)(0xc427e1d4f0)((len=3) "TCP"),           Protocol: (*api.Protocol)(0xc427e1cd80)((len=3) "TCP"),
     Port: (*intstr.IntOrString)(0xc4219a3780)({                       Port: (*intstr.IntOrString)(0xc4219a30e0)({
      Type: (intstr.Type) 0,                                            Type: (intstr.Type) 0,
      IntVal: (int32) 9090,                                             IntVal: (int32) 9090,
      StrVal: (string) ""                                               StrVal: (string) ""
     })                                                                })
    }                                                                 }
   },                                                                },
   From: ([]networking.NetworkPolicyPeer) <nil>                      From: ([]networking.NetworkPolicyPeer) {
  }                                                                  }
 }                                                                  }
}                                                                  }
                                                                  }

note the ingress rule From field is nil on the left and an empty (but non-nil) slice on the right

this is the troublesome line:

kubernetes/pkg/apis/extensions/v1beta1/conversion.go

Line 337 in 0a69594

out.From = make([]NetworkPolicyPeer, len(in.From))

which initializes the From field to an empty slice when converting, even if the starting object has a nil From field

…master Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Ensure network policy conversion round trips nil from field xref kubernetes#53906 ```release-note NONE ```

Automatic merge from submit-queue. Ensure network policy conversion round trips nil from fields Fixes #53906 (inability to apply network policies against v1 in 1.7) * Changes non-mutation check to API equality (which treats nil and [] slices identically) * Fixed bug in manual conversion that took an input of `nil` and produced an output of `[]` ```release-note Restores the ability to apply network policy objects against the networking.k8s.io/v1 API ```

cmluciano · 2017-10-31T20:17:23Z

fixed by #54109

k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 13, 2017

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. kind/bug Categorizes issue or PR as related to a bug. labels Oct 13, 2017

k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 13, 2017

liggitt closed this as completed Oct 14, 2017

gurvindersingh pushed a commit to Uninett/research-lab that referenced this issue Oct 17, 2017

revert the api version to beta for network policy as currently kubect…

e866d80

…l get gives back beta which make us to run ansible without any issue Refer: kubernetes/kubernetes#53906

boeboe changed the title ~~Networkpolicy extensions/v1beta1 to networking.k8s.io/v1 kubectl issue~~ Networkpolicy extensions/v1beta1 to networking.k8s.io/v1 kubectl apply issue Oct 17, 2017

liggitt changed the title ~~Networkpolicy extensions/v1beta1 to networking.k8s.io/v1 kubectl apply issue~~ cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden Oct 17, 2017

liggitt reopened this Oct 17, 2017

k8s-ci-robot assigned cmluciano Oct 17, 2017

liggitt mentioned this issue Oct 17, 2017

Ensure network policy conversion round trips nil from fields #54106

Merged

This was referenced Oct 18, 2017

Ensure network policy conversion round trips nil from field #54109

Merged

Ensure network policy conversion round trips nil from field #54110

Closed

cmluciano closed this as completed Oct 31, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden #53906

cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden #53906

boeboe commented Oct 13, 2017 •

edited

Loading

boeboe commented Oct 13, 2017

boeboe commented Oct 13, 2017

k8s-ci-robot commented Oct 13, 2017

liggitt commented Oct 13, 2017 •

edited

Loading

boeboe commented Oct 17, 2017 •

edited

Loading

liggitt commented Oct 17, 2017

liggitt commented Oct 17, 2017

cmluciano commented Oct 17, 2017

cmluciano commented Oct 17, 2017

caseydavenport commented Oct 17, 2017

boeboe commented Oct 17, 2017 •

edited

Loading

caseydavenport commented Oct 17, 2017

liggitt commented Oct 17, 2017 •

edited

Loading

cmluciano commented Oct 31, 2017

cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden #53906

cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden #53906

Comments

boeboe commented Oct 13, 2017 • edited Loading

boeboe commented Oct 13, 2017

boeboe commented Oct 13, 2017

k8s-ci-robot commented Oct 13, 2017

liggitt commented Oct 13, 2017 • edited Loading

boeboe commented Oct 17, 2017 • edited Loading

liggitt commented Oct 17, 2017

liggitt commented Oct 17, 2017

cmluciano commented Oct 17, 2017

cmluciano commented Oct 17, 2017

caseydavenport commented Oct 17, 2017

boeboe commented Oct 17, 2017 • edited Loading

caseydavenport commented Oct 17, 2017

liggitt commented Oct 17, 2017 • edited Loading

cmluciano commented Oct 31, 2017

boeboe commented Oct 13, 2017 •

edited

Loading

liggitt commented Oct 13, 2017 •

edited

Loading

boeboe commented Oct 17, 2017 •

edited

Loading

boeboe commented Oct 17, 2017 •

edited

Loading

liggitt commented Oct 17, 2017 •

edited

Loading