• v1.17.0-rc.1
  • Downloads for v1.17.0-rc.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.17.0-beta.2
    • Other notable changes
• v1.17.0-beta.2
  • Downloads for v1.17.0-beta.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.17.0-beta.1
    • Action Required
    • Other notable changes
• v1.17.0-beta.1
  • Downloads for v1.17.0-beta.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.17.0-alpha.3
    • Action Required
    • Other notable changes
• v1.17.0-alpha.3
  • Downloads for v1.17.0-alpha.3
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.17.0-alpha.2
    • Action Required
    • Other notable changes
• v1.17.0-alpha.2
  • Downloads for v1.17.0-alpha.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.17.0-alpha.1
    • Action Required
    • Other notable changes
• v1.17.0-alpha.1
  • Downloads for v1.17.0-alpha.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.16.0
    • Action Required
    • Other notable changes

Documentation

• kubeadm: fix a panic in case the KubeProxyConfiguration feature gates were not initialized. (#85524, @Arvinderpal)
• kubeadm: fix stray "node-cidr-mask-size" flag in the kube-controller-manager manifest when IPv6DualStack is enabled (#85494, @tedyu)
• CRDs can have fields named type with value array and nested array with items fields without validation to fall over this. (#85223, @sttts)
• Resolves error from v1.17.0-beta.2 with --authorizer-mode webhook complaining about an invalid version (#85441, @liggitt)
• Promote CSIMigrationAWS to Beta (off by default since it requires installation of the AWS EBS CSI Driver) (#85237, @leakingtapan)
  • The in-tree AWS EBS plugin "kubernetes.io/aws-ebs" is now deprecated and will be removed in 1.21. Users should enable CSIMigration + CSIMigrationAWS features and install the AWS EBS CSI Driver (https://github.com/kubernetes-sigs/aws-ebs-csi-driver) to avoid disruption to existing Pod and PVC objects at that time.
  • Users should start using the AWS EBS CSI CSI Driver directly for any new volumes.

Documentation

• Renamed FeatureGate RequestManagement to APIPriorityAndFairness. This feature gate is an alpha and has not yet been associated with any actual functionality. (#85260, @MikeSpreitzer)
  • Action required: change references to feature gate RequestManagement into references to APIPriorityAndFairness
• ACTION REQUIRED: kubeadm: add a new "kubelet-finalize" phase as part of the "init" workflow and an experimental sub-phase to enable automatic kubelet client certificate rotation on primary control-plane nodes. (#84118, @neolit123)
  • Prior to 1.17 and for existing nodes created by "kubeadm init" where kubelet client certificate rotation is desired, you must modify "/etc/kubernetes/kubelet.conf" to point to the PEM symlink for rotation:
  • "client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem" and "client-key: /var/lib/kubelet/pki/kubelet-client-current.pem", replacing the embedded client certificate and key.
• action required: kubeadm deprecates the use of the hyperkube image (#85094, @rosti)

• Following metrics have been turned off: (#83837, @RainbowMango)
  • • apiserver_request_count
  • • apiserver_request_latencies
  • • apiserver_request_latencies_summary
  • • apiserver_dropped_requests
  • • etcd_request_latencies_summary
  • • apiserver_storage_transformation_latencies_microseconds
  • • apiserver_storage_data_key_generation_latencies_microseconds
  • • apiserver_storage_transformation_failures_total
• OpenAPI v3 format in CustomResourceDefinition schemas are now documented. (#85381, @sttts)
• The official kube-proxy image (used by kubeadm, among other things) is now (#82966, @danwinship)
  • compatible with systems running iptables 1.8 in "nft" mode, and will autodetect
  • which mode it should use.
• Kubenet: added HostPort IPv6 support (#80854, @aojea)
  • HostPortManager: operates only with one IP family, failing if receives portmapping entries with different IP families
  • HostPortSyncer: operates only with one IP family, skipping portmap entries with different IP families
• Implement the documented API semantics of list-type and map-type atomic to reject non-atomic sub-types. (#84722, @sttts)
• kubeadm: Fix a bug where kubeadm cannot parse kubelet's version if the latter dumps logs on the standard error. (#85351, @rosti)
• EndpointSlices are not enabled by default. Use the EndpointSlice feature gate to enable this feature. (#85365, @robscott)
• Feature gates CSIMigration to Beta (on by default) and CSIMigrationGCE to Beta (off by default since it requires installation of the GCE PD CSI Driver) (#85231, @davidz627)
  • The in-tree GCE PD plugin "kubernetes.io/gce-pd" is now deprecated and will be removed in 1.21. Users should enable CSIMigration + CSIMigrationGCE features and install the GCE PD CSI Driver (https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) to avoid disruption to existing Pod and PVC objects at that time.
  • Users should start using the GCE PD CSI CSI Driver directly for any new volumes.
• kube-controller-manager: Fixes bug setting headless service labels on endpoints (#85361, @liggitt)
• When upgrading to 1.17 with a cluster with EndpointSlices enabled, the endpointslice.kubernetes.io/managed-by label needs to be set on each EndpointSlice. (#85359, @robscott)
• Remove redundant API validation when using Service Topology with externalTrafficPolicy=Local (#85346, @andrewsykim)
• Following metrics have been turned off: (#83838, @RainbowMango)
  • • scheduler_scheduling_latency_seconds
  • • scheduler_e2e_scheduling_latency_microseconds
  • • scheduler_scheduling_algorithm_latency_microseconds
  • • scheduler_scheduling_algorithm_predicate_evaluation
  • • scheduler_scheduling_algorithm_priority_evaluation
  • • scheduler_scheduling_algorithm_preemption_evaluation
  • • scheduler_scheduling_binding_latency_microseconds
• CSI Migration: Fixes issue where all volumes with the same inline volume inner spec name were staged in the same path. Migrated inline volumes are now staged at a unique path per unique volume. (#84754, @davidz627)
• kube-controller-manager (#79993, @aramase)
  • --node-cidr-mask-size-ipv4 int32 Default: 24. Mask size for IPv4 node-cidr in dual-stack cluster.
  • --node-cidr-mask-size-ipv6 int32 Default: 64. Mask size for IPv6 node-cidr in dual-stack cluster.
  • These 2 flags can be used only for dual-stack clusters. For non dual-stack clusters, continue to use
  • --node-cidr-mask-size flag to configure the mask size.
  • The default node cidr mask size for IPv6 was 24 which is now changed to 64.
• The following information is available through environment variables: (#83123, @aramase)
  • status.podIPs - the pod's IP addresses
• update github.com/vishvananda/netlink to v1.0.0 (#83576, @andrewsykim)
• kubectl: --resource-version now works properly in label/annotate/set selector commands when racing with other clients to update the target object (#85285, @liggitt)
• --runtime-config now supports an api/beta=false value which disables all built-in REST API versions matching v[0-9]+beta[0-9]+. (#84304, @liggitt)
  • --feature-gates now supports an AllBeta=false value which disables all beta feature gates.
• kube-proxy now supports DualStack feature with EndpointSlices and IPVS. (#85246, @robscott)
• Add table convertor to componentstatus. (#85174, @zhouya0)
• kubeadm: added retry to all the calls to the etcd API so kubeadm will be more resilient to network glitches (#85201, @fabriziopandini)
• azure: update disk lock logic per vm during attach/detach to allow concurrent updates for different nodes. (#85115, @aramase)
• Scale custom resource unconditionally if resourceVersion is not provided (#80572, @knight42)
• Bump CSI version to 1.2.0 (#84832, @gnufied)
• Adds Windows Server build information as a label on the node. (#84472, @gab-satchi)
• Deprecated metric kubeproxy_sync_proxy_rules_latency_microseconds has been turned off. (#83839, @RainbowMango)
• Existing PVs are converted to use volume topology if migration is enabled. (#83394, @bertinatto)
• Finalizer Protection for Service LoadBalancers is now in GA (enabled by default). This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted. (#85023, @MrHohn)
• EndpointSlices are now beta and enabled by default for better Network Endpoint performance at scale. (#84390, @robscott)
• When using Containerd on Windows, the TerminationMessagePath file will now be mounted in the Windows Pod. (#83057, @bclau)
• apiservers based on k8s.io/apiserver with delegated authn based on cluster authentication will automatically update to new authentication information when the authoritative configmap is updated. (#85004, @deads2k)
• fix vmss dirty cache issue in disk attach/detach on vmss node (#85158, @andyzhangx)
• Fixes a bug in kubeadm that caused init and join to hang indefinitely in specific conditions. (#85156, @chuckha)
• kube-apiserver: Authentication configuration for mutating and validating admission webhooks referenced from an --admission-control-config-file can now be specified with apiVersion: apiserver.config.k8s.io/v1, kind: WebhookAdmissionConfiguration. (#85138, @liggitt)
• Kubeadm now includes CoreDNS version 1.6.5 (#85109, @rajansandeep)
  • • kubernetes plugin adds metrics to measure kubernetes control plane latency.
  • • the health plugin now includes the lameduck option by default, which waits for a duration before shutting down.
• Kubeadm now includes CoreDNS version 1.6.5 (#85108, @rajansandeep)
  • • kubernetes plugin adds metrics to measure kubernetes control plane latency.
  • • the health plugin now includes the lameduck option by default, which waits for a duration before shutting down.
• kube-apiserver: The ResourceQuota admission plugin configuration referenced from --admission-control-config-file admission config has been promoted to apiVersion: apiserver.config.k8s.io/v1, kind: ResourceQuotaConfiguration with no schema changes. (#85099, @liggitt)
• kube-apiserver: The AdmissionConfiguration type accepted by --admission-control-config-file has been promoted to apiserver.config.k8s.io/v1 with no schema changes. (#85098, @liggitt)
• New flag --show-hidden-metrics-for-version in kube-apiserver can be used to show all hidden metrics that deprecated in the previous minor release. (#84292, @RainbowMango)
• The ResourceQuotaScopeSelectors feature has graduated to GA. The ResourceQuotaScopeSelectors feature gate is now unconditionally enabled and will be removed in 1.18. (#82690, @draveness)
• Fixed bug when using kubeadm alpha certs commands with clusters using external etcd (#85091, @fabriziopandini)
• Fix a bug that a node Lease object may have been created without OwnerReference. (#84998, @wojtek-t)
• Splitting IP address type into IPv4 and IPv6 for EndpointSlices (#84971, @robscott)
• Pod process namespace sharing is now Generally Available. The PodShareProcessNamespace feature gate is now deprecated and will be removed in Kubernetes 1.19. (#84356, @verb)
• Fix incorrect network policy description suggesting that pods are isolated when a network policy has no rules of a given type (#84194, @jackkleeman)
• add RequiresExactMatch for label.Selector (#85048, @shaloulcy)
• Deprecated metric rest_client_request_latency_seconds has been turned off. (#83836, @RainbowMango)
• Removed dependency on kubectl from several storage E2E tests (#84042, @okartau)
• kubeadm no longer defaults or validates the component configs of the kubelet or kube-proxy (#79223, @rosti)
• Add plugin_execution_duration_seconds metric for scheduler framework plugins. (#84522, @liu-cong)
• Moving WindowsRunAsUserName feature to beta (#84882, @marosset)
• Node-specific volume limits has graduated to GA. (#83568, @bertinatto)
• kubelet and aggregated API servers now use v1 TokenReview and SubjectAccessReview endpoints to check authentication/authorization. (#84768, @liggitt) * kube-apiserver can now specify --authentication-token-webhook-version=v1 or --authorization-webhook-version=v1 to use v1 TokenReview and SubjectAccessReview API objects when communicating with authentication and authorization webhooks.
• BREAKING CHANGE: Remove plugin watching of deprecated directory {kubelet_root_dir}/plugins and CSI V0 support in accordance with deprecation announcement in https://v1-13.docs.kubernetes.io/docs/setup/release/notes/ (#84533, @davidz627)
• Adds a new label to indicate what is managing an EndpointSlice. (#83965, @robscott)
• Fix a racing issue in client-go UpdateTransportConfig. (#80284, @danielqsj)
• Enables VolumeSnapshotDataSource feature gate and promotes volume snapshot APIs to beta. (#80058, @xing-yang)
• Added appProtocol field to EndpointSlice Port (#83815, @howardjohn)
• kubeadm alpha certs command now skip missing files (#85092, @fabriziopandini)
• A new flag "progress-report-url" has been added to the test context which allows progress information about the test run to be sent to a webhook. In addition, this information is printed to stdout to aid in users watching the logs. (#84524, @johnSchnake)
• kubeadm: remove the deprecated "--cri-socket" flag for "kubeadm upgrade apply". The flag has been deprecated since v1.14. (#85044, @neolit123)
• Clients can request protobuf and json and correctly negotiate with the server for JSON for CRD objects, allowing all client libraries to request protobuf if it is available. If an error occurs negotiating a watch with the server, the error is immediately return by the client Watch() method instead of being sent as an Error event on the watch stream. (#84692, @smarterclayton)
• Following metrics from kubelet are now marked as with the ALPHA stability level: (#84987, @RainbowMango)
  • node_cpu_usage_seconds_total
  • node_memory_working_set_bytes
  • container_cpu_usage_seconds_total
  • container_memory_working_set_bytes
  • scrape_error
• Following metrics from kubelet are now marked as with the ALPHA stability level: (#84907, @RainbowMango)
  • kubelet_container_log_filesystem_used_bytes
  • kubelet_volume_stats_capacity_bytes
  • kubelet_volume_stats_available_bytes
  • kubelet_volume_stats_used_bytes
  • kubelet_volume_stats_inodes
  • kubelet_volume_stats_inodes_free
  • kubelet_volume_stats_inodes_used
  • plugin_manager_total_plugins
  • volume_manager_total_volumes
• kubeadm: enable the usage of the secure kube-scheduler and kube-controller-manager ports for health checks. For kube-scheduler was 10251, becomes 10259. For kube-controller-manager was 10252, becomes 10257. (#85043, @neolit123)
• kubeadm: prevent potential hanging of commands such as "kubeadm reset" if the apiserver endpoint is not reachable. (#84648, @neolit123)
• Mirror pods now include an ownerReference for the node that created them. (#84485, @tallclair)
• kubeadm: fix skipped etcd upgrade on secondary control-plane nodes when the command "kubeadm upgrade node" is used. (#85024, @neolit123)
• fix race condition when attach/delete azure disk in same time (#84917, @andyzhangx)
• If given an IPv6 bind-address, kube-apiserver will now advertise an IPv6 endpoint for the kubernetes.default service. (#84727, @danwinship)
• kubeadm: the command "kubeadm token create" now has a "--certificate-key" flag that can be used for the formation of join commands for control-planes with automatic copy of certificates (#84591, @TheLastProject)
• Deprecate the instance type beta label ("beta.kubernetes.io/instance-type") in favor of it's GA equivalent: "node.kubernetes.io/instance-type" (#82049, @andrewsykim)
• kube-apiserver: Fixed a regression accepting patch requests > 1MB (#84963, @liggitt)
• Promote NodeLease feature to GA. (#84351, @wojtek-t)
  • The feature make Lease object changes an additional healthiness signal from Node. Together with that, we reduce frequency of NodeStatus updates to 5m by default in case of no changes to status itself
• Following metrics from kube-controller-manager are now marked as with the ALPHA stability level: (#84896, @RainbowMango)
  • storage_count_attachable_volumes_in_use
  • attachdetach_controller_total_volumes
  • pv_collector_bound_pv_count
  • pv_collector_unbound_pv_count
  • pv_collector_bound_pvc_count
  • pv_collector_unbound_pvc_count
• Deprecate the beta labels for zones ("failure-domain.beta.kubernetes.io/zone") and (#81431, @andrewsykim)
  • regions ("failure-domain.beta.kubernetes.io/region") in favor of their GA equivalents:
  • "topology.kubernetes.io/zone" and "topology.kubernetes.io/region".
  • The beta labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" will be removed in v1.21
• kube-apiserver: fixed a bug that could cause a goroutine leak if the apiserver encountered an encoding error serving a watch to a websocket watcher (#84693, @tedyu)
• EndpointSlice hostname is now set in the same conditions Endpoints hostname is. (#84207, @robscott)
• Simple script based hyperkube image that bundles all the necessary binaries. This is a equivalent replacement for the image based on the go based hyperkube command + image. (#84662, @dims)
• configmaps/extension-apiserver-authentication in kube-system is continuously updated by kube-apiservers, instead of just at apiserver start (#82705, @deads2k)
• kubeadm: fix an issue with the kube-proxy container env. variables (#84888, @neolit123)
• Updated EndpointSlices to use PublishNotReadyAddresses from Services. (#84573, @robscott)
• The example API server has renamed its wardle.k8s.io API group to wardle.example.com (#81670, @liggitt)
• A new kubelet command line option, --reserved-cpus, is introduced to explicitly define the the CPU list that will be reserved for system. For example, if --reserved-cpus=0,1,2,3 is specified, then cpu 0,1,2,3 will be reserved for the system. On a system with 24 CPUs, the user may specify isolcpus=4-23 for the kernel option and use CPU 4-23 for the user containers. (#83592, @jianzzha)
• Utilize diagnostics tool to dump GKE windows test logs (#83517, @YangLu1031)
• Improving the performance of Endpoint and EndpointSlice controllers by caching Service Selectors (#84280, @gongguan)
• When the go-client reflector relists, the ResourceVersion list option is set to the reflector's latest synced resource version to ensure the reflector does not "go back in time" and reprocess events older than it has already processed. If the the server responds with an HTTP 410 (Gone) status code response, the relist falls back to using resourceVersion="". (#83520, @jpbetz)
• Kubernetes now requires go1.13.4+ to build (#82809, @liggitt)
• Ensure health probes are created for local traffic policy UDP services on Azure (#84802, @feiskyer)
• CRDs defaulting is promoted to GA. Note: the feature gate CustomResourceDefaulting will be removed in 1.18. (#84713, @sttts)
• Profiling is enabled by default in the scheduler (#84835, @denkensk)
• CSI Migration: GCE PD access mode now reflects read only status of inline volumes - this allows multi-attach for read only many PDs (#84809, @davidz627)
• All resources within the rbac.authorization.k8s.io/v1alpha1 and rbac.authorization.k8s.io/v1beta1 API groups are deprecated in favor of rbac.authorization.k8s.io/v1, and will no longer be served in v1.20. (#84758, @liggitt)
• Scheduler ComponentConfig fields are now pointers (#83619, @damemi)
• Adding initial EndpointSlice metrics. (#83257, @robscott)

Documentation

• Graduate ScheduleDaemonSetPods to GA. (feature gate will be removed in 1.18) action required. (#82795, @draveness)

• kube-scheduler: emits a warning when a malformed component config file is used with v1alpha1. (#84129, @obitech)
• add azure disk encryption(SSE+CMK) support (#84605, @andyzhangx)
• The certificate signer no longer accepts ca.key passwords via the CFSSL_CA_PK_PASSWORD environment variable. This capability was not prompted by user request, never advertised, and recommended against in the security audit. (#84677, @mikedanese)
• Reduce default NodeStatusReportFrequency to 5 minutes. With this change, periodic node status updates will be send every 5m if node status doesn't change (otherwise they are still send with 10s). (#84007, @wojtek-t)
  • Bump NodeProblemDetector version to v0.8.0 to reduce forced NodeStatus updates frequency to 5 minutes.
• CSI Topology feature is GA. The CSINodeInfo feature gate is deprecated and will be removed in a future release. The storage.k8s.io/v1beta1 CSINode object is deprecated and will be removed in a future release. (#83474, @msau42)
• Only validate duplication of the RequestedToCapacityRatio custom priority and allow other custom predicates/priorities (#84646, @liu-cong)
• Added kubelet serving certificate metric server_rotation_seconds which is a histogram reporting the age of a just rotated serving certificate in seconds. (#84534, @sambdavidson)
• During namespace deletion some controllers create event and log spam because they do not recognize namespace deletion as a terminal state. (#84123, @smarterclayton)
• Removed Alpha feature MountContainers (#84365, @codenrhoden)
• People can see the right log and note. (#84637, @zhipengzuo)
• Ensure the KUBE-MARK-DROP chain in kube-proxy mode=iptables. The chain is ensured for both ipv4 and ipv6 in dual-stack operation. (#84422, @aojea)
• deprecate cleanup-ipvs flag (#83832, @gongguan)
• Scheduler Policy API has a new recommended apiVersion "apiVersion: kubescheduler.config.k8s.io/v1" which is consistent with the scheduler API group "kubescheduler.config.k8s.io". It holds the same API as the old apiVersion "apiVersion: v1". (#83578, @Huang-Wei)
• Fixed a bug in the single-numa-policy of the TopologyManager. Previously, best-effort pods would result in a terminated state with a TopologyAffinity error. Now they will run as expected. (#83777, @lmdaly)
• local: support local filesystem volume with block resource reconstruction (#84218, @cofyc)
• Fix the bug that EndpointSlice for masters wasn't created after enabling EndpointSlice feature on a pre-existing cluster. (#84421, @tnqn)
• kubelet: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83204, @obitech)
• kubeadm now propagates proxy environment variables to kube-proxy (#84559, @yastij)
• Reload apiserver SNI certificates from disk every minute (#84303, @jackkleeman)
• sourcesReady provides the readiness of kubelet configuration sources such as apiserver update readiness. (#81344, @zouyee)
• Update Azure SDK versions to v35.0.0 (#84543, @andyzhangx)
• Fixed EndpointSlice port name validation to match Endpoint port name validation (allowing port names longer than 15 characters) (#84481, @robscott)
• Scheduler now reports metrics on cache size including nodes, pods, and assumed pods (#83508, @damemi)
• kube-proxy: emits a warning when a malformed component config file is used with v1alpha1. (#84143, @phenixblue)
• Update default etcd server version to 3.4.3 (#84329, @jingyih)
• Scheduler policy configs can no longer be declared multiple times (#83963, @damemi)
• This PR sets the --cluster-dns flag value to kube-dns service IP whether or not NodeLocal DNSCache is enabled. NodeLocal DNSCache will listen on both the link-local as well as the service IP. (#84383, @prameshj)
• Remove prometheus cluster monitoring addon from kube-up (#83442, @serathius)
• update the latest validated version of Docker to 19.03 (#84476, @neolit123)
• kubeadm: always mount the kube-controller-manager hostPath volume that is given by the --flex-volume-plugin-dir flag. (#84468, @neolit123)
• Introduce x-kubernetes-map-type annotation as a CRD API extension. Enables this particular validation for server-side apply. (#84113, @enxebre)
• kube-scheduler now fallbacks to emitting events using core/v1 Events when events.k8s.io/v1beta1 is disabled. (#83692, @yastij)
• Migrate controller-manager and scheduler to EndpointsLeases leader election. (#84084, @wojtek-t)
• User can now use component config to configure NodeLabel plugin for the scheduler framework. (#84297, @liu-cong)
• local: support local volume block mode reconstruction (#84173, @cofyc)
• Fixed kubectl endpointslice output for get requests (#82603, @robscott)
• set config.BindAddress to IPv4 address "127.0.0.1" if not specified (#83822, @zouyee)
• CSI detach timeout increased from 10 seconds to 2 minutes (#84321, @cduchesne)
• Update etcd client side to v3.4.3 (#83987, @wenjiaswe)
  • Deprecated prometheus request meta-metrics have been removed (http_request_duration_microseconds, http_request_duration_microseconds_sum, http_request_duration_microseconds_count, http_request_size_bytes, http_request_size_bytes_sum, http_request_size_bytes_count, http_requests_total, http_response_size_bytes, http_response_size_bytes_sum, http_response_size_bytes_count) due to removal from the prometheus client library. Prometheus http request meta-metrics are now generated from promhttp.InstrumentMetricHandler instead.
• The built-in system:csi-external-provisioner and system:csi-external-attacher cluster roles are removed as of 1.17 release (#84282, @tedyu)
• Pod labels can no longer be updated through the pod/status updates by nodes. (#84260, @tallclair)
• Reload apiserver serving certificate from disk every minute (#84200, @jackkleeman)
• Adds FQDN addressType support for EndpointSlice. (#84091, @robscott)
• Add permit_wait_duration_seconds metric for scheduler. (#84011, @liu-cong)
• Optimize inter-pod affinity preferredDuringSchedulingIgnoredDuringExecution type, up to 4x in some cases. (#84264, @ahg-g)
• When a namespace is being deleted and spec.finalizers are still being processed, stop returning a 409 conflict error and instead return the object as we would during metadata.finalizer processing. (#84122, @smarterclayton)
• client-ca bundles for the all generic-apiserver based servers will dynamically reload from disk on content changes (#83579, @deads2k)
• Reduced frequency of DescribeVolumes calls of AWS API when attaching/detaching a volume. (#84181, @jsafrane)
• Add a metric to track number of scheduler binding and prioritizing goroutines (#83535, @wgliang)
• Fix kubelet metrics gathering on non-English Windows hosts (#84156, @wawa0210)
• A new kubelet_preemptions metric is reported from Kubelets to track the number of preemptions occuring over time, and which resource is triggering those preemptions. (#84120, @smarterclayton)

Documentation

• Graduate TaintNodesByCondition to GA in 1.17. (feature gate will be removed in 1.18) action required (#82703, @draveness)

• TaintNodesByCondition was graduated to GA, CheckNodeMemoryPressure, CheckNodePIDPressure, CheckNodeDiskPressure, CheckNodeCondition were accidentally removed since 1.12, the replacement is to use CheckNodeUnschedulablePred (#84152, @draveness)
• filter plugin for cloud provider storage predicate (#84148, @gongguan)
• Fixed binding of block PersistentVolumes / PersistentVolumeClaims when BlockVolume feature is off. (#84049, @jsafrane)
• Updated kube-proxy ipvs README with correct grep argument to list loaded ipvs modules (#83677, @pete911)
• Add data cache flushing during unmount device for GCE-PD driver in Windows Server. (#83591, @jingxu97)
• Adds a metric apiserver_request_error_total to kube-apiserver. This metric tallies the number of request_errors encountered by verb, group, version, resource, subresource, scope, component, and code. (#83427, @logicalhan)
• Refactor scheduler's framework permit API. (#83756, @hex108)
• The kubectl's api-resource command now has a --sort-by flag to sort resources by name or kind. (#81971, @laddng)
• Update to use go1.12.12 (#84064, @cblecker)
• Update to Ingress-GCE v1.6.1 (#84018, @rramkumar1)
• Update Cluster Autoscaler version to 1.16.2 (CA release docs: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.16.2) (#84038, @losipiuk)
• When scaling down a ReplicaSet, delete doubled up replicas first, where a "doubled up replica" is defined as one that is on the same node as an active replica belonging to a related ReplicaSet. ReplicaSets are considered "related" if they have a common controller (typically a Deployment). (#80004, @Miciah)
• Promote WatchBookmark feature to GA. (#83195, @wojtek-t)
  • With WatchBookmark feature, clients are able to request watch events with BOOKMARK type. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session.
• kubeadm no longer removes /etc/cni/net.d as it does not install it. Users should remove files from it manually or rely on the component that created them (#83950, @yastij)
• kubeadm: enhance certs check-expiration to show the expiration info of related CAs (#83932, @SataQiu)
• Add incoming pods metrics to scheduler queue. (#83577, @liu-cong)
• An end-user may choose to request logs without confirming the identity of the backing kubelet. This feature can be disabled by setting the AllowInsecureBackendProxy feature-gate to false. (#83419, @deads2k)
• Switched intstr.Type to sized integer to follow API guidelines and improve compatibility with proto libraries (#83956, @liggitt)
• Fix handling tombstones in pod-disruption-budged controller. (#83951, @zouyee)
• client-go: improved allocation behavior of the delaying workqueue when handling objects with far-future ready times. (#83945, @barkbay)
• Added the crictl Windows binaries as well as the Linux 32bit binary to the release archives (#83944, @saschagrunert)
• Fixed an issue with informers missing an Added event if a recently deleted object was immediately recreated at the same time the informer dropped a watch and relisted. (#83911, @matte21)
• Allow dynamically set glog logging level of kube-scheduler (#83910, @mrkm4ntr)
• clean duplicate GetPodServiceMemberships function (#83902, @gongguan)
• Add information from Lease object corresponding to a given Node to kubectl describe node output (#83899, @wojtek-t)
• Gives the right error message when using kubectl delete a wrong resource. (#83825, @zhouya0)
• The userspace mode of kube-proxy no longer confusingly logs messages about deleting endpoints that it is actually adding. (#83644, @danwinship)
• Add latency and request count metrics for scheduler framework. (#83569, @liu-cong)
• ETCD version monitor metrics are now marked as with the ALPHA stability level. (#83283, @RainbowMango)
• Significant kube-proxy performance improvements when using Endpoint Slices at scale. (#83206, @robscott)
• Upgrade default etcd server version to 3.3.17 (#83804, @jpbetz)

Documentation

• Expand scheduler priority functions and scheduling framework plugins' node score range to [0, 100]. action required. Note: this change is internal and does not affect extender and RequestedToCapacityRatio custom priority, which are still expected to provide a [0, 10] range. (#83522, @draveness)
• action required: kubeadm: when adding extra apiserver authorization-modes, the defaults "Node,RBAC" are no longer prepended in the resulting static Pod manifests and a full override is allowed. (#82616, @ghouscht)
• ACTION REQUIRED: kubeadm: properly enable kubelet client certificate rotation on primary control-plane nodes, created using "kubeadm init". A side effect of this change is that for external CA users, kubeadm now requires "bootstrap-kubelet.conf" instead of "kubelet.conf" during "kubeadm init" and its phases. (#83339, @neolit123)
• Action Required: kubeadm.k8s.io/v1beta1 has been deprecated, you should update your config to use newer non-deprecated API versions. (#83276, @Klaven)

• [migration phase 1] PodFitsHostPorts as filter plugin (#83659, @wgliang)
• [migration phase 1] PodFitsResources as framework plugin (#83650, @wgliang)
• Fixed attachment of AWS volumes that have just been detached. (#83567, @jsafrane)
• [migration phase 1] PodMatchNodeSelector/NodAffinity as filter plugin (#83660, @wgliang)
• Upgrade to etcd client 3.3.17 to fix bug where etcd client does not parse IPv6 addresses correctly when members are joining, and to fix bug where failover on multi-member etcd cluster fails certificate check on DNS mismatch (#83801, @jpbetz)
• Fixed panic when accessing CustomResources of a CRD with x-kubernetes-int-or-string. (#83787, @sttts)
• Change pod_preemption_victims metric from Gauge to Histogram. (#83603, @Tabrizian)
• Expose SharedInformerFactory in the framework handle (#83663, @draveness)
• Add more tracing steps in generic_scheduler (#83539, @wgliang)
• [migration phase 1] PodFitsHost as filter plugin (#83662, @wgliang)
• The topology manager aligns resources for pods of all QoS classes with respect to NUMA locality, not just Guaranteed QoS pods. (#83492, @ConnorDoyle)
• Fix unsafe JSON construction in a number of locations in the codebase (#81158, @zouyee)
• Fixed a bug in the single-numa-node policy of the TopologyManager. Previously, pods that only requested CPU resources and did not request any third-party devices would fail to launch with a TopologyAffinity error. Now they will launch successfully. (#83697, @klueska)
• Add per-pod scheduling metrics across 1 or more schedule attempts. (#83674, @liu-cong)
• Fix validation message to mention bytes, not characters. (#80880, @DirectXMan12)
• external facing APIs in pluginregistration and deviceplugin packages are now available under k8s.io/kubelet/pkg/apis/ (#83551, @dims)
• Fix error where metrics related to dynamic kubelet config isn't registered (#83184, @odinuge)
• The VolumeSubpathEnvExpansion feature is graduating to GA. The VolumeSubpathEnvExpansion feature gate is unconditionally enabled, and will be removed in v1.19. (#82578, @kevtaylor)
• Openstack: Do not delete managed LB in case of security group reconciliation errors (#82264, @multi-io)
• The mutating and validating admission webhook plugins now read configuration from the admissionregistration.k8s.io/v1 API. (#80883, @liggitt)
• kubeadm: implemented structured output of 'kubeadm token list' in JSON, YAML, Go template and JsonPath formats (#78764, @bart0sh)
• kube-proxy: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#82927, @obitech)
• Add "podInitialBackoffDurationSeconds" and "podMaxBackoffDurationSeconds" to the scheduler config API (#81263, @draveness)
• Authentication token cache size is increased (from 4k to 32k) to support clusters with many nodes or many namespaces with active service accounts. (#83643, @lavalamp)
• Bumps the minimum version of Go required for building Kubernetes to 1.12.4. (#83596, @jktomer)
• kube-proxy iptables probabilities are now more granular and will result in better distribution beyond 319 endpoints. (#83599, @robscott)
• Fixed the bug that deleted services were processed by EndpointSliceController repeatedly even their cleanup were successful. (#82996, @tnqn)
• If a bad flag is supplied to a kubectl command, only a tip to run --help is printed, instead of the usage menu. Usage menu is printed upon running kubectl command --help. (#82423, @sallyom)
• If container fails because ContainerCannotRun, do not utilize the FallbackToLogsOnError TerminationMessagePolicy, as it masks more useful logs. (#81280, @yqwang-ms)
• Fixed cleanup of raw block devices after kubelet restart. (#83451, @jsafrane)
• Commands like kubectl apply now return errors if schema-invalid annotations are specified, rather than silently dropping the entire annotations section. (#83552, @liggitt)
• Expose kubernetes client in the scheduling framework handle. (#82432, @draveness)
• kubeadm: fix wrong default value for the "upgrade node --certificate-renewal" flag. (#83528, @neolit123)
• IP validates if a string is a valid IP address (#83104, @zouyee)
• The --certificate-authority flag now correctly overrides existing skip TLS or CA data settings in the kubeconfig file (#83547, @liggitt)
• hyperkube will now be available in a new github repository and will not be included in the kubernetes release from 1.17 onwards (#83454, @dims)
• more complete and accurate logging of stack backtraces in E2E failures (#82176, @pohly)
• Kubeadm: add support for 127.0.0.1 as advertise address. kubeadm will automatically replace this value with matching global unicast IP address on the loopback interface. (#83475, @fabriziopandini)
• Rename PluginContext to CycleState in the scheduling framework (#83430, @draveness)
• kube-scheduler: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83030, @obitech)
• Significant kube-proxy performance improvements for non UDP ports. (#83208, @robscott)
• Fixes a flaw (CVE-2019-11253) in json/yaml decoding where large or malformed documents could consume excessive server resources. Request bodies for normal API requests (create/delete/update/patch operations of regular resources) are now limited to 3MB. (#83261, @liggitt)

Documentation

• The deprecated feature gates GCERegionalPersistentDisk, EnableAggregatedDiscoveryTimeout and PersistentLocalVolumes are now unconditionally enabled and can no longer be specified in component invocations. (#82472, @draveness)
• ACTION REQUIRED: (#81668, @darshanime)
  • Deprecate the default service IP CIDR. The previous default was 10.0.0.0/24 which will be removed in 6 months/2 releases. Cluster admins must specify their own desired value, by using --service-cluster-ip-range on kube-apiserver.
• Remove deprecated "include-uninitialized" flag. action required (#80337, @draveness)

• Bump version of event-exporter to 0.3.1, to switch it to protobuf. (#83396, @loburm)
• kubeadm: use the --service-cluster-ip-range flag to init or use the ServiceSubnet field in the kubeadm config to pass a comma separated list of Service CIDRs. (#82473, @Arvinderpal)
• Remove MaxPriority in the scheduler API, please use MaxNodeScore or MaxExtenderPriority instead. (#83386, @draveness)
• Fixes a goroutine leak in kube-apiserver when a request times out. (#83333, @lavalamp)
• Some scheduler extender API fields are moved from pkg/scheduler/api to pkg/scheduler/apis/extender/v1. (#83262, @Huang-Wei)
• Fix aggressive VM calls for Azure VMSS (#83102, @feiskyer)
• Update Azure load balancer to prevent orphaned public IP addresses (#82890, @chewong)
• Use online nodes instead of possible nodes when discovering available NUMA nodes (#83196, @zouyee)
• Fix typos in certificates.k8s.io/v1beta1 KeyUsage constant names: UsageContentCommittment becomes UsageContentCommitment and UsageNetscapSGC becomes UsageNetscapeSGC. (#82511, @abursavich)
• Fixes the bug in informer-gen that it produces incorrect code if a type has nonNamespaced tag set. (#80458, @tatsuhiro-t)
• Update to go 1.12.10 (#83139, @cblecker)
• Update crictl to v1.16.1. (#82856, @Random-Liu)
• Reduces the number of calls made to the Azure API when requesting the instance view of a virtual machine scale set node. (#82496, @hasheddan)
• Consolidate ScoreWithNormalizePlugin into the ScorePlugin interface (#83042, @draveness)
• On AWS nodes with multiple network interfaces, kubelet should now more reliably report the same primary node IP. (#80747, @danwinship)
• Fixes kube-proxy bug accessing self nodeip:port on windows (#83027, @liggitt)
• Resolves bottleneck in internal API server communication that can cause increased goroutines and degrade API Server performance (#80465, @answer1991)
• The deprecated mondo kubernetes-test tarball is no longer built. Users running Kubernetes e2e tests should use the kubernetes-test-portable and kubernetes-test-{OS}-{ARCH} tarballs instead. (#83093, @ixdy)
• Improved performance of kube-proxy with EndpointSlice enabled with more efficient sorting. (#83035, @robscott)
• New APIs to allow adding/removing pods from pre-calculated prefilter state in the scheduling framework (#82912, @ahg-g)
• Conformance tests may now include disruptive tests. If you are running tests against a live cluster, consider skipping those tests tagged as Disruptive to avoid non-test workloads being impacted. Be aware, skipping any conformance tests (even disruptive ones) will make the results ineligible for consideration for the CNCF Certified Kubernetes program. (#82664, @johnSchnake)
• Resolves regression generating informers for packages whose names contain . characters (#82410, @nikhita)
• Added metrics 'authentication_latency_seconds' that can be used to understand the latency of authentication. (#82409, @RainbowMango)
• kube-dns add-on: (#82347, @pjbgf)
  • • All containers are now being executed under more restrictive privileges.
  • • Most of the containers now run as non-root user and has the root filesystem set as read-only.
  • • The remaining container running as root only has the minimum Linux capabilities it requires to run.
  • • Privilege escalation has been disabled for all containers.
• k8s dockerconfigjson secrets are now compatible with docker config desktop authentication credentials files (#82148, @bbourbie)
• Use ipv4 in wincat port forward. (#83036, @liyanhui1228)
• Added Clone method to the scheduling framework's PluginContext and ContextData. (#82951, @ahg-g)
• Bump metrics-server to v0.3.5 (#83015, @olagacek)
• dashboard: disable the dashboard Deployment on non-Linux nodes. This step is required to support Windows worker nodes. (#82975, @wawa0210)
• Fix possible fd leak and closing of dirs when using openstack (#82873, @odinuge)
• PersistentVolumeLabel admission plugin, responsible for labeling PersistentVolumes with topology labels, now does not overwrite existing labels on PVs that were dynamically provisioned. It trusts the dynamic provisioning that it provided the correct labels to the PersistentVolume, saving one potentially expensive cloud API call. PersistentVolumes created manually by users are labelled by the admission plugin in the same way as before. (#82830, @jsafrane)
• Fixes a panic in kube-controller-manager cleaning up bootstrap tokens (#82887, @tedyu)
• Fixed a scheduler panic when using PodAffinity. (#82841, @Huang-Wei)
• Modified the scheduling framework's Filter API. (#82842, @ahg-g)
• Fix panic in kubelet when running IPv4/IPv6 dual-stack mode with a CNI plugin (#82508, @aanm)
• Kubernetes no longer monitors firewalld. On systems using firewalld for firewall (#81517, @danwinship)
  • maintenance, kube-proxy will take slightly longer to recover from disruptive
  • firewalld operations that delete kube-proxy's iptables rules.
• Added cloud operation count metrics to azure cloud controller manager. (#82574, @kkmsft)
• Report non-confusing error for negative storage size in PVC spec. (#82759, @sttts)
• When registering with a 1.17+ API server, MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects can now request that only v1 AdmissionReview requests be sent to them. Previously, webhooks were required to support receiving v1beta1 AdmissionReview requests as well for compatibility with API servers <= 1.15. (#82707, @liggitt) * When registering with a 1.17+ API server, a CustomResourceDefinition conversion webhook can now request that only v1 ConversionReview requests be sent to them. Previously, conversion webhooks were required to support receiving v1beta1 ConversionReview requests as well for compatibility with API servers <= 1.15.
• Resolves issue with /readyz and /livez not including etcd and kms health checks (#82713, @logicalhan)
• fix: azure disk detach failure if node not exists (#82640, @andyzhangx)
• Single static pod files and pod files from http endpoints cannot be larger than 10 MB. HTTP probe payloads are now truncated to 10KB. (#82669, @rphillips)
• Restores compatibility with <=1.15.x custom resources by not publishing OpenAPI for non-structural custom resource definitions (#82653, @liggitt)
• Take the context as the first argument of Schedule. (#82119, @wgliang)
• Fixes regression in logging spurious stack traces when proxied connections are closed by the backend (#82588, @liggitt)
• Correct a reference to a not/no longer used kustomize subcommand in the documentation (#82535, @demobox)
• Limit the body length of exec readiness/liveness probes. remote CRIs and Docker shim read a max of 16MB output of which the exec probe itself inspects 10kb. (#82514, @dims)
• fixed an issue that the correct PluginConfig.Args is not passed to the corresponding PluginFactory in kube-scheduler when multiple PluginConfig items are defined. (#82483, @everpeace)
• Adding TerminationGracePeriodSeconds to the test framework API (#82170, @vivekbagade)
• /test/e2e/framework: Adds a flag "non-blocking-taints" which allows tests to run in environments with tainted nodes. String value should be a comma-separated list. (#81043, @johnSchnake)