Skip to content

Increase limit for object size in streaming serializer #72053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
k8s-ci-robot merged 1 commit into from
Dec 15, 2018
Merged

Increase limit for object size in streaming serializer #72053

k8s-ci-robot merged 1 commit into from
Dec 15, 2018

Conversation

@wojtek-t
Copy link
Member

@wojtek-t wojtek-t commented Dec 14, 2018 •
edited
Loading

Given that apiserver doesn't have validation for object size, and etcd allows for writing objects larger than 1MB (I think 1.5MB is the limit @jpbetz to confirm), we should allow for reading them in the default streaming serializer.

I bumped the limit significantly to also handle situation where we write protobuf to etcd and we watch jsons which are visibly better (it doesn't guarantee it will always work, but it at least leave some slack so that can be possible).

@kubernetes/sig-api-machinery-bugs

Allow for watching objects larger than 1MB given etcd accepts objects of size up to 1.5MB

@wojtek-t wojtek-t added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Dec 14, 2018
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 14, 2018
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 14, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 14, 2018
@k8s-ci-robot k8s-ci-robot requested review from gmarek and ncdc December 14, 2018 11:04
@wojtek-t wojtek-t added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Dec 14, 2018
@jpbetz
Copy link
Contributor

jpbetz commented Dec 14, 2018

xref #57073 where this issue was encountered

@jpbetz
Copy link
Contributor

jpbetz commented Dec 14, 2018

Confirmed the limit to be 1.5MB on 3.1+:

head -c $(((1024+512)*1024)) /dev/urandom | ETCDCTL_API=3 etcdctl put k
Error:  etcdserver: request is too large
head -c $(((1024+512-1)*1024)) /dev/urandom | ETCDCTL_API=3 etcdctl put k
OK

@jpbetz
Copy link
Contributor

jpbetz commented Dec 14, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 14, 2018
@fejta-bot
Copy link

fejta-bot commented Dec 14, 2018

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 5a5111f into kubernetes:master Dec 15, 2018
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jan 17, 2019
This was referenced Jan 17, 2019
Merged
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Jan 17, 2019
@k8s-ci-robot
Merge pull request #73037 from wojtek-t/automated-cherry-pick-of-#720…
0cd9833 
…53-upstream-release-1.13

Automated cherry pick of #72053 upstream release 1.13
k8s-ci-robot added a commit that referenced this pull request Jan 18, 2019
@k8s-ci-robot
Merge pull request #73039 from wojtek-t/automated-cherry-pick-of-#720…
65ecaf0 
…53-upstream-release-1.11

Automated cherry pick of #72053 upstream release 1.11
k8s-ci-robot added a commit that referenced this pull request Jan 22, 2019
@k8s-ci-robot
Merge pull request #73038 from wojtek-t/automated-cherry-pick-of-#720…
9f5c8ee 
…53-upstream-release-1.12

Automated cherry pick of #72053 upstream release 1.12
@roycaihw roycaihw mentioned this pull request Mar 18, 2019
Closed
@wojtek-t wojtek-t deleted the streaming_limit branch July 19, 2019 11:44
@EricMountain EricMountain mentioned this pull request Mar 13, 2020
Merged
@antoineco antoineco mentioned this pull request Feb 12, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gmarek gmarek Awaiting requested review from gmarek

@ncdc ncdc Awaiting requested review from ncdc

Assignees

@jpbetz jpbetz

@lavalamp lavalamp

@liggitt liggitt

@deads2k deads2k

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@wojtek-t @k8s-ci-robot @jpbetz @fejta-bot @lavalamp @liggitt @deads2k