Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code-gen: Remove lowercasing for project imports #68484

Merged
merged 4 commits into from
Oct 5, 2018

Conversation

jsturtevant
Copy link
Contributor

What this PR does / why we need it:
If a project has a uppercase letter in the project import path then the generated paths will end up in different directories when some of the files are lower-cased during generation. This PR fixes the few places where the imports are lower-cased.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/code-generator#20

Special notes for your reviewer:
There was another PR (#63892) that was started but closed with no explanation. It looks like it was missing a few spots area's where the lower casing was missed which are addressed in this PR.

I was not sure where or if there are tests for this section of code. Would be happy to add tests if you can point me in the right direction.

If you run the code-generator/generate-groups.sh against the project here you can see the lower-casing in action. The generated code is split across two directories and the project does not build but with these fixes code-generation completes as expected and the project builds.

Release note:

Code-gen: Remove lowercasing for project imports

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/code-generation sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 10, 2018
@jsturtevant
Copy link
Contributor Author

/assign @caesarxuchao

@yue9944882
Copy link
Member

/cc @kubernetes/code-generator-maintainers

@caesarxuchao
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 11, 2018
@jsturtevant
Copy link
Contributor Author

/retest

@jsturtevant
Copy link
Contributor Author

@caesarxuchao I was able to pass the tests with a few updates.

I started to add an example of the Uppercasing in _examples like the previous PR. The generation works as expected (I haven't commited this yet).

But during that further testing I notice that the verify tests don't actually run the smoke tests in the verify script. From the test output of this PR:

I0913 00:03:18.965] Smoke testing _example by compiling...
I0913 00:03:19.008] Building client-gen
W0913 00:03:19.108] warning: "vendor/k8s.io/code-generator/_example/..." matched no packages
W0913 00:03:19.109] warning: "vendor/k8s.io/code-generator/_example/..." matched no packages

There _example folder looks to have been renamed to _examples at some point. I made an update to build the projects by updating staging/src/k8s.io/code-generator/hack/verify-codegen.sh to build the samples for example:

go build ${SCRIPT_ROOT}/_examples/crd/...

but now I get the error:

./hack/verify-codegen.sh
Generating deepcopy funcs
Generating defaulters
Generating conversions
Generating clientset for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/clientset
Generating listers for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/listers
Generating informers for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/informers
Generating deepcopy funcs
Generating clientset for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/clientset
Generating listers for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/listers
Generating informers for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/informers
Generating deepcopy funcs
Generating clientset for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/clientset
Generating listers for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/listers
Generating informers for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/informers
diffing ./hack/../_examples against freshly generated codegen
./hack/../_examples up to date.
Smoke testing _example by compiling...
./hack/..
../../../../vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/clustertesttype.go:28:2: cannot find package "k8s.io/kubernetes/pkg/apis/autoscaling" in any of:
	/home/jstur/go/src/github.com/kubernetes/kubernetes/vendor/k8s.io/kubernetes/pkg/apis/autoscaling (vendor tree)
	/usr/local/go/src/k8s.io/kubernetes/pkg/apis/autoscaling (from $GOROOT)
	/home/jstur/go/src/k8s.io/kubernetes/pkg/apis/autoscaling (from $GOPATH)

Since the smoke tests have not been running it seems this has been an issue before this PR as you can see on this file in master. Looking at the vendor folder it seems that the all those packages are symlinked back into the main pkg folder. Although none of them have the prefix kubernetes. It seems that there might be another bug related to how those autoscaling imports are being generated.

Thoughts on how to proceed? I could check in the samples folder with the Uppercase as an example, get this merged and closed, then open a separate bug for the re-enabling the smoke tests and fixing the generating of the autoscaling package? What do you think?

@jsturtevant
Copy link
Contributor Author

@caesarxuchao all the tests pass is this good to merge? I can open issues for the comments above separately.

Copy link
Member

@caesarxuchao caesarxuchao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test demonstrating the client-gen works for camelcased path now?

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 24, 2018
@jsturtevant
Copy link
Contributor Author

I added an example with MixedCase. Also needed to update the verify script to build the packages correctly as stated in my previous comment.

Once the example projects were building properly, this uncovered a bug in the code generation for the sub resource scale in the existing crd example:

Smoke testing _example by compiling...
# k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/fake
vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/fake/fake_clustertesttype.go:137:46: not enough arguments in call to testing.NewRootGetSubresourceAction
	have (schema.GroupVersionResource, string)
	want (schema.GroupVersionResource, string, string)

I fixed this in the generator_fake_for_type.go by adding the correct parameters. Everything builds nows. I also tested the updates against my larger project that prompted this bug fix as mentioned in the initial PR.

@jsturtevant
Copy link
Contributor Author

@caesarxuchao the latest commits passed the tests. Ready for review. Thanks!

@caesarxuchao
Copy link
Member

LGTM. Thanks. Can you squash it to meaningful commits?

jsturtevant and others added 2 commits September 26, 2018 10:59
This commit provides a fix for the scenario where a project has an
uppercase letter in the project import path. Prior to this fix
the generated files would end up in different directories with some
of the imports being lower-cased during generation. An example of this would
be a project such as 'github.com/MixedCase/project' would result in
some of the imports with 'github.com/mixedcase/project' causing a broken
build.
The smoke tests were not being run for the example projects.  Re-enabled
the smoke tests by building each of the sample projects.
The code generation for fake types was missing the subresource path
parameter in the template which caused a compile error for the
sample projects using the scale subresource.  Also re-ran the code
generation after applying the fix.
@jsturtevant
Copy link
Contributor Author

@caesarxuchao squashed commits and tests passed. Looks ready to go. Thanks again!

@caesarxuchao
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 26, 2018
@caesarxuchao
Copy link
Member

/assign @lavalamp

for approval

@nikhita
Copy link
Member

nikhita commented Sep 29, 2018

/assign @sttts
/kind cleanup

@k8s-ci-robot k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 29, 2018
go build ./${SCRIPT_ROOT}/_examples/crd/...
go build ./${SCRIPT_ROOT}/_examples/apiserver/...
go build ./${SCRIPT_ROOT}/_examples/MixedCase/...
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@sttts
Copy link
Contributor

sttts commented Oct 5, 2018

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, jsturtevant, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2018
@k8s-ci-robot k8s-ci-robot merged commit 5b9eb1c into kubernetes:master Oct 5, 2018
jsturtevant added a commit to Azure/azure-k8s-metrics-adapter that referenced this pull request Oct 17, 2018
The PR (kubernetes/kubernetes#68484) was merged
into upstream kubernetes.  This commit removes the temporary fix for
lowercasing.  The latest version of the code gen has an update to the
patch client that use a parameter that is not in current version of
Go-Client used by this repo so dropped code gen for patching. Patching
is not being used currently so safe to drop.  May consider dropping
other verbs not used in the client in future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/code-generation cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Imports lowercased
7 participants