-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/multi tenancy #1923
base: master
Are you sure you want to change the base?
Feature/multi tenancy #1923
Conversation
Hi, @JoelSpeed Are you guys interested in adding this feature to oauth2proxy anytime soon? We plan on using implementation in this PR for our internal use but we have put in extra effort to make it backward compatible & generic as we want to contribute to the community. But if this doesn't look like something that you would want to merge then we can take a private fork & maintain a custom implementation without the overhead of backward compatibility etc. |
We are interested in this feature, but, I'm not getting much time for the project lately. I went travelling for a few months this year and am currently mid way through a house move. I expect I'll have time to come to this sometime mid to late January once things have settled down in my life a little bit. Apologies for the delays folks, but please do remember, this is a passion project for me, I don't get time day to day for it |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should mix tenants
and providers
up like this, right now, we have a single provider
configuration, so the intention from previous work was that you would then have a list of providers
, which is kind of what you've done, but you're calling it tenants
. I'd rather we stick to a single term than mix these up.
I think this has the ground work needed, but how does it relate to existing PRs that are open? Is this an extension of those? Is it intended to replace them?
pkg/apis/options/alpha_options.go
Outdated
@@ -42,7 +47,9 @@ type AlphaOptions struct { | |||
MetricsServer Server `json:"metricsServer,omitempty"` | |||
|
|||
// Providers is used to configure multiple providers. | |||
Providers Providers `json:"providers,omitempty"` | |||
Providers provideropts.Providers `json:"providers,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why has this been moved into a new package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otherwise get import cycles,
package github.com/oauth2-proxy/oauth2-proxy/v7
imports github.com/oauth2-proxy/oauth2-proxy/v7/pkg/apis/options
imports github.com/oauth2-proxy/oauth2-proxy/v7/tenant
imports github.com/oauth2-proxy/oauth2-proxy/v7/tenant/configloader
imports github.com/oauth2-proxy/oauth2-proxy/v7/pkg/apis/options: import cycle not allowed
basically ConfigLoader (a TenantLoader implementation) config depends on the provider config, which is part of pkg/apis/options
package, and the options.Options depend on the TenantLoader config which depends upon the ConfigLoader config.
This is resolved by moving the provider options to a new package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why has this been moved into a new package?
resolved. moved to the original location.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why has this been moved into a new package?
resolved. moved to the original location.
pkg/apis/options/alpha_options.go
Outdated
Providers Providers `json:"providers,omitempty"` | ||
Providers provideropts.Providers `json:"providers,omitempty"` | ||
|
||
TenantLoader tenant.LoaderConfiguration `flag:"tenant-loader" cfg:"tenant_loader"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This struct is meant to represent the configuration, please don't add runtime bits to it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there are configurations, not runtime structures,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved. removed the runtime bits from options.
pkg/apis/options/legacy_options.go
Outdated
// in case of legacy options, we initilize single tenant implementation of tenant loader | ||
l.Options.TenantLoader = tenant.LoaderConfiguration{ | ||
Type: "single", | ||
Single: &single.Configuration{ | ||
Provider: &providers[0], | ||
}, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This kind of initialisation shouldn't be happening within the options part of the project
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved, removed the initialization logic away from the options
package.
@@ -1,4 +1,4 @@ | |||
package options | |||
package loginurlopts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why has this moved?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otherwise get import cycles,
package github.com/oauth2-proxy/oauth2-proxy/v7 imports github.com/oauth2-proxy/oauth2-proxy/v7/pkg/apis/options imports github.com/oauth2-proxy/oauth2-proxy/v7/tenant imports github.com/oauth2-proxy/oauth2-proxy/v7/tenant/configloader imports github.com/oauth2-proxy/oauth2-proxy/v7/pkg/apis/options: import cycle not allowed
basically ConfigLoader (a TenantLoader implementation) config depends on the provider config, which is part of
pkg/apis/options
package, and the options.Options depend on the TenantLoader config which depends upon the ConfigLoader config.This is resolved by moving the provider options to a new package.
same thing as above, since provider options depends on loginurlopts,
we can keep the provider opts and loginurlopts in the same package though,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved, moved the file/s back to its original location.
pkg/apis/options/options.go
Outdated
@@ -40,6 +42,9 @@ type Options struct { | |||
|
|||
// Not used in the legacy config, name not allowed to match an external key (upstreams) | |||
// TODO(JoelSpeed): Rename when legacy config is removed | |||
|
|||
TenantLoader tenant.LoaderConfiguration `cfg:",internal"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please don't add runtime information to structs that are intended for configuration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is the configuration for tenantLoader, not the runtime construct,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved, remved the runtime bits away from the options package.
pkg/app/pagewriter/pagewriter.go
Outdated
@@ -3,14 +3,16 @@ package pagewriter | |||
import ( | |||
"fmt" | |||
"net/http" | |||
|
|||
"github.com/oauth2-proxy/oauth2-proxy/v7/tenant/types" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need another concept, a tenant? Why not extend the provider interface?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
responded here,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the tenant struct and loader interfaces have been removed. Only a concep of tenant-id remains now. We extract tenant-id from the incoming request. We could have replaced tenant-id as well and only captured provider-id from the request. But in order to make the whole module multi-tenant, we must have the tenant-concept in some form. Since there are other components besides providers that need to be modified to make them multi-tenant. For example, basicAuth validator, email domain validator and SkipJwtBearerTokens option.
// Provider based session refreshing | ||
RefreshSession func(context.Context, *sessionsapi.SessionState) (bool, error) | ||
|
||
// Provider based session validation. | ||
// If the sesssion is older than `RefreshPeriod` but the provider doesn't | ||
// refresh it, we must re-validate using this validation. | ||
ValidateSession func(context.Context, *sessionsapi.SessionState) bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a good idea 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would you elaborate please.
I removed this since now we're loading tenant/provider from request context and calling the provider's RefreshSession
and ValidateSession
methods.
tenant/config.go
Outdated
@@ -0,0 +1,12 @@ | |||
package tenant |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this package not under pkg
like the rest of the code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will move to under pkg/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved, moved the new packages under the pkg/
folder
tenant/configloader/config.go
Outdated
Rules []*RuleConfig | ||
} | ||
|
||
// structure that is loaded from the tenant's config file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh wait a minute, so, once you have your provider configured, this adds a new dimension of tenants within the provider? This is very keycloak esque and doesn't translate well to other providers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rules are what match an incoming request to a tenant/provider. The ConfigLoader configures all the providers and keeps them in memory. Then in a middleware (pkg/middleware/tenantLoader.go
), the request is matched to a tenant and then that tenant is injeted into the request context. The rules are there to do the matching. We execute rules one by one untill we find a match.
A rule (type RuleConfig struct
) has following structure.
type RuleSource string
const (
SourceHost RuleSource = "host"
SourcePath RuleSource = "path"
SourceQueryParams RuleSource = "query"
SourceHeader RuleSource = "header"
)
type RuleConfig struct {
Source RuleSource // which part of the HTTP request contains the tenant id
Expr string // the regex expression to match and extract tenant id from the source
CaptureGroup int // capture group or sub-match that is actually the tenant id
QueryParam string // query parameter in case source is 'query'
Header string // header key in case source is 'header'
}
I think these rules are pretty generic and are not just related to keycloak. We just have to have tenant-id somewhere in the incoming request. That location is specified in the Source
field in a rule. Right now, host, path, header, and query are supported but we can expand to include cookies as well.
tenant/configloader/loader.go
Outdated
func New(conf *Configuration) (*loader, error) { | ||
tntsConf := &TenantsConfig{} | ||
|
||
err := loadConfigFromFile(conf.TenantsFile, tntsConf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sharding of config files is just going to make our configuration story harder IMO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will merge the config files,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved, only once config file for now.
thanks for the comment,
so in a middleware, we can use TenantMatcher to get the tenantId out from http.Request, and then ProviderLoader to load the provider and inject it into request Context, so that it can be used everywhere where it is needed. What are your thoughts ?
I can't find any open PRs that implement multi-tenancy. #1657 will have conflicts with this PR, but I feel they won't be huge. |
47deebb
to
14e34ba
Compare
Hi @JoelSpeed, I have updated the PR to incorporate your comments. Following changes were made
Please let me know your comments. |
Hello @salmanazmat666 and @JoelSpeed, I am very interesting in this feature, therefore to participate in the effort I opend two PR on the @salmanazmat666 repository.
Hope this can still be integrated. |
Hey @JoelSpeed, We have added more features into the PR,
You comments are highly appreciated, please have a look at the changes and let me know what you think. Since it's a large PR we can have a call to discuss to design. Regards, |
Tremendous Job @salmanazmat666 . Do you have any sample config file or instructions to run these changes? For eg.
|
Hi @adarshsingh2,
|
70fdbbb
to
a9052bc
Compare
Hi @JoelSpeed , it would be great if you could provide feedback on the PR so that we can merge it. It been a bit difficult for us to keep this PR updated. Lets finalize a way forward if something else needs to be done. We want to merge it by end of May latest. There has been a lot of interest by community in this feature. But unfortunately due to work pressure, it won't be possible for us after May to keep this one updated & in sync with our private fork. |
d92e058
to
ba59cbe
Compare
9d8dcb2
to
d053744
Compare
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
@salmanazmat666 happy to see you back at it |
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
Signed-off-by: Salman Azmat <[email protected]>
@tuunit I have updated the PR and following are the main things I did,
Please have a look at the changes and let me know if you want any other change. Regards, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll try to do a full review again soon and do some local testing as well.
contrib/local-environment/Makefile
Outdated
@@ -14,6 +14,14 @@ alpha-config-up: | |||
alpha-config-%: | |||
docker compose -f docker-compose.yaml -f docker-compose-alpha-config.yaml $* | |||
|
|||
.PHONY: multi-tenancy-up |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we rename this to:
.PHONY: multi-tenancy-up | |
.PHONY: multi-provider-up |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and rename all files (replace multi-tenancy with multi-provider)
Thanks for putting all the work in 🥳 |
Signed-off-by: Salman Azmat <[email protected]>
Hey @tuunit , did you get a chance to review the PR ? |
@salmanazmat666 it might be worth making progress by merging in the base branch? |
This pull request has been inactive for 60 days. If the pull request is still relevant please comment to re-activate the pull request. If no action is taken within 7 days, the pull request will be marked closed. |
Hi @salmanazmat666. I'm interested in this feature too, and thank you for all your work so far! I've given it a go, and wanted to check my understanding.
Our use case is to support multiple providers for a single upstream service, all from a single public service URL. It'd be good (at least from our perspective) if the provider can be specified as a query parameter to I've started experimenting with adapting what you've got so far to implement the suggestion above, but don't want to go off in another direction to where you're headed. If you'd like to catch up on this that'd be great! All the best, ~ Alex |
Signed-off-by: salmanazmat666 <[email protected]>
Signed-off-by: salmanazmat666 <[email protected]>
Thank you for this amazing product and all the incredible work you've done — it's truly impressive! 🙌 Regarding multi-tenancy functionality: quite some time has passed since this idea was discussed, and it would be fantastic to see it implemented. Are there any updates on this topic? |
Description
This PR is only meant to have a discussion around the implementation of multi-tenancy in oauth2 proxy and have comments from the maintainers about if they'll be willing to merge when the feature is fully implemented and tested. I have tested the basic Oauth2 flow with OIDC type provider and it is working as expected.
Motivation and Context
In its current form, oauth2 proxy cannot handle multiple tenants, each with different IdP configurations and requirements. Multi-tenancy is a very common design pattern in SAAS applications and issues have been raised to implement this feature on oauth2 proxy in the past like #1863 and #739. We can be sure that the requirement for this feature will increase in future.
How Has This Been Tested?
This PR is not ready to merge in its current form, and tests are broken right now.
Implementation Details
Following are the key points that'll help understand the implementation
id
or*http.Request
OAuthProxy
struct, now theTenant
struct is loaded in a middleware using theLoader
interface and injected into the request context. Later in the flow, theTenant
can be retrieved from the request context wherever needed.Loader
interface.Loader
allows you to describe a list of rules based on which we can extract tenant id from the request. Rule configuration is as followsloader.LoadById(string)
method or what request we pass toloader.Load(*http.Request)
method. This second implementation allows us to use the legacy config.Loader
interface we can implement the the functionality that loads/removes the tenants dynamically, without actually touching the actual oauth2 proxy logic.Loader
interface, and everything is backwards compatible so legacy config can be used as well without configuring the tenantLoader
. If the legacy config is used, we configure the tenantLoader
interface and initialize it using the second implementation described above, in theLegacyOptions.ToOptions()
method.TODOs
Final Comments
There is still lots of work to make the oauth2 proxy fully multi-tenant but we can take on things one by one which is possible because we are using alpha config which still allows for breaking changes.
This is really a great work by all the contributors and I want to thank you guys for this. Even if this doesn't get approved we'll be forking and implementing the features privately anyway.
Looking forward to hear back from maintainers. @JoelSpeed @NickMeves
Checklist: