Feat: bearer token authentication support #591

skvark · 2024-04-09T11:12:13Z

All Submissions:

Contributions should target the dev branch. Did you create your branch from dev?
Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

Does your submission pass tests?
Have you installed pre-commit with pip3 install pre-commit and set up hooks with pre-commit install?

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

What

This pull request adds a custom Bearer authentication token provider function for the HTTP clients. Additionally, Bearer token authentication is enabled for the GRPC clients as well.

Why

Currently, it is impossible to use the clients properly with token-based authentication, since the the only way of passing the token is via the static metadata parameter. When the token expires, the whole client needs to be recreated.

Changes

The clients have a new parameter, auth_token_provider. This can be sync or async function, that returns str Bearer token.

For the HTTP clients, this PR adds a new custom BearerAuth class that is based on httpx.Auth class (https://www.python-httpx.org/advanced/authentication/#custom-authentication-schemes). This is passed down in the auth parameter to the httpx library.

The GRPC code in connection.py has been refactored to support the new feature and the code in get_channel and get_async_channel should be simpler now.

Example

There are simple examples in the tests, but here is a proper example with azure-identity library:

import qdrant_client
from azure.identity.aio import DefaultAzureCredential

credential = DefaultAzureCredential(exclude_interactive_browser_credential=False)

async def get_token():
    result = await credential.get_token("your_qdrant_app_scope")
    return result.token

client = qdrant_client.AsyncQdrantClient(
    url=f"your.qdrant.app.address",
    https=True,
    port=443,
    auth_token_provider=get_token,
)

await client.get_collections()

netlify · 2024-04-09T11:12:30Z

✅ Deploy Preview for poetic-froyo-8baba7 ready!

Name	Link
🔨 Latest commit	`574d88a`
🔍 Latest deploy log	https://app.netlify.com/sites/poetic-froyo-8baba7/deploys/661d53a4117a930008da35bc
😎 Deploy Preview	https://deploy-preview-591--poetic-froyo-8baba7.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

joein

@skvark thank you for the contribution!

joein · 2024-04-10T14:52:11Z

qdrant_client/async_qdrant_remote.py

        if self._timeout is not None:
            self._rest_args["timeout"] = self._timeout
+        if self._auth_token_provider is not None:
+            if self.__class__.__name__ == "QdrantRemote" and asyncio.iscoroutinefunction(


I would prefer to avoid magic methods checks and avoid adding it to AsyncQdrantClient as well

Yeah, I didn't like this myself either. It's better now after your changes.

joein · 2024-04-10T14:52:19Z

qdrant_client/auth/bearer_auth.py

@@ -0,0 +1,44 @@
+import asyncio
+import threading


Forgot to clean these after removing some stuff from the auth class.

joein · 2024-04-10T14:53:05Z

qdrant_client/connection.py

        self, continuation: Any, client_call_details: Any, request: Any
    ) -> Any:
-        new_details, new_request_iterator, postprocess = self._fn(
+        new_details, new_request_iterator, postprocess = await self._fn(


nice catch 🙈

joein · 2024-04-10T14:53:42Z

qdrant_client/connection.py


-def header_adder_interceptor(new_metadata: List[Tuple[str, str]]) -> _GenericClientInterceptor:
+def header_adder_interceptor(
+    new_metadata: List[Tuple[str, str]] = [],


lists should not be default values due to their mutability

joein · 2024-04-10T14:55:56Z

qdrant_client/qdrant_remote.py


            ids = [
-                GrpcToRest.convert_point_id(idx) if isinstance(idx, grpc.PointId) else idx
+                (GrpcToRest.convert_point_id(idx) if isinstance(idx, grpc.PointId) else idx)


it seems like you did not use pre-commit or used a different python version

I did use pre-commit but it seems I had too new Python version.

joein · 2024-04-10T15:04:48Z

I've left some comments, I'll address them and merge

skvark · 2024-04-10T18:41:47Z

I've left some comments, I'll address them and merge

Thanks for fixing the issues!

…nnection

joein · 2024-04-11T20:55:47Z

Hey @skvark
I've given this PR another thought

I am somewhat reluctant on bringing new parameters to init (and I think that auth_token_provider actually affects only a small subset of users)

Afaik it's not a common practice to propagate token reissue logic into a database object, and it is usually handled on the client side, isn't it?

skvark · 2024-04-12T07:43:09Z

Hey @skvark I've given this PR another thought

I am somewhat reluctant on bringing new parameters to init (and I think that auth_token_provider actually affects only a small subset of users)

Afaik it's not a common practice to propagate token reissue logic into a database object, and it is usually handled on the client side, isn't it?

Instead of init, we could introduce some new method to the client that registers the authentication token provider.

Here's one example from SQLAlchemy. It uses an event-based approach, but the renewal mechanism is the same: https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens

Weaviate implements the OIDC auth flows themselves (but their DB also validates the tokens so it's a bit different): Weaviate

All databases should sit behind a proper authentication (and authorization) layer in addition to firewalls. Oauth 2.0 bearer tokens are the standard way of implementing that layer. We can use the current QDrant client, but not having a proper way to update the authorization headers on the existing Qdrant database client object leads to unnecessarily complicated code on our side.

joein · 2024-04-15T13:47:53Z

~~@skvark I think I'll try to make it in a separate method~~

joein · 2024-04-16T13:42:14Z

I'm going to merge it now, thank you for the contribution @skvark

skvark · 2024-04-16T14:08:09Z

I'm going to merge it now, thank you for the contribution @skvark

Thanks!

* bearer token authentication provider support * add tests and checks, move auth file to separate dir * fix error message * remove locks * rename var * refactoring: refactor exceptions, fix mypy * fix: regen async * tests: extend token tests to check token updates * new: add warning when auth token provider is used with an insecure connection * fix: propagate auth token to rest client even with prefer_grpc set --------- Co-authored-by: George Panchuk <[email protected]>

skvark added 5 commits April 8, 2024 15:37

bearer token authentication provider support

6421b48

add tests and checks, move auth file to separate dir

964abf7

fix error message

aa05ee0

remove locks

761992a

rename var

010c9bc

joein reviewed Apr 10, 2024

View reviewed changes

joein added 4 commits April 10, 2024 17:05

refactoring: refactor exceptions, fix mypy

633eb4c

fix: regen async

fe3811a

Merge branch 'dev' into feat/bearer-auth-provider

ebe5ad8

tests: extend token tests to check token updates

9ac711e

new: add warning when auth token provider is used with an insecure co…

d2cd03c

…nnection

fix: propagate auth token to rest client even with prefer_grpc set

574d88a

joein merged commit d957ca7 into qdrant:dev Apr 16, 2024

Feat: bearer token authentication support #591

Feat: bearer token authentication support #591

Uh oh!

Conversation

skvark commented Apr 9, 2024

All Submissions:

New Feature Submissions:

Changes to Core Features:

What

Why

Changes

Example

Uh oh!

netlify bot commented Apr 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for poetic-froyo-8baba7 ready!

Uh oh!

joein left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joein commented Apr 10, 2024

Uh oh!

skvark commented Apr 10, 2024

Uh oh!

joein commented Apr 11, 2024

Uh oh!

skvark commented Apr 12, 2024

Uh oh!

joein commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joein commented Apr 16, 2024

Uh oh!

skvark commented Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify bot commented Apr 9, 2024 •

edited

Loading

joein commented Apr 15, 2024 •

edited

Loading