Proxy + TLS Discussion #1173

SpencerMalone · 2024-09-19T18:05:19Z

SpencerMalone
Sep 19, 2024

Per 8a9b709, I was gonna start a thread on use of the proxy + TLS.

Some intro bits:

I don't care about client certs or peer validation so I won't be using that
My backends are AWS elasticache instances with TLS required on all interactions

This will be a bit slow to update, as the initial POC is quick for us, but a production rollout is much dicier and will be slower to release.

The build process:

./configure --enable-proxy --enable-proxy-tls --enable-tls seemed to work well for me, although docs were not present so it took some spelunking to find --enable-proxy-tls as someone who hasn't done a lot of custom building of the project
With openssl-devel 1.1.0 there was some pain...

proxy_tls.c: In function ‘mcp_tls_init’:
proxy_tls.c:30:33: warning: implicit declaration of function ‘TLS_client_method’; did you mean ‘DTLS_client_method’? [-Wimplicit-function-declaration]
     SSL_CTX *tctx = SSL_CTX_new(TLS_client_method());
                                 ^~~~~~~~~~~~~~~~~
                                 DTLS_client_method
proxy_tls.c:30:33: warning: passing argument 1 of ‘SSL_CTX_new’ makes pointer from integer without a cast [-Wint-conversion]
In file included from proxy_tls.c:6:0:
/usr/include/openssl/ssl.h:2087:10: note: expected ‘const SSL_METHOD * {aka const struct ssl_method_st *}’ but argument is of type ‘int’
 SSL_CTX *SSL_CTX_new(const SSL_METHOD *meth);
          ^~~~~~~~~~~
proxy_tls.c:36:5: warning: implicit declaration of function ‘SSL_CTX_set_min_proto_version’; did you mean ‘SSL_CTX_set_ssl_version’? [-Wimplicit-function-declaration]
     SSL_CTX_set_min_proto_version(tctx, TLS1_3_VERSION);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     SSL_CTX_set_ssl_version
proxy_tls.c:36:41: error: ‘TLS1_3_VERSION’ undeclared (first use in this function); did you mean ‘TLS1_1_VERSION’?
     SSL_CTX_set_min_proto_version(tctx, TLS1_3_VERSION);
                                         ^~~~~~~~~~~~~~
                                         TLS1_1_VERSION

openssl11-devel (1.1.1+) fixed that and then it built correctly, but I think those are still things you could work around using ifdefs if you wanted. Either way, documenting that requirement would be nice

Configuration:
The route lib was a bit spooky having not really looked in there before, but the change seemed reasonable, I added...

mcp.init_tls()
mcp.backend_use_tls(true)

inside the first lines of https://github.com/memcached/memcached-proxylibs/blob/main/lib/routelib/routelib.lua#L709. I still kind of wish that route lib was distributed in the binary in some way, it's a somewhat clunky file to be editing to add TLS, but also I understand.

Initial Execution
At this point, it just worked. TLS seemed to happily work on our TLS required nodes, everything seemed hunky dory. My next steps are more about internal roll out and testing, but as we start to flex this at scale I'll reach out if anything gets weird? I'm also not sure how "prod-ready" y'all feel about the TLS+Proxy, or if this is a thing I should totally be avoiding.

dormando · 2024-09-19T18:28:55Z

dormando
Sep 19, 2024
Maintainer

Thanks! Good notes.

RE the compile issue, it's supposed to fail to configure if your OpenSSL is too old. Were there any warnings in your configure output? 1.1.0 is EOL'ed (in fact 1.1.1 is EOL'ed but more recently so, but I was being conservative in bumping the requirement)
routelib modifications: Correct I have not added the backend TLS as options here, however, you should be able to move those two lines to the top of your config file and avoid modifying routelib entirely. I'll look into adding some native switch for this soon.
Agreed routelib needs to be in the binary, and I've been trying to get time to make this happen for over a month but there's a lot going on. It should happen very soon.

RE: the state of backend TLS support... To be completely honest the user who lit a fire about it did not end up deploying it (their priorities changed). So while I've done my own testing on it there is limited production level testing for the TLS backends. Please let me know if you run into trouble with it. TLS frontend is fine though. For connecting to the proxy via TLS it is the same exact code as connecting to TLS on a normal memcached daemon.

A few things are going on in parallel right now as I try to polish things:

We're redoing the docs site and cleaning up all the proxy docs. Beta site is here: https://memcached.github.io/ (will be docs.memcached.org later)
I'm reworking my burn-in test suite, which will let me add CI burn-in testing for all the TLS stuff in the next week or two. This should help me raise my confidence level on the backend TLS.
Planning on adding a bunch more built-in routes to routelib in the short term, narrowing the feature gap with mcrouter.

8 replies

dormando Sep 19, 2024
Maintainer

I'll work on the TLS configure issue... see: https://github.com/memcached/memcached/blob/master/configure.ac#L511 (looks like I missed updating the 1.1.0 text there as well)

Any chance you could provide more info on the distro/version you were compiling on?

Also what version of memcached were you building? Don't think you said anywhere

SpencerMalone Sep 19, 2024
Author

Ah, I went for memcached 1.6.31 for my proxy (although my backends are much further behind) since I knew I was already gonna be bleeding edge here. Distro was amazon linux 2 (centos 7.x), my SSL versions were... 1:1.0.2k-24.amzn2.0.13 (failed) and then 1:1.1.1g-12.amzn2.0.23

dormando Sep 19, 2024
Maintainer

Thought I was finally rid of centos after the june 30th EOL. I guess not :) Looks like amazon linux 2 is good until june 2025.

I'll try to address all of this asap but it's probably going to be next week.

SpencerMalone Sep 19, 2024
Author

No rush, I don't think there's any urgency on these fixes!

SpencerMalone Sep 19, 2024
Author

If we hit something more serious I'll let you know for sure though

SpencerMalone · 2024-10-11T20:19:24Z

SpencerMalone
Oct 11, 2024
Author

It'd been a while, so I'll share some updates and probably close this thread down soon:

We went to production with this and the TLS backend configuration and it worked
We're still waiting one more week before a wider rollout just to try and shake out for memory leaks, but so far looking really good
This enabled us to collapse ~420k+ connections into ~5k connections, which was the primary benefit we wanted. We still only have this released to a subset of production, at full load we're probably gonna be on the scale of ~9 million+ connections reduced to 110k conns?
With very little CPU and memory this rapidly outscaled most clients we could throw at it
We run it as a sidecar in kubernetes, so currently we have somewhere around 2000+ of these running each with very little CPU or memory
For legacy reasons we use the tagging functionality and expose multiple ports on our proxy
I think our peak is a couple hundred thousand ops/s run through a single proxy? I believe it could have scaled higher, we just didn't have a need for any one proxy to scale that high

My biggest nice to haves at this point (and I'll probably hack something for myself short term) would be:

The ability to specify exposed ports/tags in the lua configuration, as it's minorly error prone to sync the commands with sections of the lua. I think this was the only "error" we had in releasing this.
Some way to unit test configurations would be nice, although this is honestly low priority because it's not exactly hard to just run the thing for a few seconds.

1 reply

dormando Oct 11, 2024
Maintainer

Thanks so much for the update! It's hard to get this kind of feedback, so it's much appreciated.

RE: Concerns about memory usage, have you enabled some of the memory/active request limiters? There are a few unfortunately. (buffer memory limit, active request limit, and queue depth limit, for various scenarios).

RE: nice to haves...

I would've done this if it were easy. :( It could be a fair bit of hacking. There might be a quick path to at least having the lua toss an error if the server isn't listening on a specified tag. Open to feature sponsorship for this or other future requests, of course :)
While unit-testing specific sub functions independent of the proxy is impossible, it's certainly possible to write integration tests. routelib itself has them here: https://github.com/memcached/memcached-proxylibs/tree/main/lib/routelib/t - using a small library that manages the proxy and simplifies writing tests using mocked backends.

It requires installing lua and some modules via luarocks or similar. I run them with a shell script:

#!/bin/bash

MC_PATH="$HOME/p/code/memcached/" LUA_PATH="$HOME/.luarocks/share/lua/5.4/?.lua;;" LUA_CPATH="$HOME/.luarocks/lib/lua/5.4/?.so;;" lua ./t/$1.lua -o TAP

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy + TLS Discussion #1173

{{title}}

Replies: 2 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Proxy + TLS Discussion #1173

SpencerMalone Sep 19, 2024

Replies: 2 comments · 9 replies

dormando Sep 19, 2024 Maintainer

dormando Sep 19, 2024 Maintainer

SpencerMalone Sep 19, 2024 Author

dormando Sep 19, 2024 Maintainer

SpencerMalone Sep 19, 2024 Author

SpencerMalone Sep 19, 2024 Author

SpencerMalone Oct 11, 2024 Author

dormando Oct 11, 2024 Maintainer

SpencerMalone
Sep 19, 2024

Replies: 2 comments 9 replies

dormando
Sep 19, 2024
Maintainer

dormando Sep 19, 2024
Maintainer

SpencerMalone Sep 19, 2024
Author

dormando Sep 19, 2024
Maintainer

SpencerMalone Sep 19, 2024
Author

SpencerMalone Sep 19, 2024
Author

SpencerMalone
Oct 11, 2024
Author

dormando Oct 11, 2024
Maintainer