The built in pip cache support to setup-python looks pretty easy to use: https://github.com/actions/setup-python/#caching-packages-dependencies

We could probably enable it with a single variable giving the value for cache-dependency-path?