For offers host on a different controller, the consuming side needs to poll the controller to keep up to date with any API address changes. However, we weren't stopping the poll operation when all consuming apps were removed. And if the external controller was killed, the polling would fill the logs with errors.

This PR adds ref counting so that we track if an external controller record is still needed. Once all SAAS applications are removed, the external controller record is also deleted. A watcher in the polling worker causes the worker stop. To enabled the ref counting, the SAAS application gets a new attribute set to the external controller UUID.

An upgrade step is added to set the new external controller UUID attribute on any relevant SAAS applications, and also remove any orphaned external controller records.

In the polling worker, the external controller connection is now done asynchronously to avoid blocking the worker from shutting down if the external controller is removed while the connection is being made. Error handling is also fixed to properly account for not found etc.

Another fix is that getting the api addresses for the external controller was incorrectly including an empty address if there was no public DNS name recorded.

QA steps

bootstrap two earlier 2.9 controllers with logging for juju.worker.externalcontrollerupdater set to DEBUG
create a cross controller relation
upgrade the controllers
use mongo shell to check that the source-countroller-uuid is set for the remote application record on the consuming side
on the consuming side, run juju remove-saas to delete the consuming application
check log for stopping watcher for external controller ...
run juju consume to consume the offer again
check log for starting watcher for external controller ...

Bug reference

https://bugs.launchpad.net/juju/+bug/1958446