Download tools from API server #608
Conversation
| client := utils.GetHTTPClient(verify) | ||
| resp, err := client.Get(tools.URL) | ||
| if err != nil { | ||
| h.sendError(w, http.StatusBadRequest, fmt.Sprintf("failed to get %q: %v", err.Error())) |
There was a problem hiding this comment.
I think you forgot an argument? Sprintf needs 2, (%q and %v) but is given 1
|
Would it make sense to make use of utils/filestorage (and the implementation in juju/state/backups.go)? It's the package I added expressly to hide the underlying storage implementation, which seems to me like the same thing you are trying to accomplish. |
I don't think so. This PR is about hiding the fact that there's external storage altogether from the clients (i.e. cloud-init script). Right now, the cloud-init scripts go and fetch tools directly from provider storage (e.g. an S3 bucket on AWS). The end goal is to have tools stored in GridFS, replicated across each of the state servers. We'll still need a way for the cloud-init script to fetch them out of there, hence this HTTP endpoint. Introducing it early means we can easily switch out the backend later. |
|
Makes sense. |
|
Could you add 2014 to the copyright headers that don't have it please? |
|
Thanks for updating the sendError func. @davecheney is my review mentor, so he'll need to take a look too. |
Copyright law (in the US at least) only requires the first year of publication. In Australia it's automatically given; not sure what it is in the UK. I'd prefer we just used US-style unless given some other guidance by IP legal. |
| @@ -194,7 +194,7 @@ func AddAptCommands( | |||
| c.SetAptUpgrade(addUpgradeScripts) | |||
| c.SetAptGetWrapper("eatmydata") | |||
|
|
|||
| c.AddPackage("curl") | |||
| c.AddPackage("aria2") | |||
There was a problem hiding this comment.
why not use wget as i suggested to Ian a week ago? With the -c option it'll keep retrying until it gets the whole thing.
There was a problem hiding this comment.
That's not the point of using aria2. The point is to support failing over to alternative URLs in case one of the API servers becomes unresponsive.
|
LGTM. |
axw
force-pushed
the
cloudinit-download-tools-apiserver
branch
from
f327a08 to
a65707d
Compare
|
|
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
|
Build failed: Generating tarball failed |
We will be eliminating the use of provider storage for tools. This is a step in that direction: hiding the use of provider storage behind an HTTP API provided by the API server for fetching tools. Later we will change the backend to fetch the tools from GridFS, when that work has been implemented.
Now that we are serving tools in the API server, we must consider the availability of the API server at the time a machine is provisioned. The aria2 package is an alternative to curl that is capable of failing over to additional sources and optionally downloading chunks in parallel. Note: curl can be built with support for Metalink, which would have been a more conservative option. Unfortunately the version of curl packaged in Ubuntu does not have Metalink support enabled.
The local provider always has tools available locally, and user-data for lxc and kvm does not have the size limitations as in cloud providers. Thus, we can safely serialise the tools tarball into user-data in the local provider.
Log errors in sendError functions, and change signature to not return error.
axw
force-pushed
the
cloudinit-download-tools-apiserver
branch
from
a65707d to
d02f41e
Compare
|
|
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
|
Build failed: Tests failed |
We have tests for prereq checking, we don't need to do it everywhere.
|
|
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
|
Build failed: Tests failed |
|
|
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
Download tools from API server We will be eliminating the use of provider storage for tools. This is a step in that direction: hiding the use of provider storage behind an HTTP API provided by the API server for fetching tools. Later we will change the backend to fetch the tools from GridFS, when that work has been implemented. Now that we are serving tools in the API server, we must consider the availability of the API server at the time a machine is provisioned. The aria2 package is an alternative to curl that is capable of failing over to additional sources and optionally downloading chunks in parallel. Note: curl can be built with support for Metalink, which would have been a more conservative option. Unfortunately the version of curl packaged in Ubuntu does not have Metalink support enabled. The local provider always has tools available locally, and user-data for lxc and kvm does not have the size limitations as in cloud providers. Thus, we can safely serialise the tools tarball into user-data in the local provider.
We will be eliminating the use of provider storage for tools. This is a
step in that direction: hiding the use of provider storage behind an HTTP
API provided by the API server for fetching tools. Later we will change the
backend to fetch the tools from GridFS, when that work has been
implemented.
Now that we are serving tools in the API server, we must consider the
availability of the API server at the time a machine is provisioned. The
aria2 package is an alternative to curl that is capable of failing over to
additional sources and optionally downloading chunks in parallel.
Note: curl can be built with support for Metalink, which would have been a
more conservative option. Unfortunately the version of curl packaged in
Ubuntu does not have Metalink support enabled.
The local provider always has tools available locally, and user-data for
lxc and kvm does not have the size limitations as in cloud providers. Thus,
we can safely serialise the tools tarball into user-data in the local
provider.