Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download from a different workflow #3

Closed
mastoj opened this issue Aug 15, 2019 · 51 comments
Closed

Download from a different workflow #3

mastoj opened this issue Aug 15, 2019 · 51 comments
Labels
enhancement New feature or request

Comments

@mastoj
Copy link

mastoj commented Aug 15, 2019

How do you download an artifact from another workflow?

A little bit of background. We have a build workflow that builds an artifact that we publish. Then we have a separate workflow that is triggered on deployment that needs to access this artifact. I tried to just use the same name for the artifact in download-artifact and upload-artifact hoping that it would download the files, that doesn't seem to work.

@mastoj
Copy link
Author

mastoj commented Aug 15, 2019

It is a little bit unclear about the direction that you intended here. I definitely see some workarounds:

  • Create a release and use the sha as tag. Don't like this approach since this will create a release of every build and mess up the releases.
  • Use the package registry maybe? Can't do this right now since we don't have access

Maybe we are doing something completely wrong, and that is fine since it is quite new :). With that said, if we can't access the artifacts we have to revert to circle ci for the build and use github actions only for deploy.

@TingluoHuang
Copy link
Member

@chrispat for feedback

@kimble
Copy link

kimble commented Sep 2, 2019

This could also be relevant for build caching.

@cormacrelf
Copy link

And also for diffing artifacts between master and a pull request. There are hundreds of apps that do this for visual/webpage diffs, but none I can find that can diff a simple text file artifact.

@zlepper
Copy link

zlepper commented Dec 23, 2019

In addition to this: How to do this cross repositories?

Also we would like to limit the download to certain branches (Using somewhat the same logic as what TeamCity uses with their artifact dependencies), for example we can use the build output of one repository master, in another repository (creating a master build from several smaller projects). Preferably this would use the same branch if possible, or fallback to the default branch if possible.

@sjackman
Copy link

I'm very excited for this feature. I want to build and upload artifacts from a fork PR, and this part works well already. Fork PRs run without access to secrets, so in a second GitHub Action workflow that runs with access to secrets (outside of the fork PR), I want to download artifacts from GitHub and upload them to another service (BinTray). I had this arrangement working using CircleCI. This missing feature is blocking me from migrating from CircleCI to GitHub Actions, so I'm very keen to see it implemented!

@sjackman
Copy link

H, @ethomson! I work with @MikeMcQuaid on Homebrew. He told me that you may be a good person to give this issue some 👀. See also the related issue actions/upload-artifact#21 (comment).

@mikkelbd
Copy link

Same use case as OP here. Any progress on this? 🙏 It would be really helpful to access an artifact uploaded from a build-workflow in a deploy-workflow. Especially since the cache action doesn't support deployment events: actions/cache#63. Otherwise the cache could probably have been used with e.g. "artifact-name-$GITHUB_SHA" even though it's kind of a hack.

@chrispat
Copy link
Member

@mikkelbd we now have an api for downloading artifactshttps://developer.github.com/v3/actions/artifacts/#download-an-artifact so you could likely accomplish your scenario now.

@eine
Copy link

eine commented Jan 28, 2020

@chrispat, I believe that @mikkelbd, as me and probably many others, is expecting this action to support the feature. In the end, if we are all forced to write custom scripts that interact with the API, all the "Actions" infrastructure is useless. This is specially unfortunate because GitHub's CI service as a product seems to be severely biased towards Actions.

@mikkelbd
Copy link

@chrispat Thank you for the prompt response 👏 Artifacts being available in the API is a good start. Combined with https://github.com/octokit/request-action or https://github.com/actions/github-script one could probably download it without writing custom code. However, it seems that you would have to know the internal artifact_id of the artifact, or you would have to list all artifacts and then filter by name? Again, then you would have to know the run_id that uploaded the artifact? I guess that's pretty hard in another workflow run...

As @eine stated, it would be a much better experience if the download-artifact action supported this out of the box instead of forcing n developers to interact with the GitHub API and also making the workflow-file bloated with low-level stuff 😅

@sjackman
Copy link

we now have an api for downloading artifactshttps://developer.github.com/v3/actions/artifacts/#download-an-artifact so you could likely accomplish your scenario now.

@chrispat Is there an API to get all the workflow runs for a given pull request or commit SHA-1? I found this API list-workflow-runs to list all the runs for a given repo and workflow, and that could then be filtered to find the SHA1 of the commit that you're interested in, but not terribly efficient.

https://developer.github.com/v3/actions/workflow_runs/#list-workflow-runs

@chrispat
Copy link
Member

We don't currently have that parameter to the API but I will add it to our feedback list to consider for the future.

@sjackman
Copy link

sjackman commented Jan 31, 2020

pr="$(echo '${{github.event.head_commit.message}}' | sed 's/^.*#\([0-9]*\).*/\1/;q')"
git -C "$(brew --repo ${{github.repository}})" fetch origin "pull/$pr/head:pr"
sha1="$(git -C "$(brew --repo ${{github.repository}})" rev-parse pr)"
echo pr="$pr" sha1="$sha1"
run_id=$(curl -s -H 'Accept: application/vnd.github.antiope-preview+json' https://api.github.com/repos/${{github.repository}}/actions/workflows/build-bottles.yml/runs \
  | jq ".workflow_runs[] | select(.head_sha == \"$sha1\").id")
artifact_id="$(curl -s -H 'Accept: application/vnd.github.antiope-preview+json' https://api.github.com/repos/${{github.repository}}/actions/runs/$run_id/artifacts \
  | jq '.artifacts[0].id')"
echo run_id="$run_id" artifact_id="$artifact_id"
curl -L -o bottles.zip "https://${{secrets.GITHUB_PAT}}@api.github.com/repos/${{github.repository}}/actions/artifacts/$artifact_id/zip"

https://github.com/brewsci/homebrew-bio/blob/master/.github/workflows/upload-bottles.yml
I've successfully used the new workflows and artifacts endpoints to download the artifact of given PR number in a push event! Yeah! Any comments or suggestions to make this easier would be welcome.

Are artifacts of a public repo public, and why is a GITHUB_PAT needed to download an artifact? GITHUB_TOKEN wasn't sufficient.

@jpfeuffer
Copy link

I found the same oddity. It seems like downloading artifacts requires "admin" permissions instead of "read" permissions (as documented).

It would really be great if this would be a standard feature of this official GitHub action. Is there any progress on this?

@josejulio
Copy link

I found the same oddity. It seems like downloading artifacts requires "admin" permissions instead of "read" permissions (as documented).

Is this reported somewhere? If not, where would be the ideal place to report it?

@davidobrien1985
Copy link

It would be great if we'd be able to access builds from other repos in our org through this action.
Is that planned at all?

@josejulio
Copy link

It would be great if we'd be able to access builds from other repos in our org through this action.
Is that planned at all?

This action doesn't do that yet, but you can do it with curl

see: https://github.com/RedHatInsights/policies-ui-frontend/blob/master/.github/scripts/download-latest-openapi.sh#L23

You need to pass the token in the workflow for it to work correctly:
https://github.com/RedHatInsights/policies-ui-frontend/blob/master/.github/workflows/OpenAPIInSync.yml#L24

@joahjoah
Copy link

joahjoah commented May 6, 2022

I was just trying to figure out why I can't store an artifact in a workflow run, and download it in the next run, and that's why ? How is that not implemented ? this sounds like a crazy common use case to me, either to detect differences or reuse some build artifacts...

Would cache be more useful ? https://github.com/actions/cache I struggled to store a simple SARIF file and resorted to using upload artifact...

That is what I use the above-mentioned gist for. Compare changes in test coverage from different workflow runs.

Using the Github API it is fairly straightforward to find previously-stored artifacts. You can use octokit.rest.actions.listArtifactsForRepo and octokit.rest.actions.downloadArtifact

@suecharo's snippet doesn't even require you to write a custom action for it.

@Gby56
Copy link

Gby56 commented May 6, 2022

Yeah the problem with the gh run download cli is that it's not available on self hosted runners, the container image doesn't have it and I tried downloading the binary manually, I think I'm almost there but there is a remaining issue (probably caused bc the cli expects to be inside a repository, so I need to checkout...)

@demisx
Copy link

demisx commented Aug 31, 2022

In the meantime, this custom action allows to download artifacts created in a different workflow.

QuailtyDev added a commit to QuailtyDev/mui-next that referenced this issue Oct 22, 2022
This reverts commit 3bffdf2bfbf3c1a2906fed814025cf2e07881a41.

Download artifact from another workflow is not supported actions/download-artifact#3
thetic added a commit to thetic/cpputest that referenced this issue Dec 29, 2022
GitHub's own download action does not work across workflows:
actions/download-artifact#3
@ashishjullia
Copy link

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}?

Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2

I tried almost every possible way but can't make it run.

@MayCXC
Copy link

MayCXC commented Jan 16, 2023

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}?

Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2

I tried almost every possible way but can't make it run.

maybe actions/github-script#262 (comment) will help you?

@lpossamai
Copy link

Funny that we have to refer to 3rd parties for this. So many people would benefit from it.

@ashishjullia
Copy link

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}?
Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2
I tried almost every possible way but can't make it run.

maybe actions/github-script#262 (comment) will help you?

Yup, this is what I'm using rn, works well.

@djbrown
Copy link

djbrown commented Mar 19, 2023

You can use context.payload.workflow_run.id inside an action script to get the triggering workflow run id.
See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow.

@hwittenborn
Copy link

Is there any reason this hasn't been implemented yet? It seems like something that should be included, considering things like actions/checkout support specifying things like a repository just fine.

It looks like dawidd6/action-download-artifact will work just fine for getting this to all work, but having this work with a first-party action would definitely be ideal.

inomdzhon added a commit to VKCOM/gh-actions that referenced this issue Aug 23, 2023
Экшен позволяет вытаскивать опр-ый артефакт из другого воркфлоу. Использует [actions/github-script](https://github.com/actions/github-script). Основан на примере из доки [Использование данных из рабочего процесса, активирующего триггер](https://docs.github.com/ru/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow).

[actions/download-artifact@v3](https://github.com/actions/download-artifact/tree/v3) не умеет в такое (см. issue [actions/download-artifact#3](https://togithub.com/actions/download-artifact/issues/3)).

> **Note**
> Есть альтернатива [dawidd6/action-download-artifact](https://github.com/dawidd6/action-download-artifact), но этот экшен не верифицированный. Пробовал запросить у @VKCOM/vk-sec, но чёт там дело замялось. Да и выяснил, что `actions/github-script` хватает с головой.
@AndrewDryga
Copy link

action-download-artifact doesn't download the last artifact the workflow, so you might end up using a custom script like we do:
firezone/firezone#2665

@philiplinell
Copy link

philiplinell commented Dec 5, 2023

If you are not running self hosted runners (which may not have gh CLI), you can use a step such as the following to download an artifact across workflows.

  - name: Download artifact
    env:
      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |
      gh run download --name my-artifact --dir ${{ github.workspace }}

gh is available on all GitHub-hosted runners https://docs.github.com/en/actions/using-workflows/using-github-cli-in-workflows.

Code21228 added a commit to Code21228/Next.js-MUI that referenced this issue Dec 29, 2023
This reverts commit 3bffdf2bfbf3c1a2906fed814025cf2e07881a41.

Download artifact from another workflow is not supported actions/download-artifact#3
@konradpabjan
Copy link
Collaborator

With v4 this is now supported! It's also possible to download artifacts from other repositories 🎉 https://github.com/actions/download-artifact?tab=readme-ov-file#download-artifacts-from-other-workflow-runs-or-repositories

Recommend folks switch over to v4, a host of other improvements as well. https://github.blog/changelog/2023-12-14-github-actions-artifacts-v4-is-now-generally-available/

You will need to supply a PAT to download artifacts from other runs or repositories just FYI. GITHUB_TOKEN is only scoped to a single run so you will have to supply that to the action.

The options are not as extensive as some third party solutions like https://github.com/dawidd6/action-download-artifact but at least a basic version of this feature is officially supported

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests