Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot upload artifacts #160

Open
kinosang opened this issue Oct 17, 2021 · 17 comments
Open

Cannot upload artifacts #160

kinosang opened this issue Oct 17, 2021 · 17 comments
Labels
help wanted Extra attention is needed

Comments

@kinosang
Copy link

Got error of ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2384)

full logs see https://github.com/JoyMoe/OpenData/runs/3917346331?check_suite_focus=true

@eine
Copy link
Owner

eine commented Oct 17, 2021

That is probably a temporary connection error on GitHub's side. Can you please try restarting the workflow?

Just in case, remove any asset from https://github.com/JoyMoe/OpenData/releases/tag/nightly starting with tmp.*.

@kinosang
Copy link
Author

@eine after remove the release, it works well now. I'll keep watching for some days.

@lazka
Copy link
Contributor

lazka commented Oct 22, 2021

we have the same problem here https://github.com/msys2/msys2-installer/runs/3980189142?check_suite_focus=true

this started ~18 day ago

@kinosang
Copy link
Author

It works well for 8 days now. Seems no other action need to take.

@stnolting
Copy link

Same issue over here: https://github.com/stnolting/neorv32/runs/3995838174?check_suite_focus=true

The problem started yesterday; four days ago everything was still working fine.

@umarcor
Copy link

umarcor commented Oct 25, 2021

@stnolting, I cleaned the assets of the nightly release in NEORV32 and I restarted the Documentation workflow. It's green now.

@stnolting
Copy link

@umarcor
That's all? 😅 Awesome, thank you very much!

@lazka
Copy link
Contributor

lazka commented Oct 25, 2021

yeah, deleting assets before the upload seems to work around the issue

@eine
Copy link
Owner

eine commented Oct 27, 2021

The main problem here is that GitHub's internal connections seem not to be very stable, so rather frequently uploading artifacts as assets does produce failures. Until recently, tip could not handle the broken state, and that's why the recommended workaround was to remove the assets manually and restart the workflow. Most of the time, that works. However, it is undesirable, because assets are not available for the duration of the workflow (which can take up to 6h). So, users willing to retrieve tip/nightly assets do get errors during that period.

In order to maximise the availability of assets, tip first uploads new versions with a different name, then removes the old one and last it renames the new:

tip/tip.py

Lines 160 to 166 in fb0c52c

print(f" - uploading tmp.{aname!s}...")
new_asset = upload_asset(artifact, name=f"tmp.{aname!s}")
print(f" - removing...{aname!s}")
asset.delete_asset()
print(f" - renaming tmp.{aname!s} to {aname!s}...")
new_asset.update_asset(aname, label=aname)
artifacts.remove(artifact)

Until some days ago, when the first step failed ("uploading tmp."), all subsequent executions of the Action would fail, because there was no procedure for cleaning it. So, a possible less intrusive solution was to manually remove the assets shown as tmp.... and then restart the workflow. I fixed that and now the Action should handle it automatically:

tip/tip.py

Lines 139 to 152 in fb0c52c

def upload_asset(artifact, name):
try:
return gh_release.upload_asset(artifact, name=name)
except GithubException as ex:
if "already_exists" in [err["code"] for err in ex.data["errors"]]:
print(f" - {name} exists already! deleting...")
delete_asset_by_name(name)
else:
print(f" - uploading failed: {ex}")
except Exception as ex:
print(f" - uploading failed: {ex}")
print(f" - retry uploading {name}...")
return gh_release.upload_asset(artifact, name=name)

Therefore, most of the problems you had should not happen again, as long as they were related to this "uploading of tmp files failing".

However, I still found some failing cases that required me to manually remove all assets (not the release, just the assets), and not only the tmp. ones, which the Action was already handling properly. That means there might be some other cause povoking the release to be in a broken state, but I'm unsure about the root. Please, let me know if you keep having issues.

On the other hand, users with option rm: true should not be affected by this issue at all.

@lazka
Copy link
Contributor

lazka commented Nov 15, 2021

Tried again, still fails :( https://github.com/msys2/msys2-installer/runs/4202718077?check_suite_focus=true

Likely the problem is with github, still annoying :(

@lazka
Copy link
Contributor

lazka commented Nov 15, 2021

Could this be an alpine related issue? searching for the SSL error gives me various alpine user reports.

Maybe we could try the default docker image instead (python:3.10 or something)?

@eine
Copy link
Owner

eine commented Nov 15, 2021

@lazka, absolutely! If that is the case, that'd be an awesome catch! Let me run some local tests and I'll update the dockerfile.

@eine
Copy link
Owner

eine commented Nov 15, 2021

So, on the one hand I changed the base image from python:alpine to python:slim-bullseye. That's twice the size (142MB vs 64 MB), but let's hope it solves the problem. On the other hand, I added a variant of the action to subdir composite. As the name suggests, that's a Composite Action instead of a Docker Action. That will install the dependencies (PyGitHub) on the workflow, instead of using an isolated container. However, it will let us know whether the problems are due to using containers.

@lazka, @stnolting, @kinosang, if you want to try the composite variant, just changing the name should be enough: eine/tip/composite@master.

@eine eine added the help wanted Extra attention is needed label Nov 15, 2021
@lazka
Copy link
Contributor

lazka commented Nov 16, 2021

That's twice the size (142MB vs 64 MB), but let's hope it solves the problem.

Just wanted to point out that despite it being larger it is still faster to run since there are Python wheels for Debian while it compiles packages on alpine.

@lazka
Copy link
Contributor

lazka commented Nov 16, 2021

Still the same issue: https://github.com/msys2/msys2-installer/runs/4229934050?check_suite_focus=true

Sorry for the wrong lead :/

@lazka
Copy link
Contributor

lazka commented Nov 20, 2021

So, another lead: this started to happen exactly when Python 3.10 was released (5 Oct on docker hub). And the docker image is always using the newest one.

lazka added a commit to lazka/tip that referenced this issue Nov 20, 2021
Related to eine#160. The errors started arround the time when Python 3.10 was
released, so go back to 3.9.x
lazka added a commit to lazka/tip that referenced this issue Nov 20, 2021
Related to eine#160. The errors started around the time when Python 3.10 was
released, so go back to 3.9.x to see if that is related.
eine pushed a commit that referenced this issue Nov 20, 2021
Related to #160. The errors started around the time when Python 3.10 was
released, so go back to 3.9.x to see if that is related.
@lazka
Copy link
Contributor

lazka commented Nov 20, 2021

Now it's just another error: https://github.com/msys2/msys2-installer/runs/4273363943?check_suite_focus=true

But at least it only fails every second time instead of every time :) oh well...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants