Skip to content

Release automation script #10449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
krobelus wants to merge 8 commits into from
Closed

Release automation script #10449

krobelus wants to merge 8 commits into from

Conversation

@krobelus
Copy link
Contributor

@krobelus krobelus commented Apr 18, 2024

We provide releases in several different formats and channels.
There are many manual steps involved in making a release.
This makes it hard to learn or share improvements (especially share improvements that last once-and-for-all).
Also there is potential for error -- for example our Debian PPAs don't include the latest release and our website doesn't link to the latest tarball.
With our current approach, packaging tends to get more complicated, for example we'll want to add things like "Depends: librust-hexponent" to our Debian package, and set the version based on Cargo.toml.

Add a script to automate the boring steps of the release process.
This should obsolete most of https://github.com/fish-shell/fish-shell/wiki/Release-checklist

It seems to work on my Linux system but it's obviously not battle-tested yet.
The most essential TODO comments ones are for uploading Debian packages.

Iteration on the script itself is usually quite fast. I run something like

git add build_tools/release.sh
git commit -m wip
build_tools/release.sh 3.8.0 --dry-run --no-test # plus eventual --no-rebuild --no-rebuild-debs

As a temporary hack, the Debian packages are currently built in a Docker container.
It'd be easy to skip Docker when debmake is installed on the host system.
We don't yet pass through gpg-agent into Docker, so the debuild process requires interactive password input.

It takes a really long time (> 15 minutes) to build all Debian packages because we rebuild them sequentially for all Ubuntu releases (mantic, jammy, focal, bionic and xenial) even though only "debian/changelog" is different.
For iterating on the script, we can use "--no-rebuild-debs" to reuse the the packages after they've been built once.
Of course this is only valid if the tarball has not changed since.
We could probably ask debuild to validate this, or speed up the process in a different way, at least parallelize.

Future ideas:

  • we could run the script in CI instead, to keep us even more honest.
  • we should try to simplify the website release process (should not need to
    talk to the github API, the script already has enough information).
  • do something similar for the the macOS-specific parts

@krobelus krobelus marked this pull request as draft April 18, 2024 12:47
@zanchey zanchey self-requested a review April 21, 2024 15:12
@ridiculousfish ridiculousfish mentioned this pull request Jul 27, 2024
Closed
25 tasks
@zanchey
Copy link
Member

zanchey commented Aug 4, 2024

Also there is potential for error -- for example our Debian PPAs don't include the latest release and our website doesn't link to the latest tarball.

I don't think this is right. The Debian archive at https://software.opensuse.org/download.html?project=shells%3Afish%3Arelease%3A3&package=fish definitely contains the latest release, and the tarball link at fishshell.com is correct too (although it doesn't appear in the list of "Release history", which could probably be changed).

@faho faho mentioned this pull request Mar 1, 2025
Closed
@krobelus
github actions: fix warning about unexpected inputs
9faf78d 
CI runs show warnings like

	ubuntu-32bit-static-pcre2
	Unexpected input(s) 'targets', valid inputs are ['']

This is about the rust-toolchain action, which is a composite action, see
https://docs.github.com/en/actions/tutorials/create-actions/create-a-composite-action
not a full workflow
https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows

Looks like composite actions specify inputs at top level.
Also they should not need «on: ["workflow_call"]».

The unexpected inputs are still forwared, so it happens to work.
Fix the warnings.
@krobelus
github workflows autolabel: remove stale milestone code
acad6e6 
We look for a milestone that no longer exists.
Remove this until we find a solution.
@krobelus
github workflows: mac_codesign: clean up
2563adf
krobelus added a commit to krobelus/fish-shell that referenced this pull request Sep 10, 2025
@krobelus
Release automation script
9a416bf 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- The *.asc signature file for the tarball (or for any other release assets)
- GPG-signed Debian and other OBS packages
- Email to "fish-users Mailing List <[email protected]>"

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Hope that this doesn't cause security problems.

Usage:
1. run "build_tools/release.sh $version"; this will push a tag,
   kicking off .github/workflows/release.yml
2. wait for the draft release to be created at https://github.com/fish-shell/fish-shell/releases/$version
3. publish the draft (manually for now). This should unblock the
   last part of the workflow, which updates the website.

Closes fish-shell#10449

Incremental usage example:

	version=4.0.3
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$remote/fish-shell/actions"
	echo "See the draft release at https://github.com/$remote/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $remote
ridiculousfish
ridiculousfish reviewed Sep 11, 2025
View reviewed changes
.github/workflows/release.yml
on:
push:
tags:
- '*.*.*'
Copy link
Member

@ridiculousfish ridiculousfish Sep 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elegant!

.github/workflows/release.yml Outdated
with:
# Workaround for https://github.com/actions/checkout/issues/882
ref: ${{ github.ref }}
- uses: ./.github/actions/rust-toolchain@oldest-supported
Copy link
Member

@ridiculousfish ridiculousfish Sep 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A self-note that I should check that this works back to what we nominally support for 4.0.x (macOS 10.9). 4.1.x will drop support for these very old versions.

Copy link
Contributor Author

@krobelus krobelus Sep 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

macOS needs it yeah, but I don't think x86_64-unknown-linux-musl or aarch64-unknown-linux-musl have a particular reason to require 1.70 (requirements on linux version did not change), so I'll change those to use stable

@krobelus
Rename build_tools/make_pkg.sh
6056b54 
pkg is a pretty subtle name?
@krobelus
Add sphinx-markdown-builder for generating release notes
0323159 
Without this, Sphinx refuses to use the "-b markdown" builder (see next commit).
@krobelus
CHANGELOG: minor updates
2686e4c
@krobelus
github workflows staticbuild: use stable Rust on Linux
a62258a 
We use the MSRV for CI checks, and for deploying to old macOS.
But for static Linux builds , there should be no reason to use an
old Rust version.  Let's track stable.
@krobelus
Release automation script
3f58629 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes fish-shell#10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
4917ef2 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
6127638 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
ba31782 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
7d07d0f 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
2ad6f2e 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
10b43a2 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
3d1beb7 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
7909211 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Release automation script
a059f76 
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages

To-do:
- remove the corresponding entries from
  https://github.com/fish-shell/fish-shell/wiki/Release-checklist
  and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.

Usage:
1. run "build_tools/release.sh $version"; this will create and push
   a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
   https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
   last part of the workflow (website updates).

Closes #10449

Incremental usage example:

	version=4.0.3
	repository_owner=fish-shell
	remote=origin
	cd ../fish-shell-secondary-worktree
	git tag -d $version ||:
	git push $remote :$version ||:
	git reset --hard origin/Integration_$version
	for d in .github build_tools; do {
		rm -rf $d
		cp -r ../fish-shell/$d .
		git add $d
	} done
	git commit -m 'Backport CI/CD'
	echo "See https://github.com/$repository_owner/fish-shell/actions"
	echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
	../fish-shell/build_tools/release.sh $version $repository_owner $remote
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Backport release-script related changes from master
a576904 
Will commit these to master momentarily (#10449).
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Backport release-script related changes from master
1beb32c 
Will commit these to master momentarily (#10449).
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Backport release-script related changes from master
dde85b3 
Will commit these to master momentarily (#10449).
@krobelus krobelus closed this in 5e658bf Sep 12, 2025
krobelus added a commit that referenced this pull request Sep 12, 2025
@krobelus
Backport release-script related changes from master
e2005c6 
Will commit these to master momentarily (#10449).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ridiculousfish ridiculousfish ridiculousfish left review comments

@zanchey zanchey Awaiting requested review from zanchey

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@krobelus @zanchey @ridiculousfish