I wonder if the delete command should also remove the --tag (both locally and on the remote)? The underlying git command for this is:

git push --delete origin <tag> && git tag --delete <tag>

I have an alias set up, but thinking maybe this should be the default behaviour?