If you have a test environment that hasn't been precompiled, then as of #3281, running Pkg.test(; io) will call two commands like

run(pipeline(`...`, stdout=io))
run(pipeline(`...`, stdout=io))

with the same IO object. This doesn't work well for a lot of common IO objects like IOBuffer, Base.BufferStream, Base.PipeBuffer, etc, since they will have eof(stream) == true after the first command finishes, and some of them will error on the second command.

┌ Warning: Process I/O error
│   exception =
│    ArgumentError: ensureroom failed, IOBuffer is not writeable
│    Stacktrace:
│     [1] ensureroom_slowpath(io::IOBuffer, nshort::UInt64)
│       @ Base ./iobuffer.jl:303
│     [2] ensureroom
│       @ ./iobuffer.jl:325 [inlined]
│     [3] unsafe_write(to::IOBuffer, p::Ptr{UInt8}, nb::UInt64)
│       @ Base ./iobuffer.jl:424
│     [4] unsafe_write
│       @ ./io.jl:685 [inlined]
│     [5] write
│       @ ./io.jl:708 [inlined]
│     [6] write(to::IOBuffer, from::Base.PipeEndpoint)
│       @ Base ./io.jl:755
│     [7] macro expansion
│       @ ./process.jl:300 [inlined]
│     [8] (::Base.var"#764#765"{IOBuffer, Bool, Base.PipeEndpoint, IOBuffer, Base.PipeEndpoint})()
│       @ Base ./task.jl:514
└ @ Base process.jl:302

julia> io = Base.BufferStream()
BufferStream(bytes waiting=0, isopen=true)

julia> let
           t = @async begin
               for _ = 1:10
                   bytes = readavailable(io)
                   @show String(bytes)
                   @show eof(io)
               end
           end
           run(pipeline(`echo "hi"`, stdout=io))
           run(pipeline(`echo "bye"`, stdout=io))
       end
String(bytes) = "hi\n"
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = "bye\n"
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
String(bytes) = ""
eof(io) = true
Process(`echo bye`, ProcessExited(0))

However devnull and stdout work fine, which my guess is what was tested here. So is there a particular restriction on what kinds of IO are allowed? Or is the change in #3281 a regression?

The context for me is I wrote some code to try to parse Pkg.jl test results "live" during GitHub Actions CI runs to re-emit test failures as logs which can then show up as annotations in GitHub PRs. It's actually fairly handy when it works (annotations only show up in-line on the diff, so you have to have changed those tests), but this was broken by #3281: julia-actions/julia-runtest#76.