Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add a very basic precompile workload #2004

Merged
merged 2 commits into from
Mar 6, 2023
Merged

add a very basic precompile workload #2004

merged 2 commits into from
Mar 6, 2023

Conversation

KristofferC
Copy link
Member

@KristofferC KristofferC commented Dec 30, 2022 •
edited by mortenpi
Loading

Before:

❯ time julia --project docs_precompile/make.jl
julia --project docs_precompile/make.jl  7.15s user 0.23s system 93% cpu 7.901 total

After

❯ time julia --project docs_precompile/make.jl
julia --project docs_precompile/make.jl  0.63s user 0.09s system 58% cpu 1.234 total

Using Julia-1.9.0-beta2.

Edit by @mortenpi: close #2002

@KristofferC KristofferC mentioned this pull request Dec 30, 2022
Closed
mortenpi
mortenpi approved these changes Jan 15, 2023
View reviewed changes
Copy link
Member

@mortenpi mortenpi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something to note is that the first run time (i.e. precompilation + makedocs) goes up with this (14s -> 20s on the docs_precompile workload). But I think that is not a particularly problematic here, as you would mostly run into this on CI.

@tfiers
Copy link

tfiers commented Jan 15, 2023 •
edited
Loading

first run time (i.e. precompilation + makedocs) goes up with this (14s -> 20s on the docs_precompile workload). But I think that is not a particularly problematic here, as you would mostly run into this on CI.

Is it possible to cache precompilation between CI runs?
If so, that might be very worthwile to add to e.g. the PkgTemplates doc-build step

@mortenpi
Copy link
Member

Is it possible to cache precompilation between CI runs?

Yes (https://github.com/julia-actions/cache#inputs), but it looks like it might create other issues.

@mortenpi
Copy link
Member

It looks like the nightly failures are related to this PR (cf. https://github.com/JuliaDocs/Documenter.jl/actions/runs/3925278191 for a new master run). Any thoughts @KristofferC?

@inkydragon
Copy link
Contributor

Maybe run CI again.
I cannot reproduce the error locally with Version 1.10.0-DEV.665 (2023-02-27)

@mortenpi mortenpi closed this Feb 28, 2023
@mortenpi mortenpi reopened this Feb 28, 2023
@mortenpi
Copy link
Member

CI still fails it looks like.

@inkydragon
Copy link
Contributor

I can reproduce this issue on Version 1.10.0-DEV.667 (2023-02-27).

julia-latest --color=yes --project=@. -e 'using Pkg; Pkg.build(verbose=true);' 
julia-latest --color=yes --depwarn=yes --inline=yes --project=@. .\test\runtests.jl

Internal error: encountered unexpected error in runtime:
UndefRefError()
...

Documenter: Error During Test at E:\jl\Documenter.jl\test\runtests.jl:5
  Got exception outside of a @test
  LoadError: TypeError: in new, expected IOStream, got a value of type Base.PipeEndpoint
  Stacktrace:
...
full logs 
> julia-latest --color=yes --project=@. .\test\runtests.jl
[ Info: Building example/make.jl
[ Info: Building mock package docs: HTMLWriter / deployment build
Internal error: encountered unexpected error in runtime:
UndefRefError()
getindex at .\essentials.jl:13 [inlined]
copy_exprargs at .\expr.jl:68
copy at .\expr.jl:41
copy_exprs at .\expr.jl:46
copy_exprargs at .\expr.jl:68
copy at .\expr.jl:75
inflate_ir at .\compiler/ssair\legacy.jl:60 [inlined]
codeinst_to_ir at .\compiler/ssair\irinterp.jl:129 [inlined]
abstract_call_method_with_const_args at .\compiler\abstractinterpretation.jl:978
abstract_call_method_with_const_args at .\compiler\abstractinterpretation.jl:960
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:158
abstract_call_known at .\compiler\abstractinterpretation.jl:2009
abstract_call at .\compiler\abstractinterpretation.jl:2085
abstract_call at .\compiler\abstractinterpretation.jl:2059
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2246
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2500
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2767
typeinf_local at .\compiler\abstractinterpretation.jl:2976
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3064
_typeinf at .\compiler\typeinfer.jl:244
typeinf at .\compiler\typeinfer.jl:215
typeinf_edge at .\compiler\typeinfer.jl:920
abstract_call_method at .\compiler\abstractinterpretation.jl:653
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:150
abstract_call_known at .\compiler\abstractinterpretation.jl:2009
abstract_call at .\compiler\abstractinterpretation.jl:2085
abstract_call at .\compiler\abstractinterpretation.jl:2059
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2246
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2500
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2791
typeinf_local at .\compiler\abstractinterpretation.jl:2976
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3064
_typeinf at .\compiler\typeinfer.jl:244
typeinf at .\compiler\typeinfer.jl:215
typeinf_edge at .\compiler\typeinfer.jl:920
abstract_call_method at .\compiler\abstractinterpretation.jl:653
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:150
abstract_call_known at .\compiler\abstractinterpretation.jl:2009
abstract_call at .\compiler\abstractinterpretation.jl:2085
abstract_call at .\compiler\abstractinterpretation.jl:2059
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2246
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2500
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2767
typeinf_local at .\compiler\abstractinterpretation.jl:2976
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3064
_typeinf at .\compiler\typeinfer.jl:244
typeinf at .\compiler\typeinfer.jl:215
typeinf_edge at .\compiler\typeinfer.jl:920
abstract_call_method at .\compiler\abstractinterpretation.jl:653
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:150
abstract_call_known at .\compiler\abstractinterpretation.jl:2009
abstract_call at .\compiler\abstractinterpretation.jl:2085
abstract_call at .\compiler\abstractinterpretation.jl:2059
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2246
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2500
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2791
typeinf_local at .\compiler\abstractinterpretation.jl:2976
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3064
_typeinf at .\compiler\typeinfer.jl:244
typeinf at .\compiler\typeinfer.jl:215
typeinf_edge at .\compiler\typeinfer.jl:920
abstract_call_method at .\compiler\abstractinterpretation.jl:653
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:150
abstract_call_known at .\compiler\abstractinterpretation.jl:2009
abstract_call at .\compiler\abstractinterpretation.jl:2085
abstract_call at .\compiler\abstractinterpretation.jl:2059
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2246
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2500
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2791
typeinf_local at .\compiler\abstractinterpretation.jl:2976
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3064
_typeinf at .\compiler\typeinfer.jl:244
typeinf at .\compiler\typeinfer.jl:215
typeinf_ext at .\compiler\typeinfer.jl:1044
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1077
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1073
jfptr_typeinf_ext_toplevel_20819.clone_1 at C:\Users\woclass\AppData\Local\julias\julia-latest\lib\julia\sys.dll (unknown line)
_jl_invoke at C:/workdir/src\gf.c:2725 [inlined]
ijl_apply_generic at C:/workdir/src\gf.c:2907 [inlined]
jl_apply at C:/workdir/src\julia.h:1876 [inlined]
jl_type_infer at C:/workdir/src\gf.c:317
jl_generate_fptr_impl at C:/workdir/src\jitlayers.cpp:444
jl_compile_method_internal at C:/workdir/src\gf.c:2316
jl_compile_method_internal at C:/workdir/src\gf.c:2209 [inlined]
_jl_invoke at C:/workdir/src\gf.c:2717 [inlined]
ijl_apply_generic at C:/workdir/src\gf.c:2907
jl_apply at C:/workdir/src\julia.h:1876 [inlined]
do_call at C:/workdir/src\interpreter.c:125
eval_value at C:/workdir/src\interpreter.c:222
eval_body at C:/workdir/src\interpreter.c:474
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:758
top-level scope at E:\jl\Documenter.jl\test\examples\make.jl:255
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:910
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:853
ijl_toplevel_eval at C:/workdir/src\toplevel.c:919 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:969
eval at .\boot.jl:370 [inlined]
include_string at .\loading.jl:1889
_include at .\loading.jl:1949
include at .\client.jl:478
unknown function (ip: 00000167ffe90336)
jl_apply at C:/workdir/src\julia.h:1876 [inlined]
do_call at C:/workdir/src\interpreter.c:125
eval_value at C:/workdir/src\interpreter.c:222
eval_stmt_value at C:/workdir/src\interpreter.c:173 [inlined]
eval_body at C:/workdir/src\interpreter.c:620
eval_body at C:/workdir/src\interpreter.c:529
eval_body at C:/workdir/src\interpreter.c:529
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:758
macro expansion at E:\jl\Documenter.jl\test\runtests.jl:8 [inlined]
macro expansion at C:\workdir\usr\share\julia\stdlib\v1.10\Test\src\Test.jl:1504 [inlined]
top-level scope at E:\jl\Documenter.jl\test\runtests.jl:7
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:910
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:853
ijl_toplevel_eval at C:/workdir/src\toplevel.c:919 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:969
eval at .\boot.jl:370 [inlined]
include_string at .\loading.jl:1889
_include at .\loading.jl:1949
include at .\Base.jl:467
jfptr_include_27574.clone_1 at C:\Users\woclass\AppData\Local\julias\julia-latest\lib\julia\sys.dll (unknown line)
exec_options at .\client.jl:307
_start at .\client.jl:541
jfptr__start_57110.clone_1 at C:\Users\woclass\AppData\Local\julias\julia-latest\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1876 [inlined]
true_main at C:/workdir/src\jlapi.c:573
jl_repl_entrypoint at C:/workdir/src\jlapi.c:717
mainCRTStartup at C:/workdir/cli\loader_exe.c:58
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Documenter: Error During Test at E:\jl\Documenter.jl\test\runtests.jl:5
  Got exception outside of a @test
  LoadError: TypeError: in new, expected IOStream, got a value of type Base.PipeEndpoint
  Stacktrace:
    [1] IOContext
      @ .\show.jl:296 [inlined]
    [2] IOContext
      @ .\show.jl:306 [inlined]
    [3] IOContext
      @ .\show.jl:315 [inlined]
    [4] capture(f::var"#14#16"{NamedTuple{(:edit_link,), Tuple{Symbol}}, Tuple{String, String, String}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, String, MathJax2}; rethrow::Type, color::Bool)
      @ IOCapture C:\Users\woclass\.julia\packages\IOCapture\sXY71\src\IOCapture.jl:82
    [5] _quietly(f::Function, expr::Expr, source::LineNumberNode)
      @ Main.TestUtilities E:\jl\Documenter.jl\test\TestUtilities.jl:47
    [6] macro expansion
      @ E:\jl\Documenter.jl\test\TestUtilities.jl:100 [inlined]
    [7] html_doc(build_directory::Any, mathengine::Any; htmlkwargs::Any, image_assets::Any, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
      @ Main E:\jl\Documenter.jl\test\examples\make.jl:222
    [8] top-level scope
      @ E:\jl\Documenter.jl\test\examples\make.jl:255
    [9] include(fname::String)
      @ Base.MainInclude .\client.jl:478
   [10] macro expansion
      @ E:\jl\Documenter.jl\test\runtests.jl:8 [inlined]
   [11] macro expansion
      @ C:\Users\woclass\AppData\Local\julias\julia-latest\share\julia\stdlib\v1.10\Test\src\Test.jl:1504 [inlined]
   [12] top-level scope
      @ E:\jl\Documenter.jl\test\runtests.jl:7
   [13] include(mod::Module, _path::String)
      @ Base .\Base.jl:467
   [14] exec_options(opts::Base.JLOptions)
      @ Base .\client.jl:307
   [15] _start()
      @ Base .\client.jl:541
  in expression starting at E:\jl\Documenter.jl\test\examples\make.jl:253
Test Summary: | Error  Total  Time
Documenter    |     1      1  3.6s
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at E:\jl\Documenter.jl\test\runtests.jl:5

@mortenpi
Copy link
Member

Are you sure it's not reproducible with .665? It also failed on CI with 1.10.0-DEV.367. Seems unlikely that there was a bug that got fixed and then re-introduced?

@inkydragon
Copy link
Contributor

  • ❌ Version 1.10.0-DEV.667 (2023-02-27) a23b29cb1a
  • ❌ Version 1.10.0-DEV.665 (2023-02-27) 6bf77c0ed8
  • ❌ Version 1.10.0-DEV.600 (2023-02-14) 1541519457
  • ❌ Version 1.10.0-DEV.563 (2023-02-10) 500f561f72
  • ❌ Version 1.10.0-DEV.408 (2023-01-20) ba69cbaa8e
  • ❌ Version 1.10.0-DEV.303 (2023-01-09) d21a25d52c
  • ❌ Version 1.10.0-DEV.226 (2022-12-29) 5c1b99a5c3
  • ❌ Version 1.10.0-DEV.127 (2022-12-12) 4ff707227f
  • ❌ Version 1.10.0-DEV.80 (2022-11-30) f4534d16b4
  • ❌ Version 1.10.0-DEV.75 (2022-11-30) 70bda2cfe4
  • ❌ Version 1.10.0-DEV.73 (2022-11-29) e06a5915a9 ❌
  • ✅ Version 1.10.0-DEV.72 (2022-11-29) 328dd57895 ✅
  • ✅ Version 1.10.0-DEV.68 (2022-11-29) 4fd26ba135
  • ✅ Version 1.10.0-DEV.56 (2022-11-28) b34fe1d695
  • ✅ Version 1.10.0-DEV.34 (2022-11-21) 23160a109d ✅
  • ✅ Version 1.10.0-DEV.0 (2022-11-14) 7fe6b16f40 ✅
git bisect bad
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
cbfdb3facd0f2ece4088f43ef97533e9e0921081
e06a5915a9e93ed5d4c25cf819275af18adf8187
We cannot bisect more!

@mortenpi
Copy link
Member

It looks like it's this PR then: JuliaLang/julia#44527

But it might be good to see if we can get an MWE out. It looks like the error actually comes from IOCapture.jl.

@mortenpi mortenpi mentioned this pull request Feb 28, 2023
Closed
@mortenpi
Copy link
Member

I was able to reproduce something with just IOCapture.jl, which I hope will be easier to debug. The error looks a bit different, but I have a hunch it's the same underlying issue. (JuliaLang/julia#48837)

@inkydragon
Copy link
Contributor

Version 1.10.0-DEV.699 (2023-03-04) works fine for me.

@mortenpi mortenpi closed this Mar 6, 2023
@mortenpi mortenpi reopened this Mar 6, 2023
@mortenpi mortenpi merged commit f0d5cd2 into JuliaDocs:master Mar 6, 2023
@KristofferC KristofferC deleted the kc/precompile branch March 6, 2023 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mortenpi mortenpi mortenpi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Precompilation
4 participants
@KristofferC @tfiers @mortenpi @inkydragon