Disclaimer: I'm working on TruffleRuby.

GraalVM (the distribution) is a JVM & JDK, so JRuby should work fine on GraalVM, although I have not tried it.
So you should be able to first switch to GraalVM to replace deprecated Nashorn with Graal.js.

If you plan to use Graal.js (GraalVM's JavaScript), then there are significant advantages if you want any kind of interoperability between Ruby and JS, as that's much easier to do with 2 GraalVM languages and a lot faster too.
If not, it would come down to whether what you do in Ruby is faster/easier on JRuby or TruffleRuby.

@spark08 JRuby runs fine on GrallVM. For some numeric benchmarks it can run quite a bit faster (the escape analysis works better for some things like our fixnum impl -- if you want to put it in best light disable our fixnum cache (-Xfixnum.cache=false)) but in general it is similar in speed to a hotspot-based JVM.

Depending on the format of how you are storing formulas and the need for isolation good ole rhino may be the fastest to startup but not be the fastest to actually execute (e.g. very short-lived JS envs coming and going). If your formulas exist in a JS runtime for longer period of time then using another JS impl probably will work better for you (warmup will continue to yield faster executing code).

If someone has too much time, could a "hello world" example + mini-tutorial be added to the wiki, if this is the case in regards to jruby + GraalVM? Can we package commandline things written for jruby into GraalVM? How about java-swing applications - in jruby this works very well, whereas in GraalVM I have read that it does not (yet) do. There are so many questions to be had and I am not entirely sure how easy it is to test it. I myself feel barely past the newbie stage in regards to jruby actually, though I do also think I know quite a lot about ruby so far. Searching for google yields so many crap results these days ... finding information became very difficult.