Macro annotation (part 1) #16392

nicolasstucki · 2022-11-22T14:51:40Z

Add basic support for macro annotations

Introduce experimental scala.annotations.MacroAnnotation
Macro annotations can analyze or modify definitions
Macro annotation can add definition around the annotated definition
- Added members are not visible while typing
- Added members are not visible to other macro annotations
- Added definition must have the same owner
Implement macro annotation expansion
- Implemented at Inlining phase
  - Can use macro annotations in staged expressions (expanded when at stage 0)
  - Can use staged expression to implement macro annotations
  - Can insert calls to inline methods in macro annotations
- Current limitations (to be loosened in following PRs)
  - Can only be used on def, val, lazy val and var
  - Can only add def, val, lazy val and var definitions

Example

class memoize extends MacroAnnotation:
  def transform(using Quotes)(tree: quotes.reflect.Definition): List[quotes.reflect.Definition] =
    import quotes.reflect._
    tree match
      case DefDef(name, TermParamClause(param :: Nil) :: Nil, tpt, Some(rhsTree)) =>
        (Ref(param.symbol).asExpr, rhsTree.asExpr) match
          case ('{ $paramRefExpr: t }, '{ $rhsExpr: u }) =>
            val cacheTpe = TypeRepr.of[Map[t, u]]
            val cacheSymbol = 
              Symbol.newVal(tree.symbol.owner, name + "Cache", cacheTpe, Flags.Private, Symbol.noSymbol)
            val cacheRhs = '{ Map.empty[t, u] }.asTerm
            val cacheVal = ValDef(cacheSymbol, Some(cacheRhs))
            val cacheRefExpr = Ref(cacheSymbol).asExprOf[Map[t, u]]
            val newRhs = '{ $cacheRefExpr.getOrElseUpdate($paramRefExpr, $rhsExpr) }.asTerm
            val newTree = DefDef.copy(tree)(name, TermParamClause(param :: Nil) :: Nil, tpt, Some(newRhs))
            List(cacheVal, newTree)
      case _ =>
        report.error("Annotation only supported on `def` with a single argument are supported")
        List(tree)

with this macro annotation a user can write

@memoize
def fib(n: Int): Int =
  println(s"compute fib of $n")
  if n <= 1 then n else fib(n - 1) + fib(n - 2)

and the macro will modify the definition to create

val fibCache = mutable.Map.empty[Int, Int]
def fib(n: Int): Int =
  fibCache.getOrElseUpdate(
    n,
    {
      println(s"compute fib of $n")
      if n <= 1 then n else fib(n - 1) + fib(n - 2)
    }
  )

Based on

Followed by

Macro annotations class modifications (part 2) #16454

jilen · 2022-11-29T02:01:48Z

Why macro annotations cannot be used for ClassDef ? In scala2 macro annotations are mostly used for typeclass derivation, so most are used for ClassDef.

It will be great for migration if macro annotations can be used with ClassDef

nicolasstucki · 2022-11-29T07:39:52Z

Why macro annotations cannot be used for ClassDef ?

That will be added in the next PR.

aherlihy

Looks good, I played around with some macro annotations and they behave how I expected. I put some clarification questions in the comments but none of them are blockers.

aherlihy · 2022-11-27T14:41:02Z

library/src/scala/annotation/MacroAnnotation.scala

+
+  /** Transform the `tree` definition and add other definitions
+   *
+   *  This method takes as argument the reflected representation of the annotated definition.


what is a reflected representation?

Seems it should just be

* This method takes as argument the annotated definition.

aherlihy · 2022-11-27T15:34:08Z

compiler/src/dotty/tools/dotc/quoted/Interpreter.scala

@@ -32,7 +32,7 @@ import dotty.tools.dotc.reporting.Message
 import dotty.tools.repl.AbstractFileClassLoader

 /** Tree interpreter for metaprogramming constructs */
-abstract class Interpreter(pos: SrcPos, classLoader: ClassLoader)(using Context):


Background knowledge question - other than SpliceInterpreter in Splicer.scala, prior to this PR, was this class used anywhere?

The splice interpreter was the only use of this class. This class was factored out in a separate PR to make this PR simpler to review.

aherlihy · 2022-11-29T13:43:38Z

compiler/src/dotty/tools/dotc/quoted/Interpreter.scala

@@ -65,7 +65,7 @@ abstract class Interpreter(pos: SrcPos, classLoader: ClassLoader)(using Context)

    // TODO disallow interpreted method calls as arguments
    case Call(fn, args) =>
-      if (fn.symbol.isConstructor && fn.symbol.owner.owner.is(Package))


Would you mind explaining why was this check removed (or actually, what was it doing before and why is it not needed anymore?)

This check was there to make sure we only called interpretNew on a top-level class. I noticed that our current implementation of interpretNew also supports classes nested in objects.

aherlihy · 2022-11-29T13:45:35Z

compiler/src/dotty/tools/dotc/quoted/Interpreter.scala

@@ -185,7 +185,8 @@ abstract class Interpreter(pos: SrcPos, classLoader: ClassLoader)(using Context)
    loadModule(fn.moduleClass)

  private def interpretNew(fn: Symbol, args: => List[Object])(implicit env: Env): Object = {
-    val clazz = loadClass(fn.owner.fullName.toString)
+    val className = fn.owner.fullName.toString.replaceAll("\\$\\.", "\\$")


Is this to handle the names generated by newUniqueMethod/Val?

No. It is to handle classes that are nested withing objects.

For example if we have the class B.A defined in object B (which has a companion class B$), we would see the name B$.A at this point in the compiler. But we lift those classes and therfore we will get a class B$A in the bytecode. This transforms B$.A into B$A.

This should probably use mangledString and not toString to account for name encoding (e.g., try definining and using class +). But also, what about inner classes? If I have class Outer { class Inner }; val o = new Outer; new o.Inner, then the binary class name is also Outer$Inner . I have logic in https://github.com/lampepfl/dotty/blob/e8428102f8372e9c4227a3a13970e15bfc5d94c1/compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala#L147-L171 to do this which could be extracted in its own method.

I will use mangleString.

We do not need to handle inner classes for now. We do not allow macro annotations to be defined as inner classes (restriction). We wont encounter new in top-level splices neither.

aherlihy · 2022-11-29T13:48:02Z

compiler/src/dotty/tools/dotc/quoted/Interpreter.scala

-        val sym = staticRef(className.toTypeName).symbol
-        if (sym.isDefinedInCurrentRun) Some(sym) else None
-      }
+    def unapply(targetException: Throwable)(using Context): Option[Symbol] = {


Is this because you may end up in a situation where the class defined by the macro annotation does not yet have a definition?

Yes, does not have a definition in bytecode.

This can happen when we compile the macro definition and use at the same time. In that case we have the definition as a symbol in the compiler but the bytecode for the macro definition has not been generated yet.

aherlihy · 2022-11-30T11:42:34Z

compiler/src/dotty/tools/dotc/transform/PostTyper.scala

@@ -483,6 +487,15 @@ class PostTyper extends MacroTransform with IdentityDenotTransformer { thisPhase
    private def normalizeErasedRhs(rhs: Tree, sym: Symbol)(using Context) =
      if (sym.isEffectivelyErased) dropInlines.transform(rhs) else rhs

+    private def checkForMacrosAnnotations(tree: Tree)(using Context) =


I'm not sure I understand the distinction between MacroAnnotations and TastyAnnotations, other than one gets checked on a tree the other gets checked on a symbol. what's the difference, conceptually?

This is a mistake on my part. It should be TastyAnnotations. I will update that.

We started with the name Macro Annotation following the Scala 2 name of the feature. But then decided we should use a different name to make it clear that they are the same. This will help users when they google documentation.

Changed back to MacroAnnotation after #16392 (comment)

aherlihy · 2022-11-30T12:41:06Z

compiler/src/dotty/tools/dotc/Compiler.scala

@@ -47,6 +47,7 @@ class Compiler {
  /** Phases dealing with TASTY tree pickling and unpickling */
  protected def picklerPhases: List[List[Phase]] =
    List(new Pickler) ::            // Generate TASTY info
+    List(new ExpandAnnotations) ::  // Execute TASTy macro annotation


Why move the annotation expansion into a separate phase now?

I pushed a commit I should not. The macro annotation expansion will be done in Inlining.

I was experimenting to see if it would be possible to extract it in the future to support some extra features.

odersky

Otherwise LGTM

odersky · 2022-12-04T18:42:39Z

compiler/src/dotty/tools/dotc/CompilationUnit.scala

@@ -160,6 +164,9 @@ object CompilationUnit {
                Feature.handleGlobalLanguageImport(prefix, imported)
            case _ =>
        case _ =>
+      for annot <- tree.symbol.annotations do
+        if annot.tree.symbol.denot != NoDenotation && annot.tree.symbol.owner.derivesFrom(defn.TastyAnnotationClass) then


Suggested change

if annot.tree.symbol.denot != NoDenotation && annot.tree.symbol.owner.derivesFrom(defn.TastyAnnotationClass) then

if annot.tree.symbol.maybeOwner.derivesFrom(defn.TastyAnnotationClass) then

Updated. Also updated the same pattern in TastyAnnotations.isTastyAnnotation

odersky · 2022-12-04T18:47:27Z

library/src/scala/annotation/TastyAnnotation.scala

+
+/** Base trait for TASTy annotation that will transform a definition */
+@experimental
+trait TastyAnnotation extends StaticAnnotation:


I think TastyAnnotation is too unclear. I would never think of a macro when seeing that name. Instead I would think it's something to do with the internals of Tasty files.

I think the best thing to do is to go back to MacroAnnotation. We will have to explain how they are different from Scala 2 macro annotations, but it's better to do that directly. Otherwise, when someone googles for Scala 3 macro annotation, they will only find Scala 2 macro annotation stuff. I don't think we can establish TastyAnnotation as an obvious term that people will search for.

Changed back to MacroAnnotation

Yearning for that content to google for ;)
The thesis linked above is a good intro; I assume the test cases (chapter 5) which were only outlined there, not provided, are the ones in these PRs? In any case, documentation or a further reading list would be appreciated in due time :) Excited for this feature!

odersky · 2022-12-05T10:08:35Z

compiler/src/dotty/tools/dotc/transform/Inlining.scala

+            && StagingContext.level == 0
+            && TastyAnnotations.hasTastyAnnotation(tree.symbol)
+          then
+            val trees = new TastyAnnotations(Inlining.this).expandAnnotations(tree)


Style: We usually prefer an explicit self name like thisPhase over a qualified Inlining.this.

Add basic support for macro annotations. * Introduce experimental `scala.annotations.MacroAnnotation` * Macro annotations can analyze or modify definitions * Macro annotation can add definition around the annotated definition * Added members are not visible while typing * Added members are not visible to other macro annotations * Added definition must have the same owner * Implement macro annotation expansion * Implemented at Inlining phase * Can use macro annotations in staged expressions (expanded when at stage 0) * Can use staged expression to implement macro annotations * Can insert calls to inline methods in macro annotations * Current limitations (to be loosened) * Can only be used on `def`, `val`, `lazy val` and `var` * Can only add `def`, `val`, `lazy val` and `var` definitions Based on: * scala#15626 * https://infoscience.epfl.ch/record/294615?ln=en

nicolasstucki · 2022-12-07T12:38:16Z

Rebased and fixed conflicts in the Interpreter.

Enable modification of classes with `MacroAnnotation`: * Can annotate `class` to transform it * Can annotate `object` to transform the companion class Supported class modifications: * Modify the implementations of `def`, `val`, `var`, `lazy val`, `class`, `object` in the class * Add new `def`, `val`, `var`, `lazy val`, `class`, `object` members to the class * Add a new override for a `def`, `val`, `var`, `lazy val` members in the class Restrictions: * An annotation on a top-level class cannot return a top-level `def`, `val`, `var`, `lazy val`. Related PRs: * Includes #16513 * Follows #16392 * Followed by #16534

mrdziuban · 2022-12-20T00:12:07Z

@nicolasstucki thank you for all your work on this! Apologies if this isn't the best place to ask, but I was testing out the @memoize example and wondered if it's possible to reference the generated val fibCache from external code? e.g. in the SBT console, I tried

scala> @memoize def fib(n: Int): Int = {
     |   println(s"compute fib of $n")
     |   if n <= 1 then n else fib(n - 1) + fib(n - 2)
     | }
def fib(n: Int): Int

scala> fibCache
-- [E006] Not Found Error: -----------------------------------------------------
1 |fibCache
  |^^^^^^^^
  |Not found: fibCache

nicolasstucki · 2022-12-20T06:33:46Z

It is not possible to refer to the generated code from external code. This is by design.

smarter · 2022-12-20T12:56:58Z

@mrdziuban see also the discussion in https://contributors.scala-lang.org/t/scala-3-macro-annotations-and-code-generation/6035

nicolasstucki self-assigned this Nov 22, 2022

nicolasstucki force-pushed the add-def-macro-annotations branch 11 times, most recently from 7e5a5c3 to 74d7433 Compare November 24, 2022 09:36

nicolasstucki marked this pull request as ready for review November 24, 2022 11:32

nicolasstucki requested review from odersky and aherlihy November 24, 2022 13:04

nicolasstucki assigned odersky and aherlihy and unassigned nicolasstucki Nov 24, 2022

nicolasstucki force-pushed the add-def-macro-annotations branch 3 times, most recently from 0cc6b48 to fe4769a Compare November 28, 2022 13:57

nicolasstucki changed the title ~~Macro annotation (part 1)~~ Tasty macro annotation (part 1) Nov 28, 2022

nicolasstucki force-pushed the add-def-macro-annotations branch from fe4769a to a603341 Compare November 30, 2022 09:20

nicolasstucki changed the title ~~Tasty macro annotation (part 1)~~ TASTy macro annotation (part 1) Nov 30, 2022

nicolasstucki force-pushed the add-def-macro-annotations branch from a603341 to 33d1889 Compare November 30, 2022 09:25

aherlihy approved these changes Nov 30, 2022

View reviewed changes

nicolasstucki force-pushed the add-def-macro-annotations branch 2 times, most recently from 33d1889 to 3ff58dc Compare December 1, 2022 08:10

smarter mentioned this pull request Dec 1, 2022

REPL: Fix crash when printing instances of value classes #16393

Merged

nicolasstucki force-pushed the add-def-macro-annotations branch from 3ff58dc to 6853a75 Compare December 1, 2022 10:24

nicolasstucki mentioned this pull request Dec 1, 2022

Macro annotations class modifications (part 2) #16454

Merged

odersky requested changes Dec 5, 2022

View reviewed changes

nicolasstucki force-pushed the add-def-macro-annotations branch 3 times, most recently from ea3c06c to 07e395d Compare December 5, 2022 13:36

nicolasstucki changed the title ~~TASTy macro annotation (part 1)~~ Macro annotation (part 1) Dec 5, 2022

nicolasstucki requested a review from odersky December 5, 2022 16:26

nicolasstucki added 2 commits December 7, 2022 13:32

Add newUniqueVal/newUniqueMethod

9e47ba8

nicolasstucki force-pushed the add-def-macro-annotations branch from 07e395d to 19e37c8 Compare December 7, 2022 12:37

smarter mentioned this pull request Dec 7, 2022

SIP-50 - Struct Classes scala/improvement-proposals#50

Closed

odersky approved these changes Dec 12, 2022

View reviewed changes

odersky merged commit eaff261 into scala:main Dec 12, 2022

odersky deleted the add-def-macro-annotations branch December 12, 2022 11:17

Kordyjan added this to the 3.3.0-RC1 milestone Dec 12, 2022

ebruchez mentioned this pull request Dec 12, 2022

Upgrade to Saxon 12 orbeon/orbeon-forms#976

Open

nicolasstucki mentioned this pull request Dec 23, 2022

Basic Design of Macro Annotations #15626

Closed

armanbilge mentioned this pull request Mar 11, 2023

Mononcle doesn't support @Lenses in Scala3. optics-dev/Monocle#1337

Open

ckipp01 mentioned this pull request Jun 5, 2023

Support for macro annotations lampepfl/dotty-feature-requests#278

Closed

Kordyjan modified the milestones: 3.3.0-RC1, 3.3.0 Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Macro annotation (part 1) #16392

Macro annotation (part 1) #16392

nicolasstucki commented Nov 22, 2022 •

edited

Loading

jilen commented Nov 29, 2022 •

edited

Loading

nicolasstucki commented Nov 29, 2022

aherlihy left a comment

aherlihy Nov 27, 2022

nicolasstucki Dec 1, 2022

aherlihy Nov 27, 2022

nicolasstucki Dec 1, 2022

aherlihy Nov 29, 2022

nicolasstucki Dec 1, 2022

aherlihy Nov 29, 2022

nicolasstucki Dec 1, 2022

smarter Dec 1, 2022 •

edited

Loading

nicolasstucki Dec 1, 2022

aherlihy Nov 29, 2022

nicolasstucki Dec 1, 2022

aherlihy Nov 30, 2022

nicolasstucki Dec 1, 2022

nicolasstucki Dec 5, 2022

aherlihy Nov 30, 2022

nicolasstucki Dec 1, 2022

odersky left a comment

odersky Dec 4, 2022

nicolasstucki Dec 5, 2022

odersky Dec 4, 2022

nicolasstucki Dec 5, 2022

simlei Jan 31, 2023

odersky Dec 5, 2022

nicolasstucki Dec 5, 2022

nicolasstucki commented Dec 7, 2022

mrdziuban commented Dec 20, 2022

nicolasstucki commented Dec 20, 2022

smarter commented Dec 20, 2022

	if annot.tree.symbol.denot != NoDenotation && annot.tree.symbol.owner.derivesFrom(defn.TastyAnnotationClass) then
	if annot.tree.symbol.maybeOwner.derivesFrom(defn.TastyAnnotationClass) then

Macro annotation (part 1) #16392

Macro annotation (part 1) #16392

Conversation

nicolasstucki commented Nov 22, 2022 • edited Loading

Add basic support for macro annotations

Example

Based on

Followed by

jilen commented Nov 29, 2022 • edited Loading

nicolasstucki commented Nov 29, 2022

aherlihy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarter Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasstucki commented Dec 7, 2022

mrdziuban commented Dec 20, 2022

nicolasstucki commented Dec 20, 2022

smarter commented Dec 20, 2022

nicolasstucki commented Nov 22, 2022 •

edited

Loading

jilen commented Nov 29, 2022 •

edited

Loading

smarter Dec 1, 2022 •

edited

Loading