Environment

• GraphFrames version: 0.11.0
• Spark version: 3.5.5
• Scala version: 2.12 (standard EMR/Glue runtime)

Description

TwoPhase.runAQE (Connected Components with AQE enabled) throws an AnalysisException due to conflicting attributes on a self-referential join in the small-star step.

The non-AQE path (TwoPhase.run with spark.sql.adaptive.enabled=false) works correctly on the same data.

Error

org.apache.spark.sql.AnalysisException: Failure when resolving conflicting references in Join:
 Conflicting attributes: <>.
   at org.graphframes.lib.TwoPhase$.runAQE(TwoPhase.scala:385)

Reproduction

import org.graphframes.GraphFrame

 // Any graph large enough to require multiple CC iterations triggers this.
 // Minimal repro: a graph where connected components cannot converge in one iteration.
 val edges = spark.createDataFrame(Seq(
   (1L, 2L), (2L, 3L), (3L, 4L), (4L, 5L), (5L, 1L),
   (6L, 7L), (7L, 8L), (8L, 6L)
 )).toDF("src", "dst")

 val vertices = edges.select("src").union(edges.select("dst")).distinct().withColumnRenamed("src", "id")
 val g = GraphFrame(vertices, edges)

 // This fails:
 spark.conf.set("spark.sql.adaptive.enabled", "true")
 val result = g.connectedComponents.run()  // uses runAQE path internally

(Note: on small toy graphs the algorithm may converge in one iteration and not hit the failing join. Use a graph with multiple components and enough edges to force multiple iterations.)

Root Cause Analysis

The failure is in the small-star step at TwoPhase.scala:385:

  val minNbrs2 = ee.groupBy(col(SRC)).agg(min(col(DST)).as(MIN_NBR))
  ee = ee.join(minNbrs2, SRC)  // <-- self-join fails here

minNbrs2 is derived from ee, so both sides of the join share attribute src#2690L. Spark's DeduplicateRelations analyzer rule is responsible for detecting self-joins and renaming conflicting exprIds on one side.

• With AQE disabled (TwoPhase.run): The logical plan structure allows DeduplicateRelations to successfully identify and rename the conflicting nodes.
• With AQE enabled (TwoPhase.runAQE): The plan shape differs — intermediate nodes (like Deduplicate/Distinct introduced by the large-star step's projections) are not handled by the dedup rule, so conflicting attributes propagate unresolved into the join.