Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

non-default database #18

Open
derrickoswald opened this issue Aug 21, 2019 · 2 comments
Open

non-default database #18

derrickoswald opened this issue Aug 21, 2019 · 2 comments

Comments

@derrickoswald
Copy link
Owner

The CIM RDD are currently added to the default SparkSQL database:

scala> spark.sql ("show databases").show
+------------+
|databaseName|
+------------+
|     default|
+------------+

It would be good if the database could be specified.

  • how to communicate the desired database to the CIMReader
  • it would need to be created if it doesn't exist
  • code in the CIMReader would need to use the database if it wasn't the default
@mbheinen
Copy link
Collaborator

mbheinen commented Sep 24, 2019

Are you thinking the desired database would be a run time argument to the existing docker-compose scripts for sandbox and beach?

@derrickoswald
Copy link
Owner Author

It isn't an "instance" level thing, more of a "read" level option IMHO.

The spot to inject it is probably in CIMRelation.

So the developer visible entry point would be something like:

import scala.collection.mutable.HashMap
import org.apache.spark.rdd.RDD
import ch.ninecode.cim._
import ch.ninecode.model._

val opts = new HashMap[String,String]()
opts.put ("ch.ninecode.cim.spark_database", "my_database")
val element = spark.read.format ("ch.ninecode.cim").options (opts).load ("hdfs://sandbox:8020/data/CGMES_v2.4.15_RealGridTestConfiguration_EQ_v2.xml")

which would use my_database instead of default.

The database view is created in CIMSubsetter, but it's unclear how one specifies the database to use other than the "default".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants