Pipes for MarkLogic DataHub

What is Pipes?

A visual programming tool for MarkLogic. It integrates with the MarkLogic Data Hub and produces the code for a Custom Step using a no-code UI environment.

Pipes for MarkLogic DataHub is a community tool. As such, it is not supported by MarkLogic Corporation and is only updated and corrected based on a best-effort approach. Any contribution or feedback is welcomed to make the tool better.

Prerequisites

Pipes is designed to run on MarkLogic 10.0-2 and higher.

It assumes that:

• A DHF project (5.1.0 and higher) is present on the local file-system
• That DHF project is installed to your MarkLogic server
• Java 8 or higher installed

Great! How do I run it?

To use Pipes you will need to have an instance of MarkLogic with the Data Hub installed.

• Download the Pipes jar from the Releases section.
• Copy the jar to the DHF folder root you plan to use Pipes for.

Run pipes from the command line:

java -jar marklogic-pipes-1.2.jar

As soon as Pipes starts, it will print out the port number it’s running on. Example, Pipes will be available on: localhost:8080 if the port is 8080.

Pipes options (command line parameters)

Pipes has several properties that can be used on the command line, when starting the Pipes jar.

Alternatively, you can put these into an application.properties file in the same directory where you’ve copied the Pipes jar to.

Gradle environment name

environmentName

If you have multiple environments in your DHF project, you can tell Pipes which one to use.

Example:

java -jar marklogic-pipes-1.2.jar --environmentName=production

will first read gradle.properties of the project and then gradle-production.properties

Pipes port

server.port

Pipes will run on a first availble port it finds counting from 8000. If you want to specify your own port, use this parmeter

DHF Project Root

mlDhfRoot

If you want to run Pipes from another folder (i.e. you haven’t copied the Pipes jar to the DHF root project folder), you can to use this parameter to tell Pipes where to look for the DHF project.

Example:

java -jar marklogic-pipes-1.2.jar --mlDhfRoot=/users/user/dev/test-pipes

or, on Windows:

java -jar marklogic-pipes-1.2.jar --mlDhfRoot=C:/Users/user/dev/test-pipes

Open a graph automatically on Pipes log-in

You can tell Pipes to load a graph saved in the database by using this parameter

loadGraph

Example:

java -jar marklogic-pipes-1.2.jar --loadGraph=mygraph

will load a graph saved as “mygraph to the database as soon as you log in to Pipes.

Backend modules

Pipes requires several backend modules (libraries) to be installed on the MarkLogic server, both when running the Pipes UI to design your custom code and when running the custom code within the DHF flow, later.

They will be installed automatically when Pipes runs.

In case when you have 2 environments - one where you use the Pipes UI and another where you run the custom steps designed by Pipes, you will need these modules installed on both environments.

Pipes modules can be separately installed using mlBundle. To use the Pipes modules bundle in your DHF project, add the following to build.gradle:

```
repositories {
    maven {url 'https://dl.bintray.com/marklogic-community/Maven/'}
}

dependencies {
    mlBundle "com.marklogic:pipes-modules:1.2"
}
```