LONI Pipeline

1. Server library
2. Personal library
3. Workflow area
4. Connection manager
5. Provenance editor
6. Preferences
7. Search feature
8. Checking for latest updates
9. Running from the command line

3.1 Server library

When you launch the Pipeline you’ll notice the area to the left is empty. This is the server library where you can see a list of all the modules available to you when you connect to different servers. Since this is your first time launching the program, you have nothing listed there, but we’ll connect to a server soon to get access to some tools.

If you want to gain access to the tools LONI has made available through its Pipeline server apply for an account and after connecting to the server (cranium.loni.ucla.edu) you will see the library populated with all the tools available on the server.

Server library supports workflows from multiple servers. Now you can easily sort library items “by Module Type” or “by Server address”. Also you can manually right click and call Update library function which will automatically connect to servers included in your current server library and update the Server Library Content on your local computer. If you remove your library content and run Update library command, Pipeline will check for your current connections and will update only from servers you are connected to.

Even after you disconnect from a Pipeline server, the list of tools on that server will still remain cached on your computer so you can construct workflows while you are offline.

3.2 Personal library

The Personal library can be accessed by going to ‘Window->Personal Library.’ This is where your library of workflows and personally created modules are stored. Simply select a directory (or use the default one specified in the preferences) and save all your personal modules and workflows in there. Then when you open up your personal library, you’ll see all the modules in there, and you can drag them in or open up copies of them quickly and easily.

Pipeline 4.2 allows you to save and open workflows directly to/from Personal library, even without knowing the location of it. Click File -> Save To -> Personal library… to save to personal library and File -> Open From -> Personal library… to open from personal library.

The main feature of the personal library is that it can be used in just the same way as the server library to construct workflows. If you have defined lots of modules that describe executables on your local computer or even module groups, you can drag in copies of them to create even more sophisticated workflows.

3.3 Workflow area

The large area with a faucet logo on it is the workflow area. You can open a workflow or start a new workflow from here by clicking on one of the buttons at the center of the area.

You can also see a list of recently accessed workflows, but that’s not visible because we don’t have any recently used workflows yet.

3.4 Connection manager

Often times you will want to connect to different Pipeline servers to get access to their tools and build workflows out of them. To bring up your list of connections, go to the ‘Window’ menu and click on ‘Connections…’. Alternatively, you can click on the disconnected circles at the bottom right of the window, and in the popup menu click on ‘Connections…’.

By default, Pipeline has LONI server (cranium.loni.ucla.edu) as saved connections. You need to apply for an account to actually connect to it. You can click Edit and put your LONI username, and then click Save. Once you’ve entered the connection, go ahead and click ‘Connect’. After 30 seconds or so you’ll notice that your server library has been populated with tools from the server. By clicking New, you can add a connection to any Pipeline server that you want to access.

3.5 Provenance editor

Since LONI Pipeline version 4.2, provenance files can be generated along with your output files. By default, provenance generation not enabled. To enable this, make sure you have the latest version of Pipeline, open the workflow, select File -> Workflow properties, then check Generate provenance files.

Pipeline 4.2 includes a provenance manager, which enables tracking data, workflow and execution history of all processes. This functionality improves the communication, reproducibility and validation of newly proposed experimental designs, scientific analysis protocols and research findings. This includes the ability to record, track, extract, replicate and evaluate the data and analysis provenance to enable rigorous validation and comparison of classical and novel design paradigms. Provenance editor allows you to view and edit provenance. To open provenance editor, you can click Tools -> Provenance Editor, and open a provenance file (.prov). If provenance was enabled for the workflow, you can also open the provenance file using the provenance editor by right-click finished module -> Execution Logs -> Output Files tab -> select the file with .prov extension, and click Edit Provenance on lower right.

3.6 Preferences

Bring up the preferences dialog:
OS X: Go to the ‘LONI Pipeline’ menu and select ‘Preferences.’
Windows: Go to ‘Tools’->’Options’
Linux/Unix: Go to ‘Edit’->’Preferences’

General Tab:

• Personal Library is where your library of workflows and personally created modules are stored.
• Email Notification: you can put your email address so that the Pipeline server will send you an email notification once your workflow has been completed.
• Automatically check for available updates: if checked, Pipeline will check for update every time it starts.
• Animate Server Library: if selected, server library will hide automatically when workflow is executing, and will show up automatically when workflow is stopped.
• Enable Smartline when designing workflows: If enabled, Pipeline will create translation module between image files automatically when making connections. You can always enable Smartline by holding the Shift key while dragging connections.

Cache Tab:

• The execution cache directory is where all intermediate data is written to while executing a workflow.
• The library module cache can be cleared here (similar to a web browser cache).

Local Execution Tab:

• Maximum simultaneous jobs is the number of jobs that will run in parallel when executing a workflow that has many jobs that are ready to submit at once. Once this limit is reached, the other jobs will be queued up until other ones complete.
• Use Grid Plugin, if you have a grid manager locally, checking this will enable your local workflows sent to the grid manager. You must specify your grid plugin jar file path and the class name. Please check Grid Plugin API Developers Guide and Server Guide for more information.
• Environment Variables for your local execution should be defined here.

3.7 Search feature

The Search feature in both the Library and Personal Library panels allow you to query for the modules that reside in the respective menu panes. The Search function will return results drawn from the module’s name, author list, citations, tags, description, and parameter fields.

3.8 Checking for latest updates

In the Help menu, you can check to see if you have the latest version of the Pipeline client by clicking on “Check for Updates…”. Pipeline will inform you if there is a newer version available, and you can download and install it by following the instructions.

3.9 Running from the command line

Sometimes you may not want to use the graphical interface to the Pipeline (you may have several workflows that have minor differences in them, but you want to run them all at once). To launch the Pipeline’s command line interface, you need to go into the directory where your Pipeline.jar file is located:

• OS X Iinside the OS X bundle at LONI Pipeline.app/Contents/Resources/Java.
• Windows In your install location, which is usually C:\Program Files\LONI\Pipeline\.

Once inside that directory, you can simply launch it with the following: (Linux/Unix users can take advantage of the startCLI.sh script included with the download)

java -cp Pipeline.jar ui.cli.Main [OPTIONS] PIPEFILE [paramID1=value1 ... paramIDN=valueN]

Below is a detailed explanation for each argument:

[OPTIONS] lets you specify how you want Pipeline to run your workflow, these are the flags:

• -execute validates then executes the specified workflow.
• -validate just validates the specified workflow and exits.
• -display displays all the required/enabled parameters in the workflow.
• -sync program will not exit until workflow finishes executing. This flag is automatically enabled for a workflow that executes locally.
• -credentials=FILE specify an XML file containing credentials to a Pipeline server that will be needed for execution or validation. Format is the following:
  <?xml version="1.0" encoding="UTF-8"?>
<connections>
<conn address="cranium.loni.ucla.edu" user="jane" password="secret" />
</connections>
• -help prints out usage help
• -version prints out the LONI Pipeline version

PIPEFILE is the path of the .pipe file you wanted to run.

[paramID1=value1 ... paramIDN=valueN] lets you bind values to different parameters in the workflow. All bindings specified on the command line are in addition to any values already bound to them. Multiple bindings to the same parameter id will also be added onto the same parameter.
Some example bindings:
MyModule.OutputFile_0=pipeline://localhost//home/user/Desktop/myData.nii.gz
YourModule.InputImage_0=pipeline://localhost/C:\data\yourData.nii.gz
HerModule.InputAtlas_1=pipeline://cranium.loni.ucla.edu//usr/local/data/atlas1.img