LONI Pipeline

1. Validation
2. Executing a workflow
3. Client Disconnect/Reconnect
4. Server status
5. Stopping a workflow
6. Module statuses
7. Viewing output
8. Debugging execution
9. Report a Bug

5.1 Validation

When you execute a workflow, the first thing the Pipeline does is validate it and check for errors. When the Pipeline does validation checks, modules show “Validating” status. You can also do that without actually executing the workflow. To start the stand alone validation go to Execution->Validate, and validation will automatically begin. If a connection is needed to a server the Pipeline will prompt you for a username and password. If any errors are found a dialog will pop up listing all the errors found in the workflow.

If your workflow is very large, you may want to run validation periodically on it as you’re building to check for errors early on.

5.2 Executing a workflow

Once you’ve completed editing your workflow, you can execute the workflow by simply clicking on the ‘Play’ button at the bottom of the workflow area. If the program needs a connection to a server, it will prompt you for a username and password. If you’ve already stored a username in your list of connections and already entered password on previous runs, then it will automatically connect for you.

Once all necessary connections have been made and validation has completed the workflow will begin to execute.

Depending on the result of the execution, and how busy the Pipeline server is, you may see several statuses for these modules:

• Initializing – Appears when instance is preparing for submission
• Queued – Appears when instance already submitted the job and it is on the queue
• Running – Appears when job starts execution
• Complete – Appears when this module finishes executing successfully
• Error – Appears when this module has error(s). Since version 4.2, Pipeline will continue to run whole workflow execution for succeeded instances after marking failed instances.
• Backlogged – Appears when maximum job count has been reached for current server and the server temporary blocked the instance until new slots available
• Staging – Appears when transferring files to/from server from/to localhost
• Incomplete – Appears when module has finished its execution and some instances failed
• Cancelled – Appears when parents of current module failed and continuation of current module became pointless

These statuses give more detailed information about the modules. The status of a particular module is shown next to the module on the workflow area. Hover the mouse on the module, a popup detailed box will appear. It is also shown in execution logs with different colors that are appropriate to their statuses.

5.3 Client Disconnect/Reconnect

Once your workflow is running on the Pipeline server, it will continue to run even though you quit Pipeline or shut down your computer. This is helpful in the case that you want to start a workflow and then check the progress on a different computer (i.e. if you start the workflow at work and want to check on the results from home). After you have pressed play on the workflow and it is executing, quit out of the Pipeline. Make sure that you do NOT press stop, otherwise the workflow will stop running. Your workflow continues to execute even though the window is no longer open. To see the executing workflow again, start up the Pipeline client and use the Connection Manager to connect to the server on which you are running the workflow. Then go to Window -> Active Sessions on the top menu, a dialog with a list of all your active workflows will pop up. You can see the file name of your workflow, start time, finish time (if it’s still running, it will show “Running”). You can select any workflow in the list and click Reconnect. Your workflow will be open on the canvas with its latest status.

Note that workflows older than 15 days will be automatically removed from Active Sessions list. You can remove it manually as well, you can either press the Reset button when it’s open on the main window, or click “Remove” on the Active Sessions dialog.

5.4 Server status

When you start your workflow, it is being submitted to the Pipeline server and queued up. Pipeline 4.2 gives information about the status of the server(s) connected. Information about how busy the server is, how many total slots are available and how many jobs are currently being executed and queued on the server. All this information appears at the bottom right corner of Pipeline window when you connect to the server. For those who set up their own Pipeline servers, please refer to the Server Guide: Configuration section for more information on how to enable this feature on your server.

5.5 Module statuses

While a workflow is executing, you can press the Stop button if desired. If you press the Stop button, then execution of the workflow is permanently stopped. There is no way to resume execution of the workflow at the point when you pressed Stop. When a workflow is stopped, modules show an “Incomplete” status.

5.6 Stopping a workflow

While a workflow is executing, you can press the Stop button if desired. If you press the Stop button, then execution of the workflow is permanently stopped. There is no way to resume execution of the workflow at the point when you pressed Stop. When a workflow is stopped, modules show an “Incomplete” status.

5.7 Viewing output

As the modules continue executing you can view the output and error streams of any completed module. You can bring up the log viewer by going to Window->Log Viewer or more easily, right-clicking on the module that you want to view information about and click on ‘Execution Logs.’ This will bring up the log viewer and set its focus on the module that was clicked.

Once the log viewer is open, in the left hand column you can select the instance of the module that you want to view output for. In our example, we only have one instance, but if you provided multiple inputs to a single parameter of a module you would see just as many instances in the viewer’s left hand column.

When you select a particular instance, you will see information about the server that it resides on in the ‘Info’ tab on the right hand side. An important piece of information you’ll see if the command string that was submitted to the server to execute that instance of the module. You may want to take advantage of that piece of knowledge if you’re ever trying to debug a module definition or trying to determine the cause of a problem in your Pipeline server setup.

The output and error log tabs each contain the data captured from the application’s output and error streams, respectively. The output files tab contains a list of all the files created by that instance of the module, and allows you to download them to your local system, by selecting the files you want and clicking ‘Get Files.’ If you want to get all the output files of all the instances of a module, select all the instances you want in the left-hand column, then select all the output files in the right-hand tab and click ‘Get Files.’

Starting from version 4.2 Pipeline generates provenance files for outputs. By clicking the “Edit Provenance” button in Output Files tab, Pipeline will download and show in Provenance Editor the selected Provenance file.

5.8 Debugging execution

Inevitably, some of the instances (or all of them) of a module will fail sometimes and the module will have a red ring around it denoting the failure. In this case, using the log viewer as mentioned in the previous section will show all the failed instances of the module highlighted in red. With the information from the output and error stream you can diagnose nearly all the problems you may encounter while executing a workflow.

5.9 Report a Bug

If you find a bug in the Pipeline, you can file a bug report through the Pipeline client. Select Help -> Report a Bug from the top menu bar. If desired, fill out the optional fields for name, email and Pipeline server username. You can also attach the workflow being processed and enter in any details about the bug. Please be as specific as possible in your bug description. Submitting the form will send the Pipeline team an email with all of the information, allowing us to debug your problem.