cookiecutter-keplerpython

Cookiecutter template for a Kepler Workflow that runs a Python script within the Python Actor. The workflow is compatible with CRBS Workflow Service See: https://github.com/audreyr/cookiecutter.

Usage

Generate a Kepler Workflow project:

cookiecutter https://github.com/slash-segmentation/cookiecutter-keplerpython

Generated Source Tree Structure

The above will create a new workflow source tree that looks like the following:

  README.md
  src/
      (repo name).kar
  test/
       README.md
       successful_run.bats
       test_helper.bash
       bin/
           command
        

• src/(repo name).kar

  • This is the actual Kepler workflow. The file name is set to the value set for repo_name when running cookiecutter to generate the source tree.

• test/

  • Contains bats unit test to run the Kepler workflow via the command line and verify correct operation.

Workflow

The workflow will look like the one in the screenshot below and include a single python actor redirected to a display actor

Python Actor

Double clicking on the Python Actor will display Python code. When run the Kepler workflow calls the fire method, an tiny excerpt is shown below:

# This is a simple actor that copies the input to the output.
# You can remove the ports, add new ports, and modify the script.

import ptolemy.data
import time
import os


class Main :
  """Skeleton Actor compatible with CWS"""
  def fire(self) :
    """Skeleton implementation that follows best practices for CWS"""

    .
    .
    .
    # see if mycmd input port has a token
    input_val = ''
    if self.mycmd.numberOfSources() > 0:
      input_val = self.mycmd.get(0).stringValue()
    .
    .
    .
    # fire output token
    self.output.broadcast(ptolemy.data.StringToken(input_val))

    return
.
.
.

When opening the actor there will be additional code, this code provides utility methods that generate files (WORKFLOW.FAILED.txt, README.txt, workflow.status) used by CRBS Workflow Service

Python Actor parameters

The Python Actor has some parameters set within as seen in the image below (this menu can be seen by right clicking on the actor and choosing Configure Actor):

The following parameters are set:

• cws_outputdir
• cws_user
• cws_jobname
• cws_jobid
• cws_workflowname
• cws_notifyemail

From within the fire method and Main class these variables are accessible by this Python code:

# gets value of cws_outputdir parameter 
foo = self.cws_outputdir.stringValue()

Python Actor input/output ports

The Python Actor supports input and output via Ports. Right clicking on the Python Actor and clicking on Configure Ports will show the configured ports. Below is a screenshot of the ports configured for the workflow generated by this template:

Currently there are two import ports mycmd and text and one output port output Access to these can be made from within fire method and Main class via this Python code:

# call self.(portname).numberOfSources() to see if there is a token to retrieve
if self.mycmd.numberOfSources() > 0:
  # self.(portname).get(#) gets the token, the .stringValue() converts it to a string
  foo = self.mycmd.get(0).stringValue()

bar = None
if self.text.numberOfSources() > 0:
  bar = self.text.get(0).stringValue()


# writing can be done via self.(portname).broadcast()
self.output.broadcast(ptolemy.data.StringToken('hello'))

Developer Topic: Updating workflow in this template

This section describes the best approach to updating the Kepler workflow in this template. This is not straight forward since this template constructs the kar file on the fly due to limitations with Cookiecutter and its internal template engine Jingja

Structure of a Kepler workflow kar file

Kepler 2.4+ workflows are stored in .kar files which are actually Zip files with the following structure:

 workflowname.urn.lsid.kepler-project.org.ns..#####.###.###.xml
 META-INF/
          MANIFEST.MF

How this template works

This template uses a feature of Cookiecutter to invoke a script after initial token replacement and source tree creation. The script is: hooks/post_gen_project.py which recreates the kar file from data stored in {{cookiecutter.repo_name}}/src

Step 1 Creating Workflow to edit

The xml file within {{cookiecutter.repo_name}}/src has {{ }} replacement tokens which prevent Kepler from loading the workflow. To get a workflow that can be edited, simply run Cookiecutter on this template to create a kar file. Use the defaults Cookiecutter when creating this repo since they will be needed in Step 3 below.

Once the source tree is created, changes can then be made to the kar file. Be sure to save kar.

Step 2 Extracting xml from kar and copying to template source tree

Make a temp directory, copy kar file and rename suffix to .zip as seen with example myworkflow.kar below:

mkdir tmp
cp myworkflow.kar tmp/myworkflow.zip
cd tmp
unzip myworkflow.zip
cp myworkflow.urn.* ~/src/cookiecutter-keplerpython/\{\{cookiecutter.repo_name\}\}/src/{{cookiecutter.repo_name}}.urn.lsid.kepler-project.org.ns..70097.362.547.xml

Step 3 Adjusting tokens in xml file

Using a text editor (Vi, emacs, etc..) replace the following in the xml file copied over in Step 2:

Repo Name

myworkflow

with

{{cookiecutter.repo_name}}

Full Name

Christopher Churas

with

{{cookiecutter.full_name}}

Project Name

NOTE: There are two places this will need to be replaced

My Workflow

with

{{cookiecutter.project_name}}

Save the xml file

Step 4 Testing

Try Cookiecutter on the new template and then try to load the workflow via Kepler.

If Cookiecutter fails odds are there is text in the xml file that matches the token replacement signature used by [Jinja][jinga]. A workaround is to find the problem {{ characters and escape them using this technique: http://jinja.pocoo.org/docs/dev/templates/#escaping

Step 5 Fixing LSIDs

Kepler gives each workflow and everything it touches a unique identifier (LSID). This template lacks access to Kepler code to generate new LSIDs. To get around it. One should load Kepler and open the workflow generated from this template, then save the workflow with a new name, then save again with the old name. I know its weird, but it appears to work.