Virtualenv¶

Most likely you will want to install into a virtualenv to keep your code separate from the system’s python packages.

This portion is completely optional

python setup.py bootstrap_virtualenv venv --prompt="(redscheduler)"
source venv/bin/activate

Install redscheduler¶

pip install -r requirements.txt
python setup.py install

For python 2.6 you will need to also install ordereddict

pip install ordereddict

Local Config File¶

You will need to ensure that you have setup your config file which needs to be located under your home directory as .redscheduler.config

If you are already familiar with the yaml format, then you will notice that the config file utilizes yaml. If you are not familiar with yaml format, then you may want to go read about that here

There is an example config file you can copy as your template in the repository called redscheduler.config.example that contains comments about what each configuration item means.

{ISSUEDIR}

get_dir_listing:
    cli: "ls -l"

/path/to/foo
/path/to/bar

jobdefs:
    example:
        ...
        uploads:
            - /tmp/bob.txt
            - {ISSUEDIR}/../../etc/passwd

jobdefs:
    example:
        ...
        uploads:
            - {ISSUEDIR}/stdout.txt
            - {ISSUEDIR}/stderr.txt
            - {ISSUEDIR}/foo/myoutputfile.txt

Redmine Setup¶

The redmine setup is quite simple.

Remine Job Issues¶

Think of each issue in Redmine as a Job instance that will be run. So if you make 100 issues in redmine, then 100 jobs will be run on the computer

Essentially the description portion of each issue defines the arguments that are combined with the local config’s job_def’s cli.

You can also attach files to the issue and reference them in the description of the issue like this attachement:foo.txt

Full Example¶

Assume we are using the example config(redscheduler.config.example) that comes with this project.

• output_directory is set to /tmp

We want to define a job that will convert comma separated files to tab separated files and then upload the result back to the issue when it is completed.

We know that in the shell you can simply use the tr command to do translate output by by substituting the tab \t character with a comma ,. Here is the shell command that would do this:

tr ',' '\t' < /path/to/input.csv

Now that we know how the shell can do it, we will need to convert it to a job_def + Redmine issue.

First, what will remain the same every time we run the command?:

tr ',' '\t' <

We know from reading CLI Example that you cannot have the < in the job_def’s cli field so we will need to make a shell script that we will use instead.

mkdir -p ~/jobdefscripts
cat <<EOF > ~/jobdefscripts/replace-comma-with-tab.sh
#!/usr/bin/env bash
tr ',' '\t' < \$1 > output.tsv
EOF
chmod 755 ~/jobdefscripts/replace-comma-with-tab.sh

You can see that the shell script is setup to direct the first argument given to it $1 into the tr command and then direct the output into a file called output.tsv

Now, we can define our job in our ~/.redscheduler.config:

job_def:
    ...
    replace-comma-with-tab:
        cli: "/home/username/jobdefscripts/replace-comma-with-tab.sh"
        uploads:
            - {ISSUEDIR}/output.tsv
            - {ISSUEDIR}/stderr.txt

Make sure to create the tracker called replace-comma-with-tab

Now you can create jobs to run this task easily. You upload your files you want converted to the issue and reference them in the issue description.

Here would be an example description:

attachment:my.csv

Assuming, you have uploaded the attachment named my.csv to the issue when you created it.

Now, when the job runs, you can expect the following to happen

1. /tmp/Issue_# is created for you(where # is replaced with the issue id)

2. /tmp/Issue_#/output.tsv and /tmp/Issue_1/stderr.txt are created with output from running your command

   In this case, stdout.txt should be empty as you are redirecting output

   into output.tsv

   If your command generates errors, you will get output in stderr.txt

3. /tmp/Issue_#/output.tsv is created with the output from your command.

4. Additionally, your command is run within the /tmp/Issue_# directory.

   If any relative paths are used for files, they will be relative to that directory

5. When the task runs it will change the issue status from New to In Progress to Completed or Error.

6. When the task completes it will upload output.tsv and stderr.txt back to the issue

Create your first issue¶

Here we can see that issue #10243 was created. We will need this number to run the issue.

Run the issue¶

$> runissue 10243
Updated issue status with output location and set to In Progress
Creating /tmp/Issue_10243 to run in
Running echo -e foo
bar bar foo
Return code was 0
Saving issue with new status Completed

Check output file contents¶

You can see that the issue wrote all standard output into the defined stdout file, output.stdout.

$> cat /tmp/Issue_10243/output.stdout
foo
bar bar foo

Additionally, stderr was empty(Size: 0)

$> stat /tmp/Issue_10243/stderr.txt
  File: `/tmp/Issue_10243/stderr.txt'
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 803h/2051d  Inode: 356         Links: 1
Access: (0644/-rw-r--r--)  Uid: (500/username)   Gid: (500/username)
Access: 2015-07-08 14:49:24.369910594 -0400
Modify: 2015-07-08 14:49:24.369910594 -0400
Change: 2015-07-08 14:49:24.369910594 -0400

Verify issue¶

Here you can see that the issue was first updated to show that it was running by changing the status from New to In Progress. It also creates a note showing the output location that the job will be run in(/tmp/Issue_10243)

After the job completes it updated the issue again, changing the status from In Progress to Completed. Also, because our config has uploads defined, the output.stdout file was uploaded back to the issue.