fnndsc/pl-shexec repository overview

⁠pl-shexec

Run a command over every file of a tree, as a ChRIS plugin; and the pattern for turning any command-line program into one.

pl-shexec is a ChRIS <https://chrisproject.org/>_ DS plugin that does two things. In a pipeline it is the plugin for the jobs too small to deserve one of their own: convert every image, zip every directory, rename, copy one file out of each subject, count lines. It takes a command template, runs it once per selected file of every directory in its input tree, and writes the results, and a record of each run, into an output tree that mirrors the input.

.. code-block:: bash

shexec --fileFilter 'glob:*.jpg'                                            \
    --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
                    %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"      \
    /incoming /outgoing

It is also the design for wrapping a large existing program as a plugin. Put the program in the image, set the default command, and the plugin is done: the program's whole command line stays available through the template rather than a hand-picked subset of flags, and the tree walk, the records, the parallelism and the exit status come from the stack. With for example FreeSurfer in the image, one recon-all per subject directory is:

.. code-block:: bash

shexec --analyzeFileIndex f --fileFilter 'glob:*.nii.gz'                     \
    --exec "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile}
                      -s %{inputWorkingDir|basename} -sd %{outputWorkingDir}" \
    /incoming /outgoing

pl-freesurfer is an example of the pattern: a plugin that depends on this one the way this one depends on pfdo_run, contributing the FreeSurfer image, that template as the default --exec, and nothing else. See Building a plugin on pl-shexec_.

The engine is pfdo_run <https://github.com/FNNDSC/pfdo_run>_; the plugin is pfdo_run with its directories supplied by ChRIS and its own run timed.

⁠The template

--exec is not a shell command. It is a template whose %{...} tags name the file and directories in hand, resolved once per file, and run directly without a shell, so a path with a space in it stays one argument and nothing is interpreted that was not written. Three tags describe the job:

========================== ============================================== %{inputWorkingDir} the input directory being processed %{outputWorkingDir} its counterpart in the output tree %{inputWorkingFile} the file being processed ========================== ==============================================

Functions derive one name from another: %{inputWorkingFile|rmext}.png turns scan.jpg into scan.png; %{inputWorkingDir|basename} is the directory's own name. Every flag of the plugin is a tag too, and so are %{timestamp}, %{hostname} and %{user}. The grammar is pftag <https://github.com/FNNDSC/pftag>_'s; shexec --man lists it under TAGS. A template with no tag draws a warning, since the same command would then run for every file.

⁠What each run leaves behind

Each directory of the output tree gets, beside its results, one file, jobs.jsonl, with one line per job that ran there:

.. code-block:: text

scan.png
jobs.jsonl

The line is the whole record of the job as JSON: the command as executed, everything it printed and everything it complained, its exit status, the host it ran on, and when it started and ended. pflog read out/**/jobs.jsonl prints a day's jobs as a table, --failed keeps the ones that went wrong, and a log collector indexes the lines by field.

.. code-block:: text

STARTED                     DURATION  STATUS  NAME
2026-10-14T16:02:07-04:00      4.213  ok      convert /incoming/a/scan.jpg /outgoing/a/scan.png
2026-10-14T16:02:12-04:00      0.004  failed  convert /incoming/b/bad.jpg /outgoing/b/bad.png

--jobFiles also writes each job as four files beside its result, scan.jpg-cmd, -stdout, -stderr and -returncode, the stream files rendered as a terminal would have shown them, for the reader who opens them in an editor. --noJobLogging keeps no record at all. --json prints the whole run's record on standard output.

The plugin's own run is timed by pflog <https://github.com/FNNDSC/pflog>_: one line of JSON on standard error, when it began, when it ended, the seconds, and whether it succeeded. CUBE keeps that line with the rest of the plugin's output, and any log collector indexes it.

⁠Selecting files and directories

The filters are pftree's. --fileFilter and --dirFilter take substrings, or glob: and re: patterns, joined by --fileFilterLogic and --dirFilterLogic; --maxDepth bounds the descent. --analyzeFileIndex chooses which of a directory's selected files get the command: a for all, f for the first, l for the last, m for the middle, or an index. f is how one command runs per directory, started on the first file, as for a DICOM series.

Directories run in parallel under --threads. At --verbosity 2 each command's output streams to the console as it runs.

⁠Exit status

0 when at least one directory's jobs all succeeded; 1 when every directory failed or the run could not start; 2 when --exec is missing or a flag's value is unusable. A failed template, a command that cannot be found, or a non-zero exit fails its directory and the run continues; the record says why.

⁠Wrapping a program

A plugin for a program with its own command line, FreeSurfer, ANTs, FSL, DCMTK, is pl-shexec with three changes and no new code:

• the image carries the program, with whatever licence or data it needs;
• the default --exec is the program's usual invocation as a template, so the common case needs no command at all and the unusual case overrides it;
• the default --analyzeFileIndex chooses the unit, f for one run per directory, as for a subject or a DICOM series, a for one run per file.

The user of such a plugin gets the program's entire interface, because anything the program accepts can be written into the template, and gets the same records, the same filters, the same threads and the same truthful exit status as every other plugin on the stack. This is the reason pl-shexec exists in the form it has.

⁠Building a plugin on pl-shexec

pl-shexec is a library as well as a plugin, in the same way pfdo_run is a library to pl-shexec. A downstream plugin does not copy it; it depends on the shexec package and calls plugin_build with a spec of what differs:

.. code-block:: python

# freesurfer/__main__.py, the whole of pl-freesurfer's code
from shexec import PluginSpec, plugin_build

from freesurfer import __version__

RECON_ALL: str = (
    "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile} "
    "-s %{inputWorkingDir|basename} -sd %{outputWorkingDir}"
)

PLUGIN = plugin_build(PluginSpec(
    name="freesurfer",
    version=__version__,
    title="FreeSurfer recon-all over every subject",
    minMemory="8Gi",
    minCpu="4000m",
    defaults={"exec": RECON_ALL, "analyzeFileIndex": "f", "fileFilter": "glob:*.nii.gz"},
    bannerTitle="FreeSurfer recon-all over an input tree",
    description=DESCRIPTION,
    examples=EXAMPLES,
))
main = PLUGIN.main

PluginSpec holds everything a plugin changes: its name and version, which become its command, its span and its --version; its title, category and resource limits for ChRIS; the defaults of its flags, above all --exec, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: omit=(...) removes them from its parser, its ChRIS description and its manual, and fixes their values. A flag that suits a shell but not ChRIS goes in chrisHidden instead: it stays on the command line and in the manual and is left out of the description ChRIS reads, so in ChRIS it keeps its default. jobMemory="16Gi" says how much memory one job needs at most; --threads N is then capped at the memory the process may use, its container's or Slurm job's limit or the machine's available memory as hostlib.memory_limit reads it, divided by that figure, and the cap is said on standard error. pl-freesurfer uses both: one recon-all at a time in a 32Gi ChRIS container, several at once on a workstation or a Slurm node with the memory for them. plugin_build returns the parser with those defaults applied, the manual, and main already wrapped in pflog's timed and chris_plugin, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.

The rest of pl-freesurfer is a pyproject.toml naming chris_plugin and shexec as its dependencies, the first so that chris_plugin_info finds the plugin, a setup.cfg, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its __main__ is the same call with an empty default command. The pf* stack's docs/plugin.adoc has the template.

⁠Running it

From ChRIS, add the plugin to a pipeline and give it --exec and filters as parameters.

From the command line, with the image:

.. code-block:: bash

docker run --rm -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw            \
    fnndsc/pl-shexec shexec --fileFilter txt                                \
    --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing

docker run --rm fnndsc/pl-shexec shexec --man

With Apptainer, apptainer exec docker://fnndsc/pl-shexec shexec .... The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.

As a Python package, pip install . in this repository gives the shexec command and the shexec module, whose plugin_run(options, inputdir, outputdir) is the body of the plugin and whose shexec.models re-exports every record pfdo_run, pfdo, pftree, hostlib and pflog produce.

⁠Development

.. code-block:: bash

uv venv --python /usr/bin/python3.14 .venv
uv pip install --python .venv/bin/python -e ".[dev]"
.venv/bin/ruff check . && .venv/bin/pytest -q
PATH=$PWD/.venv/bin:$PATH chris_plugin_info --dock-image docker.io/fnndsc/pl-shexec:2.1.0 shexec

The CI workflow lints and tests on Python 3.11 to 3.14, builds the image on every push to main, pushes it to Docker Hub and ghcr.io, and on a v* tag uploads the plugin's description to cube.chrisproject.org and publishes the shexec package to PyPI, so that a plugin built on pl-shexec can depend on it. The tag must match __version__ in shexec/__init__.py.

⁠Layout

.. code-block:: text

shexec/
├── __main__.py   the plugin: parser, manual, @chris_plugin @timed main
├── shexec.py     ShExec, options_build, plugin_run
└── models.py     ShExecOptions, and every record below re-exported

docs/purpose.adoc says why the plugin exists, docs/architecture.adoc how it is built, docs/history.adoc where it came from.