Batch mode analysis of PET data

What is a script / batch file?

Batch file or shell script is a collection of software commands in a text file.

In their simplest form, they contain the commands just as you would have typed them on the CLI (command shell). This text file then can be used as a program: when you double-click the text file icon in a file browser, or enter its name to the command prompt window (shell), it executes the commands that it contains, line by line, as if you had given the commands by yourself.

Why to do analysis in batch mode?

Then why should one write the commands first into a text file, and then execute it, instead of writing the commands directly to the command prompt?

1. For documentation and for easy correction of mistakes

When you save the script with the original PET data, everyone can later see from it what you actually have done, and, if necessary, make a correction to it, and run everything again with no trouble at all.

Sometimes you will notice an error in a data file only when you are inspecting the analysis results. If the data file can be repaired, it is easy to do the analysis again with the script.

2. To use text processing tools

You can create the script file in any text editor or word processing program you like, as long as you save the file in ASCII (text only) format. On Windows computers Notepad is a good editor for this purpose, since it always saves files in pure text format without formatting and is readily available.

You can use copy-paste, find-replace, and whatever functions you like to edit the file, instead of writing and editing the same commands again and again to the command-prompt window.

3. To have time for doing something else

You may need to have to analyze a whole bunch of PET studies, and the analysis software takes a few moments to execute, not so long that you had time to concentrate on doing something else, but long enough to get bored. The analyses go with a swing, if you write all the commands into a text file, start it, and go for a cup of coffee. You could also start it later, so that the batch mode analysis executes while you attend to a meeting, or during the night.

4. To get rid of command prompt window

If you are not familiar with the Windows or Linux command-line interface (CLI), batch files are one way of keeping you apart from it in the future, too. You can just type the commands in a text editor, save the file, then double-click its icon in file browser, and close your eyes while the commands run in the command shell window.

How to make the script / batch files?

Web tutorials for writing script and batch files for Windows and Linux/macOS are listed in references.

It is a good practice to test inside the scripts that each executed command was successful, and if not, then stop the execution of the script promptly with error message. Examples below will show how to do this in practice.

Batch files (*.bat) in Windows and scripts in Unix/Linux/macOS look similar, and if they are kept very simple, can even be run on both platforms as such. There are also some differences between different versions of Windows and different Unix/Linux/macOS command shells.

Alternatively, scripts or small programs can be written and executed in specialized program environments, like Matlab/Octave, R, and Python. CSC offers easy-to-use notebooks environments for working with R and Jupyter, free of charge for Finnish researchers. Boutiques is a framework for integrating and executing command-line applications across computing platforms (Glatard et al., 2018). Porcupine is a visual tool for creating pipeline scripts for analysis of neuroimaging data (van Mourik et al., 2018).

Creating batch file in Windows

Use Notepad or similar program to create a text-only file. If you use MS Office Word or other word processor, make sure to save the file in ASCII format, without any formatting (MS-DOS text). Save with file name extension .bat. Execute it like any other program from command line, without giving the file name extension, or by double-clicking its icon.

Example #1 (MS Windows, but would work also in Unix/Linux/macOS)

This bat file converts regional TAC data files from old format to DFT format from the specified studies, then runs Logan plot analysis for each of them, and then collects the results and calculates the mean values and standard deviations into a table in HTML format:

nci2dft uf0498.roi.kbq uf0498.dft
nci2dft uf0499.roi.kbq uf0499.dft
nci2dft uf0504.roi.kbq uf0504.dft
nci2dft uf0505.roi.kbq uf0505.dft 
logan uf0498.dft cer 15 999 uf0498logan.res
logan uf0499.dft cer 15 999 uf0499logan.res
logan uf0504.dft cer 15 999 uf0504
logan.res logan uf0505.dft cer 15 999 uf0505logan.res
rescoll loganmeans.html *logan.res

Example #2 (MS Windows)

This batch file does exactly the same things as the previous example, but uses the for-loop:

for %%f in (uf0498 uf0499 uf0504 uf0505) do nci2dft %%f.roi.kbq %%f.dft
for %%f in (uf0498 uf0499 uf0504 uf0505) do logan %%f.dft cer 15 999 %%flogan.res
rescoll loganmeans.html *logan.res

Example #3 (MS Windows)

If the analysis commands are exactly the same for each PET study, the only difference being the study number, you could define the study number as a variable, and then just copy-paste the analysis command part:

set studynr=ua2826
imglhk3 -W %studynr%ap_pure.delay.kbq %studynr%dy1.img %studynr%k3.img
img2tif -rb -s %studynr%k3.img %studynr%k3_plane20.tif 20 1
img2dft %studynr%k3.img ..\%studynr%*.roi %studynr%k3.dft
dftavg -rm %studynr%k3.dft
dftavg -h %studynr%k3.dft
taclist %studynr%k3.dft

set studynr=ua2831
imglhk3 -W %studynr%ap_pure.delay.kbq %studynr%dy1.img %studynr%k3.img
img2tif -rb -s %studynr%k3.img %studynr%k3_plane20.tif 20 1
img2dft %studynr%k3.img ..\%studynr%*.roi %studynr%k3.dft
dftavg -rm %studynr%k3.dft
dftavg -h %studynr%k3.dft
taclist %studynr%k3.dft

set studynr=ua2831
...

Be cautious about space characters (also trailing spaces) with set.

Example #4 (MS Windows)

The previous example could also be written with two batch files, with runall.bat calling runone.bat once for each study:

runall.bat:

call runone ua2826
if not %errorlevel%==0 goto FAILED
call runone ua2831
if not %errorlevel%==0 goto FAILED
call runone ua2831
if not %errorlevel%==0 goto FAILED
...

:PASSED
@echo ======================================================== 
@echo Batch file executed successfully. 
@echo ======================================================== 
goto END 
:FAILED
@echo.
@echo failed!
@echo. 
:END
pause
exit

runone.bat:

imglhk3 -W %1ap_pure.delay.kbq %1dy1.img %1k3.img
if not %errorlevel%==0 exit /b 1
img2tif -rb -s %1k3.img %1k3_plane20.tif 20 1
img2dft %1k3.img ..\%1*.roi %1k3.dft
if not %errorlevel%==0 exit /b 1
dftavg -rm %1k3.dft
if not %errorlevel%==0 exit /b 1
dftavg -h %1k3.dft
if not %errorlevel%==0 exit /b 1
taclist %1k3.dft

Notice that in runone.bat we verify after each command that the program return code is 0 (successful), and if it is not, then runone.bat stops execution immediately, and returns error also to the calling batch file, runall.bat, which then also stops, displaying text failed!.

Notice also, that runall.bat pauses in the end after displaying the fail or success message and waits for the user to press a key. These two last lines should be omitted, if the batch file is run from command shell, but they are needed if batch is run by double-clicking its icon; otherwise command-tool window disappears before user can see the end result.

'If' statement can be used with multiple commands with parens (), for example:

setLocal EnableDelayedExpansion

if %CM%==1 (
fitk2 plasma.dat blood.dat tissue.dat 60 k2.res
if not %errorlevel%==0 exit /b 1
) else if %CM%==2 (
fitk4 plasma.dat blood.dat tissue.dat 60 4.5 k4.res
if not %errorlevel%==0 exit /b 1
)

Note in case you need to another batch file inside parenthesis you need to have the setlocal command in the beginning of the main batch file, because otherwise the next command may be run immediately without waiting for the called batch file to be executed.

More examples


Creating script on Unix/Linux/macOS

Use textedit, nedit, gedit, or any other ASCII text editor. There are no requirements for the file name or its extension, but .sh (shell) extension is commonly used. After you have saved the file, give everyone the permission to execute it for example with command:

chmod 777 your_script_name

If you write the Unix/Linux scripts in Windows, convert the script file to Unix/Linux format with dos2unix before trying to run it. Although the file may look fine without conversion, you will get strange errors if you don't do this.

When executing the script, you have to give the full name of the script file, including the path (usually ./) and extension.


Example #1 (Unix/Linux/macOS, and even Windows)

This script converts old regional data format to DFT format from specified studies, then runs Logan plot analysis for each of them, and then collects the results and calculates the mean values and standard deviations into a table in HTML format:

nci2dft uf0498.roi.kbq uf0498.dft
nci2dft uf0499.roi.kbq uf0499.dft
nci2dft uf0504.roi.kbq uf0504.dft
nci2dft uf0505.roi.kbq uf0505.dft
logan uf0498.dft cer 15 999 uf0498logan.res
logan uf0499.dft cer 15 999 uf0499logan.res
logan uf0504.dft cer 15 999 uf0504logan.res
logan uf0505.dft cer 15 999 uf0505logan.res
rescoll loganmeans.html *logan.res

Example #2 (Unix/Linux/macOS)

If the analysis commands are exactly the same for each PET study, the only difference being the study number, you could define the study number as a variable, and then just copy-paste the analysis command part:

#! /bin/sh
LC_NUMERIC=C

studynr=ua2826
imglhk3 -W $studynr'ap_pure.delay.kbq' $studynr'dy1.img' $studynr'k3.img'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
img2tif $studynr'k3.img' $studynr'k3_plane20.tif' 20 1
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
img2dft $studynr'k3.img' '../'$studynr'*.roi' $studynr'k3.dft'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
dftavg -rm $studynr'k3.dft'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
dftavg -h $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
taclist $studynr'k3.dft'

studynr=ua2831
imglhk3 -W $studynr'ap_pure.delay.kbq' $studynr'dy1.img' $studynr'k3.img'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
img2tif $studynr'k3.img' $studynr'k3_plane20.tif' 20 1
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
img2dft $studynr'k3.img' '../'$studynr'*.roi' $studynr'k3.dft'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
dftavg -rm $studynr'k3.dft'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
dftavg -h $studynr'k3.dft'
if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi
taclist $studynr'k3.dft'

studynr=ua2831
...

Notice that the script verifies after each command that the program return code was 0 (successful), and stops the execution if it was not, displaying Failed! on the screen.

In this example, LC_NUMERIC=C is not necessary, but it may be important if working on system where locale settings assume that decimal separator is comma.

For more examples you may want to look the simulation scripts, and TPCCLIB application tests, which are bash scripts (with name test_*.sh), available in the source code packages or from the Gitlab repository.


Using batch commands from command line

In Windows command shell

Apply one command to all data files

You can use 'for' loop to execute a certain command to all of your files, for example to convert all of your ECAT 6.3 format images in your current working directory to ECAT 7 image files, give this command:

for %g in (*.img) do e63to7 %g

This can be done also in batch files, but there you must use '%%' instead of '%'. You can download this script to your data folder and double-click it to convert all ECAT 6.3 images in that folder, assuming that you have e63to7.exe installed on your computer.

You can also execute more than one command to each file by separating the commands with '&', for example:

for %g in (*.v) do (@echo %g & lmhdr %g)

With this command you can list all study numbers from regional TAC files into a new text file studies.txt:

for %g in (*.dft) do tacstudy -value %g >> studies.txt

In Linux and macOS command shell

Apply one command to all data files

You can use 'for' loop to execute a certain command to all of your files, for example to convert all of your DFT format TAC files in your current working directory to PMOD format, give this command:

for file in ./*.dft; do tacformat -hdr=no -f=pmod $file; done

When executing the script, you have to give the full name of the script file, including the path (usually ./) and extension.



See also:


Literature

Wikipedia: Shell script, Batch file, and Bourne shell

Wikibooks: Bash Shell Scripting

Linux Shell Scripting Tutorial - A Beginner's handbook

Janssens J. Data Science at the Command Line. O'Reilly, 2014, ISBN 978-1-491-94785-2.

Beginning Portable Shell Scripting (buy e-book from Apress)

Pro Bash Programming (buy e-book from Apress)

String manipulation in DOS

LinuxCommand.org


Tags: , , , ,

Updated at: 2020-02-22
Created at: 2009-05-22
Written by: Vesa Oikonen