Batch mode analysis of PET data
What is a script / batch file?
Batch file or shell script is a collection of software commands in a text file.
In their simplest form, they contain the commands just as you would have typed them on the CLI (command shell). This text file then can be used as a program: when you double-click the text file icon in a file browser, or enter its name to the command prompt window (shell), it executes the commands that it contains, line by line, as if you had given the commands by yourself.
Why to do analysis in batch mode?
Then why should one write the commands first into a text file, and then execute it, instead of writing the commands directly to the command prompt?
1. For documentation and for easy correction of mistakes
When you save the script with the original PET data, everyone can later see from it what you actually have done, and, if necessary, make a correction to it, and run everything again with no trouble at all.
Sometimes you will notice an error in a data file only when you are inspecting the analysis results. If the data file can be repaired, it is easy to do the analysis again with the script.
2. To use text processing tools
You can create the script file in any text editor or word processing program you like, as long as you save the file in ASCII (text only) format. On Windows computers Notepad is a good editor for this purpose, since it always saves files in pure text format without formatting and is readily available.
You can use copy-paste, find-replace, and whatever functions you like to edit the file, instead of writing and editing the same commands again and again to the command-prompt window.
3. To have time for doing something else
You may need to have to analyze a whole bunch of PET studies, and the analysis software takes a few moments to execute, not so long that you had time to concentrate on doing something else, but long enough to get bored. The analyses go with a swing, if you write all the commands into a text file, start it, and go for a cup of coffee. You could also start it later, so that the batch mode analysis executes while you attend to a meeting, or during the night.
4. To get rid of command prompt window
If you are not familiar with the Windows or Linux command-line interface (CLI), batch files are one way of keeping you apart from it in the future, too. You can just type the commands in a text editor, save the file, then double-click its icon in file browser, and close your eyes while the commands run in the command shell window.
How to make the script / batch files?
Web tutorials for writing script and batch files for Windows and Linux/macOS are listed in references.
It is a good practice to test inside the scripts that each executed command was successful, and if not, then stop the execution of the script promptly with error message. Examples below will show how to do this in practice.
Batch files (*.bat
) in Windows and scripts in Unix/Linux/macOS look similar, and
if they are kept very simple, can even be run on both platforms as such.
There are also some differences between different versions of Windows and different Unix/Linux/macOS
command shells.
Alternatively, scripts or small programs can be written and executed in specialized program environments, like Matlab/Octave, R, and Python. CSC offers easy-to-use notebooks environments for working with R and Jupyter, free of charge for Finnish researchers. Boutiques is a framework for integrating and executing command-line applications across computing platforms (Glatard et al., 2018). Porcupine is a visual tool for creating pipeline scripts for analysis of neuroimaging data (van Mourik et al., 2018).
Creating batch file in Windows
Use Notepad or similar program to create a text-only file. If you use MS Office Word or other
word processor, make sure to save the file in ASCII format, without any formatting (MS-DOS text).
Save with file name extension .bat
. Execute it like any other program from command
line, without giving the file name extension, or by double-clicking its icon.
Example #1 (MS Windows, but would work also in Unix/Linux/macOS)
This bat file converts regional TAC data files from old format to DFT format from the specified studies, then runs Logan plot analysis for each of them, and then collects the results and calculates the mean values and standard deviations into a table in HTML format:
nci2dft uf0498.roi.kbq uf0498.dft nci2dft uf0499.roi.kbq uf0499.dft nci2dft uf0504.roi.kbq uf0504.dft nci2dft uf0505.roi.kbq uf0505.dft logan uf0498.dft cer 15 999 uf0498logan.res logan uf0499.dft cer 15 999 uf0499logan.res logan uf0504.dft cer 15 999 uf0504 logan.res logan uf0505.dft cer 15 999 uf0505logan.res rescoll loganmeans.html *logan.res
Example #2 (MS Windows)
This batch file does exactly the same things as the previous example, but uses the for-loop:
for %%f in (uf0498 uf0499 uf0504 uf0505) do nci2dft %%f.roi.kbq %%f.dft for %%f in (uf0498 uf0499 uf0504 uf0505) do logan %%f.dft cer 15 999 %%flogan.res rescoll loganmeans.html *logan.res
Example #3 (MS Windows)
If the analysis commands are exactly the same for each PET study, the only difference being the study number, you could define the study number as a variable, and then just copy-paste the analysis command part:
set studynr=ua2826 imglhk3 -W %studynr%ap_pure.delay.kbq %studynr%dy1.img %studynr%k3.img img2tif -rb -s %studynr%k3.img %studynr%k3_plane20.tif 20 1 img2dft %studynr%k3.img ..\%studynr%*.roi %studynr%k3.dft dftavg -rm %studynr%k3.dft dftavg -h %studynr%k3.dft taclist %studynr%k3.dft set studynr=ua2831 imglhk3 -W %studynr%ap_pure.delay.kbq %studynr%dy1.img %studynr%k3.img img2tif -rb -s %studynr%k3.img %studynr%k3_plane20.tif 20 1 img2dft %studynr%k3.img ..\%studynr%*.roi %studynr%k3.dft dftavg -rm %studynr%k3.dft dftavg -h %studynr%k3.dft taclist %studynr%k3.dft set studynr=ua2831 ...
Be cautious about space characters (also trailing spaces) with set.
Example #4 (MS Windows)
The previous example could also be written with two batch files, with runall.bat
calling runone.bat
once for each study:
runall.bat:
call runone ua2826 if not %errorlevel%==0 goto FAILED call runone ua2831 if not %errorlevel%==0 goto FAILED call runone ua2831 if not %errorlevel%==0 goto FAILED ... :PASSED @echo ======================================================== @echo Batch file executed successfully. @echo ======================================================== goto END :FAILED @echo. @echo failed! @echo. :END pause exit
runone.bat:
imglhk3 -W %1ap_pure.delay.kbq %1dy1.img %1k3.img if not %errorlevel%==0 exit /b 1 img2tif -rb -s %1k3.img %1k3_plane20.tif 20 1 img2dft %1k3.img ..\%1*.roi %1k3.dft if not %errorlevel%==0 exit /b 1 dftavg -rm %1k3.dft if not %errorlevel%==0 exit /b 1 dftavg -h %1k3.dft if not %errorlevel%==0 exit /b 1 taclist %1k3.dft
Notice that in runone.bat
we verify after each command that the program return code
is 0 (successful), and if it is not, then runone.bat
stops execution immediately, and
returns error also to the calling batch file, runall.bat
, which then also stops,
displaying text failed!.
Notice also, that runall.bat
pauses in the end after displaying the fail or success
message and waits for the user to press a key. These two last lines should be omitted, if the batch
file is run from command shell, but they are needed if batch is run by double-clicking its icon;
otherwise command-tool window disappears before user can see the end result.
'If' statement can be used with multiple commands with parens (), for example:
setLocal EnableDelayedExpansion if %CM%==1 ( fitk2 plasma.dat blood.dat tissue.dat 60 k2.res if not %errorlevel%==0 exit /b 1 ) else if %CM%==2 ( fitk4 plasma.dat blood.dat tissue.dat 60 4.5 k4.res if not %errorlevel%==0 exit /b 1 )
Note in case you need to another batch file inside parenthesis you need to have the
setlocal
command in the beginning of the main batch file, because otherwise the next
command may be run immediately without waiting for the called batch file to be executed.
More examples
- Convert all percentage values to fractions in all plasma parent fraction files in the current working directory
- Convert all DICOM images in subfolders to Analyze format and calculate sum images
- Convert multiple DICOM images to NIfTI
Creating script on Unix/Linux/macOS
Use textedit
, nedit
, gedit
, or any other ASCII text editor.
There are no requirements for the file name or its extension, but .sh
(shell) extension
is commonly used. After you have saved the file, give everyone the permission to execute it for
example with command:
chmod 777 your_script_name
If you write the Unix/Linux scripts in Windows, convert the script file to Unix/Linux format with dos2unix before trying to run it. Although the file may look fine without conversion, you will get strange errors if you don't do this.
When executing the script, you have to give the full name of the script file, including the
path (usually ./
) and extension.
Example #1 (Unix/Linux/macOS, and even Windows)
This script converts old regional data format to DFT format from specified studies, then runs Logan plot analysis for each of them, and then collects the results and calculates the mean values and standard deviations into a table in HTML format:
nci2dft uf0498.roi.kbq uf0498.dft nci2dft uf0499.roi.kbq uf0499.dft nci2dft uf0504.roi.kbq uf0504.dft nci2dft uf0505.roi.kbq uf0505.dft logan uf0498.dft cer 15 999 uf0498logan.res logan uf0499.dft cer 15 999 uf0499logan.res logan uf0504.dft cer 15 999 uf0504logan.res logan uf0505.dft cer 15 999 uf0505logan.res rescoll loganmeans.html *logan.res
Example #2 (Unix/Linux/macOS)
If the analysis commands are exactly the same for each PET study, the only difference being the study number, you could define the study number as a variable, and then just copy-paste the analysis command part:
#! /bin/sh LC_NUMERIC=C studynr=ua2826 imglhk3 -W $studynr'ap_pure.delay.kbq' $studynr'dy1.img' $studynr'k3.img' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi img2tif $studynr'k3.img' $studynr'k3_plane20.tif' 20 1 if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi img2dft $studynr'k3.img' '../'$studynr'*.roi' $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi dftavg -rm $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi dftavg -h $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi taclist $studynr'k3.dft' studynr=ua2831 imglhk3 -W $studynr'ap_pure.delay.kbq' $studynr'dy1.img' $studynr'k3.img' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi img2tif $studynr'k3.img' $studynr'k3_plane20.tif' 20 1 if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi img2dft $studynr'k3.img' '../'$studynr'*.roi' $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi dftavg -rm $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi dftavg -h $studynr'k3.dft' if [ $? -ne 0 ] ; then echo Failed! ; exit 1 ; fi taclist $studynr'k3.dft' studynr=ua2831 ...
Notice that the script verifies after each command that the program return code was 0 (successful), and stops the execution if it was not, displaying Failed! on the screen.
In this example, LC_NUMERIC=C
is not necessary, but it may be important if
working on system where locale
settings assume that
decimal separator is comma.
For more examples you may want to look the
simulation scripts, and
TPCCLIB application tests,
which are bash scripts (with name test_*.sh
), available in the source code packages
or from the Gitlab repository.
Using batch commands from command line
In Windows command shell
Apply one command to all data files
You can use 'for' loop to execute a certain command to all of your files, for example to convert all of your ECAT 6.3 format images in your current working directory to ECAT 7 image files, give this command:
for %g in (*.img) do e63to7 %g
This can be done also in batch files, but there you must use '%%' instead of '%'.
You can download this script to
your data folder and double-click it to convert all ECAT 6.3 images in that folder, assuming that
you have e63to7.exe
installed on your computer.
You can also execute more than one command to each file by separating the commands with '&', for example:
for %g in (*.v) do (@echo %g & lmhdr %g)
With this command you can list all study numbers from regional TAC files into a new text file
studies.txt
:
for %g in (*.dft) do tacstudy -value %g >> studies.txt
In Linux and macOS command shell
Apply one command to all data files
You can use 'for' loop to execute a certain command to all of your files, for example to convert all of your DFT format TAC files in your current working directory to PMOD format, give this command:
for file in ./*.dft; do tacformat -hdr=no -f=pmod $file; done
When executing the script, you have to give the full name of the script file, including the
path (usually ./
) and extension.
See also:
- Software FAQ
- Download TPC CLI SW
- Windows Subsystem for Linux
- Linux/Unix tools for editing text files
- Repository of batch files for pre-processing input data
- Repository of batch files for simulations
- Examples for converting multiple DICOM images to NIfTI
Literature
Wikipedia: Shell script, Batch file, and Bourne shell
Wikibooks: Bash Shell Scripting
Linux Shell Scripting Tutorial - A Beginner's handbook
Janssens J. Data Science at the Command Line. O'Reilly, 2014, ISBN 978-1-491-94785-2.
Beginning Portable Shell Scripting (buy e-book from Apress)
Pro Bash Programming (buy e-book from Apress)
Tags: Script, Shell, bash, CLI, WSL
Updated at: 2020-02-22
Created at: 2009-05-22
Written by: Vesa Oikonen