Parameter Sweeping Within Input Files

Existing mechanisms

AURORA

OFG23KISTI.pdf Application Repository : The APDL (Application Parameter Description Language) XSD created by the AURORA developers defines the "ParameterFileType" element.

GRIDBUS

Gridbus Resource Broker : The XPML (eXtensible Parametric Modeling Language) XSD "substitute" task element, defined by the Gridbus developers, is based on the Nimrod plan file nomenclature.

Taken from section 4.2.2 (The XPML application description file format) of the Gridbus manual for v.3.0 :

  <xpml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://schemas.gridbus.org/xpml/2006/01/xpml XPMLSchema.xsd">
    <parameter name="X"
               type="integer"
               domain="range">
      <range from="1" to="15" interval="1" />
    </parameter>
    <parameter name="time_base_value"
               type="integer"
               domain="single">
      <single value="0" />
    </parameter>
    <task>
      <substitute>
        <source file="output" />
        <destination file="output.$jobname" />
      </substitute>
      <copy>
        <source location="local" file="calc.$OS" />
        <destination location="node" file="calc.$OS" />
      </copy>
      <execute>
        <command value="./calc.$OS" />
        <arg value="$X" />
        <arg value="$time_base_value" />
      </execute>
      <copy>
        <source location="node" file="output" />
        <destination location="local" file="output.$jobname" />
      </copy>
    </task>
  </xpml>

"A substitute command is meant for substitution of parameter (also known as "variables") values, in local files which are then used during local operations / remote job executions. Typically, the values of the parameters are determined at runtime, and there could be scenarios in which certain input text files need to be tailored for each job using these parameter values. Any of the parameters can used as a variable in the files used in a substitute command by pre-fixing "$" to the parameter name."

NIMROD

Nimrod : The plan file node:substitute (non-XSD) nomenclature.

Taken from the Nimrod Portal Manual - Plan file example.

example plan file :

  parameter x float range from 1 to 10 step 1;

  task main
    copy inputfile.sub node:.
    node:subsitute inputfile.sub inputfile
    node:execute ${HOME}/bin/program inputfile
    copy node:output output.${x}
  endtask

example inputfile.sub :

  X: ${x}
  Y: 7
  Z: 9

Example Input Files

Chaste - (Cancer, Heart, and Soft Tissue Environment)

The following represents a sample XML input file (conforming to the schema definition ChasteParameters.xsd).
Currently, for a parameter sweep run, a script would be expected to generate multiple copies of this file, each with different parameter values, and multiple runs of the chaste executable invoked.
This parameter data file, passed as a command-line parameter, is compiled into a C++ object by the Chaste executable e.g. user@host:~/run/chaste -f inputFile.xml -otherparams.

  <?xml version="1.0" encoding="UTF-8"?>
  <ChasteParameters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                    xsi:noNamespaceSchemaLocation="ChasteParameters.xsd">
    <!-- Problem definition -->
    <SimulationDuration>4.0</SimulationDuration>
    <Domain>Bi</Domain>
    <IonicModel>BackwardEulerFoxModel2002Modified</IonicModel>

    <!-- Mesh definition -->
    <Mesh>
      <Slab>
        <SlabX>10.0</SlabX>
        <SlabY>0.25</SlabY>
        <SlabZ>10.0</SlabZ>
        <InterNodeSpace>0.25</InterNodeSpace>
      </Slab>
    </Mesh>

    <!-- Conductivities -->
    <IntracellularConductivities longi="1.75" trans="1.75" normal="1.75" />
    <ExtracellularConductivities longi="7.0"  trans="7.0"  normal="7.0" />
    
    <!-- Stimuli (as many <Stimulus> definitions as needed) -->
    <Stimulus> <!-- #1 -->
      <Strength>-80000.0</Strength>
      <Duration>1.0</Duration>
      <Delay>0</Delay>
      <Location>
        <CornerA X="-5.0" Z="-5.0" Y="-0.125"/>
        <CornerB X="-4.50" Z="5.0" Y="0.125"/>
      </Location>
    </Stimulus>	
    <Stimulus> <!-- #2 -->
      <Strength>-80000.0</Strength>
      <Duration>1.0</Duration>
      <Delay>2.0</Delay>
      <Location>
        <CornerA X="-5.0" Z="-5.0" Y="-0.125"/>
        <CornerB X="0.0" Z="0.0" Y="0.125"/>
      </Location>
    </Stimulus>	

    <!-- Output -->
    <OutputDirectory>ChasteResults</OutputDirectory>
    <MeshOutputDirectory>Slab</MeshOutputDirectory>
  </ChasteParameters>

Example endowed.inp template file for sweeping over RE and AOA values using AURORA (as used by 2D_Comp fluid dynamics software.)

  AMACH  0.6d0
  RE  ${RE}
  AOA  ${AOA}
  TOL  1.0d-4
  TINF  290.0d0
  CFL  0.5d0
  IERRWRT  1
  TOTPES  6

  NSEULER  -1
  KWKEBL  0
  INTWRT2  50
  ITMAX2  10000

Proposal Parameter Sweep Information Set :

  <?xml version="1.0" encoding="UTF-8"?>
  <xsd:schema xmlns:sweep="https://schemas.ogf.org/jsdl/2007/04/sweep"
              xmlns:xsd="http://www.w3.org/2001/XMLSchema"
              targetNamespace="https://schemas.ogf.org/jsdl/2007/04/sweep"
              elementFormDefault="qualified" version="1">

    <xsd:element name="Function" abstract="true" />

    <xsd:element name="Parameter" type="xsd:string" />

    <xsd:element name="Assignment">
      <xsd:complexType mixed="false">
        <xsd:sequence>
          <xsd:element ref="sweep:Parameter" maxOccurs="unbounded"/>
          <xsd:element ref="sweep:Function"/>
        </xsd:sequence>
      </xsd:complexType>
    </xsd:element>

    <xsd:complexType name="Sweep_Type" mixed="false">
      <xsd:sequence>
        <xsd:sequence maxOccurs="unbounded">
          <xsd:element ref="sweep:Assignment" minOccurs="0" />
          <!-- the newly added element for input file parameter sweeping support -->
          <xsd:element ref="sweep:TemplatedFile" minOccurs="0" />
        </xsd:sequence>
        <xsd:element ref="sweep:Sweep" minOccurs="0" maxOccurs="unbounded" />
      </xsd:sequence>
    </xsd:complexType>

    <xsd:element name="Sweep" type="sweep:Sweep_Type"/>

    <!-- Start the definion of new elements and their types -->

    <xsd:element name="TemplatedFile" type="sweep:TemplatedFile_Type"/>
    <xsd:element name="TemplatedFileSweep" type="sweep:TemplatedFileSweep_Type"/>
    <xsd:element name="TemplatedFileAssignment" type="sweep:TemplatedFileAssignment_Type"/>

    <xsd:complexType name="TemplatedFile_Type" mixed="false">
      <xsd:sequence>
        <xsd:element ref="sweep:Parameter"/>
        <xsd:element name="Location" type="xsd:string"/>
        <xsd:element name="Name" type="xsd:string"/>
        <xsd:element ref="sweep:TemplatedFileSweep" maxOccurs="unbounded"/>
      </xsd:sequence>
    </xsd:complexType>

    <xsd:complexType name="TemplatedFileSweep_Type" mixed="false">
      <xsd:sequence>
        <xsd:element ref="sweep:TemplatedFileAssignment" maxOccurs="unbounded"/>
        <xsd:element ref="sweep:TemplatedFileSweep" minOccurs="0" maxOccurs="unbounded"/>
      </xsd:sequence>
    </xsd:complexType>

    <xsd:complexType name="TemplatedFileAssignment_Type" mixed="false">
      <xsd:sequence>
        <!-- e.g., ${AOA}, or ${RE} -->
        <xsd:element name="PlaceHolder" type="xsd:string" maxOccurs="unbounded">
          <xsd:attribute name="default" type="xsd:anyType" use="required" />

          <!--+
              | or perhaps ... !!
          <xsd:complexType>
            <xsd:sequence>
              <xsd:element name="default" type="xsd:anyType" />
            </xsd:sequence>
          </xsd:complexType>
              +-->

        </xsd:element>
        <xsd:element ref="sweep:Function"/>
      </xsd:sequence>
    </xsd:complexType>
    
    <!-- End the definion of new elements and their types -->
  </xsd:schema>

Example:

Assumption: 2D_Comp steadiness 0 -mesh NACA0012.msh -param endowed.inp

  <jsdl:JobDescription>
    <jsdl:Application>
      <jsdl-posix:POSIXApplication>
        <jsdl-posix:Executable>/usr/ngs/2D_Comp</jsdl-posix:Executable>
        <jsdl-posix:Argument>steadiness</jsdl-posix:Argument>
        <jsdl-posix:Argument>0</jsdl-posix:Argument>
        <jsdl-posix:Argument>-mesh</jsdl-posix:Argument>
        <jsdl-posix:Argument>NACA0012.msh</jsdl-posix:Argument>
        <jsdl-posix:Argument>-param</jsdl-posix:Argument>
        <jsdl-posix:Argument>/user/dave/endowed.inp</jsdl-posix:Argument>
      </jsdl-posix:POSIXApplication>
    </jsdl:Application>
    <jsdl:DataStaging>
      <jsdl:FileName>NACA0012.msh</jsdl:FileName>
      <jsdl:FilesystemName>WORKINGDIR</jsdl:FilesystemName>
      <jsdl:CreationFlag>overwrite</jsdl:CreationFlag>
      <jsdl:DeleteOnTermination>true</jsdl:DeleteOnTermination>
      <jsdl:Source>
        <jsdl:URI>gsiftp://ngs.rl.ac.uk:2811/home/ngsportal/NACA0012.msh</jsdl:URI>
      </jsdl:Source>
    </jsdl:DataStaging>

    <sweep:Sweep>
      <sweep:Assignment>
        <sweep:Parameter>/*//jsdl-posix:POSIXApplication/jsdl-posix:Argument[2]</sweep:Parameter>
        <sweepfunc:Values>
          <sweepfunc:Value>-1</sweepfunc:Value>
          <sweepfunc:Value>0</sweepfunc:Value>
        </sweepfunc:Values>
      </sweep:Assignment>

      <sweep:Sweep>
        <sweep:Assignment>
          <sweep:Parameter>/*//jsdl-posix:POSIXApplication/jsdl-posix:Argument[4]</sweep:Parameter>
          <sweep:Parameter>substring(/*//DataStaging/Source/URI/,43,*)</sweep:Parameter>
          <sweepfunc:Values>
            <sweepfunc:Value>NACA0012.msh</sweepfunc:Value>
            <sweepfunc:Value>NACA1412.msh</sweepfunc:Value>
          </sweepfunc:Values>
        </sweep:Assignment>

1) serial TemplatedFile example:

        <sweep:Sweep>
          <sweep:TemplatedFile>
            <sweep:Parameter>/*//jsdl-posix:POSIXApplication/jsdl-posix:Argument[6]</sweep:Parameter>
            <sweep:Location>http://localhost/pub/</sweep:Location>
            <sweep:Name>endowed.inp</sweep:Name>
            <sweep:TemplatedFileSweep>
              <sweep:TemplatedFileAssignment>
                <sweep:PlaceHolder default="3">${AOA}</sweep:PlaceHolder>
                <sweepfunc:Loop start="2" end="4" step="2"/>
              </sweep:TemplatedFileAssignment>
            </sweep:TemplatedFileSweep>
            <sweep:TemplatedFileSweep>
              <sweep:TemplatedFileAssignment>
                <sweep:PlaceHolder default="5">${RE}</sweep:PlaceHolder>
                <sweepfunc:Loop start="10" end="20" step="10"/>
              </sweep:TemplatedFileAssignment>
            </sweep:TemplatedFileSweep>
          </sweep:TemplatedFile>
        </sweep:Sweep>
      </sweep:Sweep>
    </sweep:Sweep>
  </jsdl:JobDescription>

Generates:

  steadiness -1 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_2.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_3.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_4.inp

  steadiness -1 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_2.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_3.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_4.inp

  steadiness -0 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_2.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_3.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_4.inp

  steadiness -0 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_2.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_3.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_4.inp

File contents:

  FileName       ${AOA}  ${RE}
  endowed_1.inp  2       5
  endowed_2.inp  4       5
  endowed_3.inp  3       10
  endowed_4.inp  3       20

2) parallel TemplatedFile example:

        <sweep:Sweep>
          <sweep:TemplatedFile>
            <sweep:Parameter>/*//jsdl-posix:POSIXApplication/jsdl-posix:Argument[6]</sweep:Parameter>
            <sweep:Location>http://localhost/pub/</sweep:Location>
            <sweep:Name>endowed.inp</sweep:Name>
            <sweep:TemplatedFileSweep>
              <sweep:TemplatedFileAssignment>
                <sweep:PlaceHolder default="3">${AOA}</sweep:PlaceHolder>
                <sweepfunc:Loop start="2" end="4" step="2"/>
              </sweep:TemplatedFileAssignment>
              <sweep:TemplatedFileAssignment>
                <sweep:PlaceHolder default="5">${RE}</sweep:PlaceHolder>
                <sweepfunc:Loop start="10" end="20" step="10"/>
              </sweep:TemplatedFileAssignment>
            </sweep:TemplatedFileSweep>
          </sweep:TemplatedFile>
        </sweep:Sweep>

Generates:

  steadiness -1 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_2.inp

  steadiness -1 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_2.inp

  steadiness -0 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_2.inp

  steadiness -0 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_2.inp

File contents:

  FileName       ${AOA}  ${RE}
  endowed_1.inp  2       10
  endowed_2.inp  4       20

3) nested TemplatedFile example:

        <sweep:Sweep>
          <sweep:TemplatedFile>
            <sweep:Parameter>/*//jsdl-posix:POSIXApplication/jsdl-posix:Argument[6]</sweep:Parameter>
            <sweep:Location>http://localhost/pub/</sweep:Location>
            <sweep:Name>endowed.inp</sweep:Name>
            <sweep:TemplatedFileSweep>
              <sweep:TemplatedFileAssignment>
                <sweep:PlaceHolder default="3">${AOA}</sweep:PlaceHolder>
                <sweepfunc:Loop start="2" end="4" step="2"/>
              </sweep:TemplatedFileAssignment>
              <sweep:TemplatedFileSweep>
                <sweep:TemplatedFileAssignment>
                  <sweep:PlaceHolder default="5">${RE}</sweep:PlaceHolder>
                  <sweepfunc:Loop start="10" end="20" step="10"/>
                </sweep:TemplatedFileAssignment>
              </sweep:TemplatedFileSweep>
            </sweep:TemplatedFileSweep>
          </sweep:TemplatedFile>
        </sweep:Sweep>

Generates:

  steadiness -1 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_2.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_3.inp
  steadiness -1 -mesh NACA0012.msh -param endowed_4.inp

  steadiness -1 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_2.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_3.inp
  steadiness -1 -mesh NACA1412.msh -param endowed_4.inp

  steadiness -0 -mesh NACA0012.msh -param endowed_1.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_2.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_3.inp
  steadiness -0 -mesh NACA0012.msh -param endowed_4.inp

  steadiness -0 -mesh NACA1412.msh -param endowed_1.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_2.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_3.inp
  steadiness -0 -mesh NACA1412.msh -param endowed_4.inp

File contents:

  FileName       ${AOA}  ${RE}
  endowed_1.inp  2       10
  endowed_2.inp  2       20
  endowed_3.inp  4       10
  endowed_4.inp  4       20

Attachments:

OFG23KISTI.pdf [ParameterSweepExtensionInputFileSweeping/OFG23KISTI.pdf]