Process of multi application Job Submissions

Keywords

Dependency, applications, process.

Goal and description

Choreography of multiple jobs that are executed in different application environment. Example: Extract, Transform and Load (ETL) processes. A typical ETL process is composed of:

1. A set of extract activities retrieve data from different applications (SAP, J2EE, RDBMS, z/OS)
2. A set of processing and transformation activities are applied to the extracted data to generate a new set of information (this activities could be computational intensive and could be split in multiple jobs processed in parallel)
3. A set of load activities are applied to the generated data to load them on target applications (Web Site, J2EE, RDBMS)

Actors/stakeholders

There are tree types of users:

• Application Programmers: define Job Definitions, Job Flows and scheduling policies in the scheduler database. They use scheduling flow language for scheduling policies objects through the CLI. They use the GUI for defining them in a graphical way (property panels, graph view, etc.)

• Administrators: define the scheduling topology, resource allocation policies, manage production plan generation. They use GUI or CLI to manage resource objects and policies

• Operators: monitor the execution of the production plan, job and Job flow execution instances status. Perform problem determination to determine the root cause of a failure or a delay. Perform changes to the production plan to fix or bypass the problem. They use the GUI or the CLI to query job and Job flow instances status, display the flow of job dependencies. Perform changes to the production plan.

Assumptions

Preconditions

Main flow of events (Basic, alternate, exceptional)

Postconditions

Success requirements

Support of Job execution activities that represent the execution of a scheduled Job.

A Job activity is long running and has an asynchronous behavior. The start of execution of a Job activity submits to the Job Execution Engine the related scheduled Job. The Job Execution Engine reports back asynchronously the Job status changes and the status of the activity need to reflect the status of the Job. The activity terminates when the Job Execution engine reports that the scheduled Job terminated. If the scheduled job terminates in error the activity final status will be in error.

Job activities can return output values like program execution return code.

Variables can be used to substitute values for Job activities parameters and to receive values returned by Job execution.

Support for connecting Job activities with conditional dependencies.

If a Job activity fails should be possible to specify fault handling/recovery activities to be performed automatically.

Should be possible to have additional Job specific statuses to be reported as Job activity statuses. Those Job specific statuses do not change the flow execution behavior but are just for reporting purposes.

There is the need to attach scheduling specific properties to the process and activities level (scheduling policies). Operators should be able to change those properties at a template level and instance level during execution.

Special Requirements

Issues

Sources/references

IBM Tivoli Workload Scheduler product manges distributed and integrated end-to-end enterprise workload scheduling and job choreography

OGF sponsor/stakeholder/interested party