In the Management Console you can find a GUI where you can choose if you want to schedule a job using the Windows/Unix ("Data Integrator scheduler") or the Business Objects Enterprise scheduler. And then you just participate from their features - or shortcomings. On the plus side though, you can use any scheduler to start a DI job.
And what does this GUI do? It creates a batch script which, when called, starts this particular job and then asks either the jobserver to issue an "at" command (e.g. at 23:00 everyday run the batch file) or connects to the CMC, creates a "program object" there and schedules that within BOE.
If the schedule is made active, this information is sent to the jobserver or CMC to actually create the batch file and call the "at" command. The jobserver returns a unique ID assigned by the OS and this information is then saved in the AT_ID column. For this to work, the OS has to allow us to use the "at" command, a typical error in Windows is the scheduled-task service is not running or in Unix the user running the jobserver does not have the permissions to use the crontab.
Everytime the active schedule is viewed in the Management Console, the jobserver is asked if the OS does still know about this AT_ID or not. If it does, the schedule is still active otherwise the schedule will be marked as no longer active - somebody used the OS commands and removed it.
If everything worked, when you got to Start -> Control Panel -> Scheduled Tasks (On the Jobserver Computer!!!!) you will find this created schedule (in Unix type "crontab -l").
All the schedule is about is starting a batch script located in the DI\logs\Jobserver\ directory named with the GUID of the job at a specific time/recurrance pattern.
This created batch file is using the AL_RWJobLauncher program to send to a particular jobserver ("inet:SERVER_HOSTNAME:JOBSERVER_PORT") or jobserver group the signal to start a job, the information about the job with all its parameters, global variables,... can be quite long, so the job information is saved in a text file of its own. Note: Windows allows command parameters to be 1024 chars at most - way too little for some DI scenarios, just imagine the job has 100 global variables defined.
The text file with the job information is passed to the jobserver, and the jobserver in turn forks an al_engine process with the provided parameters. So if you want to know what these parameters are, simply run the al_engine command from a dos box to get a list of supported flags. The most important ones are
the ones the al_engine requires to login to the repository and read the job definition
And in the Windows Scheduler screen you find the schedule was run at 4:40pm.
Before anything, we need a simple way to generate the batch- and text file without the scheduling screen. In the Management Console for the batch job configuration there is the link to "export execution command" for jobs.
Inside the dialog the same parameters and global variables can be defined like in the schedule and by specifying a filename including a path, both files get created there.
But keep in mind, the files are created on the jobserver computer, not the local one. If no path is provided, then the files are placed into the DI\logs directory on that server. Whoever now starts that batch file, at that moment the job will be started. So your scheduler should call this batch file to start a job.
In most cases the scheduler wants to get back a return code as well indicating if the job was successful or not. Unlike the batch file created via the schedule dialog, this batch file here automatically got an additional flag, the -w to wait for the job to complete. With this flag the AL_RWJobLauncher polls the jobserver frequently for the status of the job and returns only once the jobserver says it got completed. The frequency of this polling is set by the parameter -t10 to poll every ten seconds and that is also the reason why there is a delay between seeing the job got finished and the JobLauncher to return - a delay up to the polling interval. The joblauncher returns a status code due to the -w option as well, but as this can be only a number between 0 and 255 we take the DI error number modulo 256.
If you have many jobs, you would need to go through the same process multiple times, export execution command and add that to your scheduler. But actually, the txt file with the job GUID etc are all very similar, the main difference will be global variables and the GUID of course.
The GUID can be found out by looking at the AL_LANG table.
btw, when executing a job, in the first line of the trace log it reads "Reading Job <54a3dd1....", so via the trace log the GUID can be found as well. This GUID does never change unless the job object got deleted from the repository and a new one created with same name, so it is absolutely stable.
But we do not even have to do that, instead of using the -G"GUID" option, you can say -sCreate_XML_File as well.
Another question asked often is how to dynamically assign the global variable values as they seem to be encoded in some sort. Yes, they are encoded as they might contain special characters like blanks. To avoid troubles, they get encoded. But it is not required to do so, their values can be specified in clear text also.
Building Job Chain:
To start a dataflow whenever an event, likely the existence of a file, occurs. We can solve that via multiple ways:
And what does this GUI do? It creates a batch script which, when called, starts this particular job and then asks either the jobserver to issue an "at" command (e.g. at 23:00 everyday run the batch file) or connects to the CMC, creates a "program object" there and schedules that within BOE.
The details
Via above screen a schedule was created. It got named "WikiSchedule" and is supposed to run at 4:40pm on the 29th of April, not recurring and active. When clicking on apply all this GUI information is saved in a repository table called AL_SCHED_INFO. This is done to store the list of scheduled somewhere. The OS scheduler cannot be used for that as e.g. a none-active schedule is not sent to the OS scheduler. So whenever in the Management Console somebody views all schedules, this list is created out of this repository table. But the repository table is aligned towards the needs of the batch file and many GUI elements are hidden inside the fields. For example the JOB_COMMAND contains all the information required to start the job process like repository information, monitor sample rate, global variables,... If the schedule is made active, this information is sent to the jobserver or CMC to actually create the batch file and call the "at" command. The jobserver returns a unique ID assigned by the OS and this information is then saved in the AT_ID column. For this to work, the OS has to allow us to use the "at" command, a typical error in Windows is the scheduled-task service is not running or in Unix the user running the jobserver does not have the permissions to use the crontab.
Everytime the active schedule is viewed in the Management Console, the jobserver is asked if the OS does still know about this AT_ID or not. If it does, the schedule is still active otherwise the schedule will be marked as no longer active - somebody used the OS commands and removed it.
If everything worked, when you got to Start -> Control Panel -> Scheduled Tasks (On the Jobserver Computer!!!!) you will find this created schedule (in Unix type "crontab -l").
All the schedule is about is starting a batch script located in the DI\logs\Jobserver\ directory named with the GUID of the job at a specific time/recurrance pattern.
This created batch file is using the AL_RWJobLauncher program to send to a particular jobserver ("inet:SERVER_HOSTNAME:JOBSERVER_PORT") or jobserver group the signal to start a job, the information about the job with all its parameters, global variables,... can be quite long, so the job information is saved in a text file of its own. Note: Windows allows command parameters to be 1024 chars at most - way too little for some DI scenarios, just imagine the job has 100 global variables defined.
The text file with the job information is passed to the jobserver, and the jobserver in turn forks an al_engine process with the provided parameters. So if you want to know what these parameters are, simply run the al_engine command from a dos box to get a list of supported flags. The most important ones are
the ones the al_engine requires to login to the repository and read the job definition
- -S.......servername
- -N.......database type
- -Q.......database name
- -U.......username
- -P.......password
and - -G.......the GUID of the job, the internal name of it
- -s.......In case the GUID should not be used but the job name instead
- -T.......the bitmask for the trace options
- -r.......monitor sample rate
- -GV......global variable values
- ...
- ...
And in the Windows Scheduler screen you find the schedule was run at 4:40pm.
Using Other Scheduler:
Inside the dialog the same parameters and global variables can be defined like in the schedule and by specifying a filename including a path, both files get created there.
But keep in mind, the files are created on the jobserver computer, not the local one. If no path is provided, then the files are placed into the DI\logs directory on that server. Whoever now starts that batch file, at that moment the job will be started. So your scheduler should call this batch file to start a job.
In most cases the scheduler wants to get back a return code as well indicating if the job was successful or not. Unlike the batch file created via the schedule dialog, this batch file here automatically got an additional flag, the -w to wait for the job to complete. With this flag the AL_RWJobLauncher polls the jobserver frequently for the status of the job and returns only once the jobserver says it got completed. The frequency of this polling is set by the parameter -t10 to poll every ten seconds and that is also the reason why there is a delay between seeing the job got finished and the JobLauncher to return - a delay up to the polling interval. The joblauncher returns a status code due to the -w option as well, but as this can be only a number between 0 and 255 we take the DI error number modulo 256.
If you have many jobs, you would need to go through the same process multiple times, export execution command and add that to your scheduler. But actually, the txt file with the job GUID etc are all very similar, the main difference will be global variables and the GUID of course.
The GUID can be found out by looking at the AL_LANG table.
btw, when executing a job, in the first line of the trace log it reads "Reading Job <54a3dd1....", so via the trace log the GUID can be found as well. This GUID does never change unless the job object got deleted from the repository and a new one created with same name, so it is absolutely stable.
But we do not even have to do that, instead of using the -G"GUID" option, you can say -sCreate_XML_File as well.
Another question asked often is how to dynamically assign the global variable values as they seem to be encoded in some sort. Yes, they are encoded as they might contain special characters like blanks. To avoid troubles, they get encoded. But it is not required to do so, their values can be specified in clear text also.
Building Job Chain:
But before starting with that, I would actually question the requirement. What you mean with a job - executing a dataflow or multiple dataflows - DI names "workflow" - and the job chain could be the DI job. So you would have one Job with multiple workflows, each workflow containing the dataflows. With that you have much tighter control over the dependencies (see Parallel vs sequential Execution).
If you thought you need two jobs because- there is another external task between? Use one job with a script between the workflows and in the script you call this task via the exec() function.
- some dataflows should not be called every time? There is a Conditional object in DI.
- you have to wait a few minutes or until an event occurs? In DI there is a while object, a sleep function,....
Nevertheless, if you really, really, really believe you need to chain job together, the task is simple. Export the execution commands for the jobs like shown in the chapter Using other schedulers and build one mast batch file that is calling each.
Event Based Scheduling:
- In the script that creates the file, add one line to call the batch file with the joblauncher.
- Use a scheduler that supports events. There are many available, some of them are even freeware.
- Start the job and then loop until the file exists. The base layout can be found here: Selective Reading and Postprocessing
- Schedule the job every hour and test if the file exists in a Conditional object.
No comments:
Post a Comment