Managing concurrency - Community

April 5, 2016 01:49 PM

Accepted Answer

A small subset of the data warehouse extraction work involves running about 20 programs. Currently, they are being run serially from a .bat script. Overall, there are probably 150+ things that need to be run by adTempus.

SET PROG_LIST=^

prog_00.exe ^

prog_01.exe ^

prog_02.exe ^

...

prog_18.exe ^

prog_19.exe ^

prog_20.exe

FOR %%p IN (%PROG_LIST%) DO (

%%~p 2>\logdir\%%~p.log

)

I would like to create separate adTempus jobs for each of these. The goal is to be able to run two or more concurrently. How "concurrent" would depend on the resources available on the system. There might be a small, 4 CPU, system for development or test, but a 12 CPU system for production.

I realize that load-based scheduling is not yet available in adTempus. I would like to know if there is some way I could decide on hardcoded concurrency now that could easily be managed as system resources scale?

One way to do this might be to have a job that starts programs 0, 5, 10, and 15. These, in turn, would be grouped into (0,1,2,3,4), (5,6,7,8,9)... with these running serially from responses. As the system scaled up, perhaps groups of three (3) would be possible to get faster completion.

For what I have been doing, this seems like an extremely difficult, time intensive way to manage concurrency. Is there a better way in adTempus 4.2 to do this?

Bill Staff (599 posts)

April 6, 2016 06:39 AM

Accepted Answer

You can set a limit on the number of jobs that can run concurrently in a Queue (see Limiting Job Execution). So you could assign all of these jobs to the same Queue and set a limit of 4 concurrent jobs. You can trigger all of the jobs at the same time and they will sit in the queue until an execution slot is available. If you need them to execute in a particular order you may need to create dependencies using conditions, or trigger them in the order you need them (if all jobs have the same priority, they will get executed in the order they were triggered, absent any conditions that affect them).

If you then migrate the jobs to another server with more capacity, you can just increase the limit for the queue on that server.

Paul Watson (101 posts)

April 6, 2016 07:07 AM

Accepted Answer

Thanks. This is very helpful. I will give it a try.