Scheduled jobs run twice - Management Console bug?

dataservices administration
#1

I have searched for this issue and found that other people had similar issues but no real resolution so I wanted to bring this to people’s attention.

Environment:
SAP BusinessObjects Data Services XI 12.2.3.2
Windows Server 2008 R2 64-bit
Data Services Management Console using supplied Tomcat version that came with BODS.

Issue:
When first creating a schedule for a job through the Management Console (MC), there is no problem - everything runs fine.

The Windows Server Task Scheduler contains entry “AT1” for the designated time.

Then I disable (but not remove) the schedule. The AT1 entry remains but is not executed so I assume the MC sets this to disabled.

But then I re-enable the schedule for that one job and sure enough, an ADDITIONAL entry is created in the Task Scheduler.

I now have AT1 and AT2 - both are active and kick off at the same time, resulting in table locks on in our Staging area!

We have had this issue twice now in our Production environment and I have been able to re-produce this problem in our Development environment so it does not seem to be server/configuration specific.

Do other people have this issue as well? Is SAP aware of it? Has it been resolved in an update?

We have tried to install the 12.2.3.4 update which failed due to the installer problem that is known and documented. We are looking to upgrade to BODS 4.x in the near future but this will be some time away as it will be one push for both the BO Reporting and BODS area.

Any suggestions?

ErikR :new_zealand: (BOB member since 2007-01-10)

Jobs not kicking off on reactivating the schedule
#2

Interesting findings. When I’ve encountered a double execution of a job it was a one-off happening. The job had the .bat file created by the Export Execution Command and then the .bat is called by a third party scheduling tool. In this case it was blamed on al_joblauncher.exe.

Your situation is quite different though in that you are seeing duplicate entries in the Windows scheduler. Good troubleshooting skills right there! It will be interesting to see how this is resolved.

eganjp :us: (BOB member since 2007-09-12)

#3

The double execution of jobs continues to be an issue with forum members. The issue where the DS Management Console mistakenly creates a second entry in the O/S scheduler is an obvious bug. In the case of one of my clients they are not using the DS Management Console to schedule jobs so there are no entries in the O/S schedule. Instead, a third party is used to call the exported execution command (.bat file).

As a way of provoking some discussion towards other causes of this problem I’ll throw out a couple of questions:
For the jobs that are being executed more than once, are the export files for these jobs older? In other words, do the export files that are newer exhibit the same problem?

We saw a situation where the export execution command was quite old and no longer agreed with the configuration of the job server. The .bat file was exported at a time when the repository was on JobServer_6 and subsequently the repository was moved to JobServer_1. The .bat file was still configured to write the log to JobServer_6 and that’s where the logs showed up. It isn’t clear if the job was executing on JobServer_6 or JobServer_1. However, I don’t think this is a smoking gun since the double execution of the job was rarely seen.

eganjp :us: (BOB member since 2007-09-12)

#4

Thanks for getting this topic back in discussion Jim. I have just checked the .bat and .txt generation dates for the jobs that we are having problems with. They have all been generated fairly recently (in the last 12 months), and against the same version of DI and repository.

We have only noticed this problem of more than one run (can be twice - but we have also had some jobs being activated 3 times) of a job in the last few days. However, it may have been going on for a while, in some cases this is not important as it may simply be picking up no data as the first instance had already processed the data, so we may not necessarily notice it affecting business.

We have checked the ControlM logs (most of our jobs are scheduled through ControlM), and I have confirmed that ControlM only kicks off the bat script once.

Another important point may be that we are using a server group. I have checked both server logs, and it is only the one server that has been involved in running these jobs in question.

It is still a mystery as to why this is happening and help would be appreciated.

Schmidty (BOB member since 2012-01-19)

#5

So far I think it has been consistent that the third party scheduling tool does not appear to be the problem. Our logs also show that the scheduler (AutoSys in our case) is only intiating the .bat file once. We are not on a Server Group so I think that rules out Server Groups as the problem.

I think the double execution has the potential to cause corruption. But this depends on how the jobs are written and how close together the double execution is. Often our second execution fails when it tries to write to the same table and it fails to get a lock on the entire table (due to using Bulk Loader).

eganjp :us: (BOB member since 2007-09-12)

#6

if a schedule is getting executed twice then it can happen only from the scheduler DS doesn’t have it’s own scheduler or mechanish to trigger schedule

this can happen in following situation
1 - the disable schedule or remove schedule from DS is not working as mentioned in original post, could be a bug in 12.2.3.2

2 - Schdule information is deleted from DS (repo was recreated etc), so DS doesn’t know about the tasks that exists in scheduler, creating a new schedule will create multiple schedule for the same job and could cause multiple execution of same job at same time

from DS 12.2.3.4/5 there was change in the location of bat and txt file, since creating files in log folder in some cases was causing files getting overwritten in case of multiple repos, the bat and txt file are now save under schedule folder under job server, it’s possible that if you are deleting a schedule that was created using version earlier than this, it’s not getting deleted (on remove or disable) leaving the old scheduler task

manoj_d (BOB member since 2009-01-02)

#7

We are having the same problem. We use the scheduler through the DS Management Console. Disabling and then enabling a schedule causes a new task to be created in the Windows task scheduler, which ends up causing the job to be run twice at the same time.

For now, we just know to go in and delete the duplicate task in Windows task scheduler but this is a major hassle…if anyone comes up with any solution, we would love to see it.

mikebowen :us: (BOB member since 2010-08-10)

#8

what is the complete version of DS ?

before deleting the duplciate task can you check the task details and check the location of the .bat file that is being called for each schedule ? also check the date when each of these tasks are created

is this happening even for the schedule that are created using the DS version that is currently installed

manoj_d (BOB member since 2009-01-02)

#9

Observed same issue twice in last 2 weeks . We are using Autosys and DS Job Server group. Autosys logs show that job was started once.

Here are few interesting observations :

  1. There was some sort of delay between Autosys starting job and changing status of job to running ( around 155 seconds delay). Normally , status change is done within 5 seconds.

  2. I see Autosys changing status of Job as RUNNING and SUCCESS twice in the log.

Mayank R (BOB member since 2012-02-05)

#10

try changing the following parameter in the DSConfig.txt to 1
RWJobLaunchMaxRetry=1

if the problem is with Job Launcher not getting the response back from job server but the job is getting started, the laucher will re-try to start the job which may cause multiple execution of the same job, setting the value to 1 will only try once

manoj_d (BOB member since 2009-01-02)

#11

Which section of the DSConfig.txt file does RWJobLaunchMaxRetry belong in?

eganjp :us: (BOB member since 2007-09-12)

#12

add it under following section
[int]

manoj_d (BOB member since 2009-01-02)

#13

Hi Manoj ,

If RWJobLaunchMaxRetry is set to greather than 1 and Job Launcher does not get any response back from Job Server , where will Job Launcher send 2nd request : Job Server or Job Server Group ?

In my case , we are using Job Server group and I am wondering if making changes in DSConfig.txt will make any difference since DSConfig.txt is specific to a Job Server.

Mayank R (BOB member since 2012-02-05)

#14

Job launcher runs local, I’d say that you need to change the setting for all jobservers in your job server group.

Johannes Vink :netherlands: (BOB member since 2012-03-20)

#15

Right. But I still need to know if 2nd attempt is made on local job server or Job server group. If its local job server , then it won’t resolve my issue as I am observing duplicate executions on different job servers ( that are part of same Server Group ) .

Mayank R (BOB member since 2012-02-05)

#16

Manoj is the authoritative source on this but my guess is that if the retry is set to 1 it would be on the same local server. If it’s a server group then that server is responsible for figuring out which server within the group to actually run the job on.

Now, what happens if you set the retry to zero? I’m thinking that may be what I want. I really don’t want the jobs executed twice.

eganjp :us: (BOB member since 2007-09-12)

#17

In between this is what SAP Support says for RWJobLaunchMaxRetry:

“The setting you mention is not relevevant at this point as the job is
not being executed twice on same job server but at the same time on
two different job servers.”

Waiting for Manoj… 8)

Mayank R (BOB member since 2012-02-05)

#18

if you set this to 0, the job will not be executed at all, there is bug which instead of disabling the retry is disabling the execution itself if this parameter is set to 0, hence I suggest to set this to 1, this will be fixed in DS 4.0 SP3.4

ok, if the job is going to 2 different job servers, then it’s not happening in the launcher, the execution script is being called twice

what is the SAP Message # ?

manoj_d (BOB member since 2009-01-02)

#19

Hi Manoj ,

Message # is 187184

Mayank R (BOB member since 2012-02-05)

#20

I am using DS 4.2 SP3 and this issue is still there though SAP note says its rectified in DS 4.0. I have manually disable the job from windows Task Scheduler right now, will monitor and update.

Arun.K :us: (BOB member since 2011-10-18)