Bods Degree of Parallelism not working ?

dataservices
1

Hi all,

I am running a job which is executing for 10 minutes,

So I have changed the Degree of parallelism from default to 6.
Again the job is running for approximately 10 minutes.

There is no difference.
What could be the reason…

Thanks

laxmiuma99 (BOB member since 2016-11-24)

2

A Job running in 10 minutes is a very small set of information to solve a problem. You need to be far more specific, for instance identifying if the whole of the 10 minutes is stolen by a single DF or script or is evenly distributed inside the job. Then you could post the details of that single part of the job to search for a solution. Just as an example, a bad designed lookup could ruin an entire DF.

Besides, if you have a HW bottleneck (Network, disk) the parellelism would even worsen the performance problem.

Regards,

Andrés

aidelia :argentina: (BOB member since 2006-02-02)

3

I agree with Andres. DoP and run time are just two pieces of a very complicated puzzle. Are other things running? Which piece of the job takes 10 minutes? What are the CPU, memory, network and disk stats looking like on the DS and DB servers?

  • E

eepjr24 :us: (BOB member since 2005-09-16)

4

Thank you

Andrés and Ernie Phelps

I will look at more inner details as you suggested, and I will come up If I have any more issues

Thanks

Laxmi

laxmiuma99 (BOB member since 2016-11-24)

5

By setting DOP = 6 you do not FORCE it, you are only specifying the upper max limit. If and how DOP is applied also depends on the data structures and how/if DS can do an effective round robin or not. If you have no primary keys and there really is no way to partition the data, you will find that DS will often ignore the DOP setting and just process everything as is - as it really needs to some way of reliably splitting the data and merging it again.

ErikR :new_zealand: (BOB member since 2007-01-10)

6

Perfectly stated Erik!

Rarely do I change the DOP. In fact, many times I’m over-riding the default of 2 and changing it to 1 for specific Dataflows. I addressed this issue in the following blog post: https://www.prokarma.com/blog/2016/09/12/etl-performance-tuning-low-hanging-fruit

In my experience, you often need to redesign the Dataflow or the database is simply not very effective at quickly generating a result set. I tune from both directions and rarely do I find a process that I can’t improve.

eganjp :us: (BOB member since 2007-09-12)