BusinessObjects Board

R4.1 > NCOA crashes with just over 1k input:

dataservices data quality
#1

If the dataflow spawns an additional thread, NCOA will crash with input between 1,000 and 2,000 rows.

My file in question has 1,069 records and will not run without generating an error indicating ‘there must be at least 100 unique records to run ncoa’. The job does however run perfectly fine if limiting the reader to only 1,000 rows. (it also runs fine in debug)

If I delete everything post ncoa transform so that it does not spawn an additional sub dataflow … then it also runs fine with 1,069 records.

crazy stuff :cheers:

jlynn73 :us: (BOB member since 2009-10-27)

#2

when DOP is set to 2, or some multiple that allows an individual thread to initially contain less than 100 rows.

Knowledge Base article 1372882

really simple fix to set the DOP to 1. I always thought setting it to 0 would cause it to not multi-thread, which has always confused me when it still split into 2 threads. Boy do I feel like a noob. :oops:

jlynn73 :us: (BOB member since 2009-10-27)

#3

Tnx for reporting this back…

I did read the Note yesterday, but did not realise it was related to this topic.

But if I read your description, then it is a very strange bug?

  1. Input file contains 1,069 records.
  2. Commit size is 1.000
  3. So in the second batch the DF will ALWAYS fail if the DOP is more than 1?

So it will also fail with the 4th batch if the record set is 3001 or 3999? Etc etc

Johannes Vink :netherlands: (BOB member since 2012-03-20)

#4

Yes, I would assume that it will crash if your dop is set at 3,4,5 and your input qty is 2050,3050,4050…

Was the pre R4.1 default global_dop set at 2 before ?

I was definitely not expecting this. We’ve also been having excessive application connections errors that I believe are due to this default dop setting being used.

jlynn73 :us: (BOB member since 2009-10-27)

#5

DOP by standard has always been 1 (or 0…). It is a setting that you really deliberately need to modify if you want to spawn more processes.

Johannes Vink :netherlands: (BOB member since 2012-03-20)