BusinessObjects Board

DATA Quality Match transform

dataservices data quality
#1

I have used match transform for intra file match based on 1 column.which is used as break key.

break key has only 2 possible values.

match process works fine when there are around 20k records.When I run the same job for 3 lac(0.3 million) records,it hangs.

It doesn’t show any profress in processing records.

What could be the possible reasons behind it ?
database is sql server 2008.

Please let me know for any questions and early response is appriciated.

Thanks in advance for your support.

Nas7484 (BOB member since 2010-07-28)

#2

you typically route blank/null break keys around your match.

if you group up 500,000 rows into one large break group… depending on how much memory is available (and logical record length) … you’ll run out of ram and hang.

the max break group size setting is used to catch these before finding out about them the hard way. I usually have this set at 10k. If your break group is that large you may want to change it to reduce the number of compares.

jlynn73 :us: (BOB member since 2009-10-27)

#3

I am not passing NULL break key.Though in my case break key is creating 2 large groups but those are not as large as u mentioned so its really weird :frowning:

Though break key should improve the performance but when i remove break key and run the job…it process the records.

Could you please give some more insight on this behaviour…

Nas7484 (BOB member since 2010-07-28)