If the problem isn’t one huge overflow file but the sum of the overflow files getting to big you have multiple options:
Use auditing to let the load fail if the error count gets too high
Re-read the overflow files into a database table (without constraints) and delete the file afterwards
If you generate too many errors in just one flow you probably want to do some simple validation transforms to at least catch some of the problems and reduce the error count.
Btw. of what use are overflow files if they get this large? Who is going to look through all the problems there?
In your dataflow click on the “audit” button in the menu bar. Choose your target table and audit function “Count” and in the rules tab set your rule to raise an exception if the error count variable is too high.
Make sure your job is executed with auditing enabled.
Btw. I’m not sure if this applies to BODS 4.0, we are still on an older version here.
If you enable auditing on an object (e.g. a table) it generates two variables for you to use in the rules. Default is $Count_YOUR_TABLE_NAME and $CountError_YOUR_TABLE_NAME. As said, no idea if it is done the same way in 4.0, you might want to read the matching chapters in the technical manuals.
But I tried with a limit of 10 error in the audit (if 10 error job fail) with 1000 record wrong … the overflow file is writing anyway every wrong (1000) record.
And the job fail after writing every record in the overflow file…
I don’t normally recommend using an overflow file when there will be a large number of rows rejected. The overflow file for most of my clients will rarely have any rows written to it. The overflow file is the exception rather than the rule.
The above approach leads me to ask the obvious question: Why do you have so many rejected rows?
In some systems I’ve worked on the ETL is not allowed to reject ANY rows. To prevent rejection the ETL will provide (valid) dummy values and the ETL will go back the next day and try to provide real values in place of the dummy values.
(BOB member since 2009-05-18)
(BOB member since 2009-05-18)
(BOB member since 2007-09-12)