We have a job which normally runs for about an hour. It recently reported the following error.
(11.5) 01-19-07 10:51:40 (E) (4904:3440) RUN-050604: |Session JOB_ABC|Workflow WF_ABC|Dataflow DF_ABC_3
Cannot write pipe, due to error .
It’s on DI 11.5 with SQL 2000 repository and backend DW databases (on separate servers, Win2000 OS). The DI job takes a big xml file from shared network folder then process the data, put it in tables, and then reads from tables and spits smaller xml files into SQL DB.
The error message was reported in the data flow which pulls data from tables and convert and pushes it to xml tables.
At this stage we are not sure where this error came from - from DI or SQL or NT? Can anyone shed some light? Much appreciated!
I just checked with our Prod DBA and asked him to check corresponding SQL logs. He confirmed that this is an application issue (DI?) and no SQL logs are found for corresponding time.
I found there was other posts on this forum mentioning this intermittent error on DI. However no good answers were supplied.
[quote:2a656ca6fc=“rachel.l.su”][…]The DI job takes a big xml file from shared network folder then process the data[…]
[/quote]
Very likely, that’s the problem. Consuming too much memory. Execute it and watch the al_engine processes. I guess one gets close to 2GB before it fails.
You should look into the XML Pipeline transform to shred one XML file into multiple chunks of smaller pieces. This way, we need memory for the smaller elements only.
The first xml read bit does consume a fair bit of memory - up to 1.4GB. However, the failed step does not consume that much memory. Having said that, this occurred in production so it’s possible that there are other processes consuming memory thus affected this one.
Would this be the only possible cause of this error? So that we should concentrate on this?
Yes the XML pipeline worked! :o It reduces memory usage to 250MB for a 41MB input XML file (relatively simple XSD but with approx 200,000 repeating nodes)! It does take 20~30% longer to process this step, but still a good outcome as this is not the most time-consuming part of the job so overall not causing major delays but did resolve our memory issue. Thanks again!
One more question I want to ask - how does the xml pipeline work? In this case we’re reading xml file from a folder, so I guess the xml-pipeline “reading” is been pushed down to the OS? (reading from the manual)? We got our own java adapter…can we still get the benefit of xml-pipeline transform on top of our adapter? Or do we need to add some codes to our adapter to support xml-pipeline? Looking forward to your feedback.
hi dastocks, we haven’t asked for the hotfix yet. Thanks for the information though!
hi dnewton, due to current implementation, we can’t do much about changing the file location. Since the number of files we’re getting is only 10, the network time should be minimal…and it’s all within the same intranet so should be ok. Thanks for the advice!
[quote:96a41dbb7b=“rachel.l.su”]
hi dnewton, due to current implementation, we can’t do much about changing the file location. Since the number of files we’re getting is only 10, the network time should be minimal…and it’s all within the same intranet so should be ok. Thanks for the advice!
[/quote]
What we generally do is first have a Script step in the job, and that step runs a batch file which copies the file from the network to a local directory on the job server.
…We got our own java adapter…can we still get the benefit of xml-pipeline transform on top of our adapter? Or do we need to add some codes to our adapter to support xml-pipeline? Looking forward to your feedback.
…
[/quote]
What does your adapter do? We are having some issues with creating an adapter. There aren’t many sources of information about them. We need our adapter to call a Java API and use the data structure returned from the API as a source in DI. Is that anything like you are doing? It seems the only thing I can find is that the adapters want to work with files or tables as the source.
Yes the adapter reads from DB and return XML data structure to DI. So I can confirm that can be done. However I’m not previliged to sharing the details - firstly I didn’t write it, and secondly it’s the company’s IP. Please ask others.
The XML-Pipeline can only used together with XML files. Normally, we read XML files and parse them, with the XML pipeline you configure a SAX parser and we read the output of the sax parser instead of the XML file. So like in your case, instead of reading one XML node with 20’000 childs, the SAX parser will shred that into 20’000 XML “files” - the output of the pre-parsing.
It seems you implemented a function based adapter: https://boc.sdn.sap.com/node/5946
Something that acts and feels like a stored procedure. You call it with input parameters, then the procedure dows something and once it is done, will return a result.
I haven’t implemented the “send email” part in above sample yet, but you can always go to the \DI\adapters\sdk\sample\testadapter…" directory and strip down the sample code there to the function based version. In case you need help, let me know.
It is a Windows limitation or better, a limitation of 32Bit Operating Systems.
2^31 = 2GB, one address line is reserved.
And it accounts for the al_engine process. DOP is changing the number of threads but still all executed in one process so DOP does not help to overcome that.
All above said is valid for 11.5 and earlier, in 11.7 you can swap to disk, split one dataflow into multiple processes etc.
(BOB member since 2005-03-10)
(BOB member since 2004-12-17)
(BOB member since 2004-01-30)
(BOB member since 2004-01-30)
(BOB member since 2005-05-02)