We pull most of our data using ABAP flows with dat files, but we’ve attempted to pull some tables over to our warehouse directly. With large tables such as MARD and RESB the job completes successfully with no errors but we only get a fraction of the rows. I assume there must be some limit on how much data SAP allows to be pulled directly?
craiggsmith (BOB member since 2009-06-01)
What version ?
If you are using 4.1 and have your data store properly configured for RFC streaming you should be getting everything. We have a table pull of COSP which returns over 10 million rows…
Leigh Kennedy 
(BOB member since 2012-01-17)
We are still on 4.0. I understand there were some significant changes in 4.1. It seems to be a limit on the total amount of data (around 500 MB) as opposed to the number of rows. We are able to pull narrow tables of 10M records but not wider ones. They stop at very odd numbers of rows, but consistent for a given table.
craiggsmith (BOB member since 2009-06-01)
Does it stop at some time limit? I assume that 10 minutes for any foreground jobs for SAP
ganeshxp 
(BOB member since 2008-07-17)
Our foreground job limit was 30 minutes but we had to increase that to a couple hours. MARD stops after about 40 minutes but it’s not always the same. RESB runs about an hour and 15 minutes. I found other tables that pull more data (several GB) without an issue. We had to split RESB into 20 data flows (by columns). There are other tables with more columns, but maybe it’s a combination of rows and columns; I’ll have to do more analysis.
craiggsmith (BOB member since 2009-06-01)
Why will you have to pull the same data from source system 20+ times and that too for the RESB, MARD etc., over the network?
Isn’t that an option to bring into a STAGE System and then use it? Maybe I lack the knowledge of your system architecture…
Between, direct pulls are meant only for small tables…
ganeshxp (BOB member since 2008-07-17)
That’s the whole problem, that’s what we’re trying to do.
craiggsmith (BOB member since 2009-06-01)
And why should you use the DIRECT download method?
ganeshxp (BOB member since 2008-07-17)
I didn’t write this code but I think it was just an attempt at a quick solution. That method also seems to be faster and less prone to error. If we have to do an ABAP flow that’s fine, but I would like to understand where the limit is. We’re pulling tables like MBEW just fine, and it has 5.4M rows and twice the columns.
craiggsmith (BOB member since 2009-06-01)
I agree with you! I have also seen this behavior. But our standards are not to use direct downloads. So I always use direct download to get the table structure for stage 
So I have seen those times, for big tables it halted somewhere in middle. Having said that, DS should practically error out in those cases…Don’t you think so too?
ganeshxp (BOB member since 2008-07-17)
Yes, that’s the strange thing, no errors on either the Data Services side or the SAP ECC side that we could find. SAP must be sending a proper end of file so DS doesn’t know; it should be sending some kind of error.
craiggsmith (BOB member since 2009-06-01)