Hi, everyone ! I have a tricky issue with web-services and BODS 4.1. I try to load data via web-services (SOAP). There are different methods. When I try to load not a big amount of data, everything is OK. But when I try to get many raws with many varchar fields, DS wait for a long time before I have an answer. So, particular example:
There is a simple Job , that just call method and load answer to temp-table. In Input query I just put one input parameter (‘True’). In Function_Call I just call my method from web-service and put input parameter to it. And as you can see from log-file, DS waits for about 20 minutes and when Function_Call has all response, then it loads to temp-table in 15 secs. So, there are 102 000 raws.
But when I try to receive an answer from web-service via CURL utility (without BODS), it takes only 1 min 40 sec. And I save response to text file, it is about 104 Mb
When I saved response to XML, and loads with BODS, it takes only 15 sec.
So, from my point of view, when I try to load from web-services directly, BODS waits, while it forms package from response in memory, and only after that loads data. But we have enough memory (32 Gb), and not all of memory is in use during job running. Maybe there are some tuning features of BODS ? But I can’t find them. Please, help me with this issue
go to %LINK_DIR%\ext\webservices-c folder and edit axis2.xml
add following line
before
run the job from command line with -D -np additional option, check the aix2_log.txt file, and check the time it took to process the request, if you see almost the same time as curl ,then the problem is in DS when converting the response
what is the nesting level of response are you unnesting the response ?
We are seeing similar performance issues. Our issue is in making 20-40 simultaneous web service calls and collecting them back into a table. We can do 60-75 per minute when using Data Services. The remote host sees the concurrent processes, but then it sees silence for several seconds (when Data Services is processing).
We are looking to outsource the web service calling into using the Curl utility. Can you go into some more detail as to how you’re preparing the data and then, how you’re making the call using Curl?
manoj_d, for some web-service’s methods there are only 2 nested levels.
nhohman, I call web-services, that were prepared on site of some accounting russian program and I can’t change these services. About curl - I use .bat file, where write credentials and web-service name.
So, we solved this problem in other way: instead of direct access to the web-services, we manipulate with XML-files. These files create as response of web-services and have response data. And when one particular dataflow directly appeals to web-servise, job executes 17 min. When this dataflow manipulate with XML-file, that has the same data, job executes only 1.5 min (this time includes waiting response from accounting program and XML-file formation)
I am also facing same issue. We are calling webservice as datastore and performance of the batch job is poor. 40000 calls per 80 minutes. Any suggestions to improve performance. How to send more parallel requests.
Thanks for quick reply Mohan. I couldn’t get how to set up continuous work flow. Can you shed some light on continuous workflow. My data flow is simple. I have records in staging table (around 40000) records and i am calling webservice using row generation. (we had defined webservice as datastore).
Yes Manoj. I am looking at the performance by looking at the number of records processed. It looks the webservice is taking more time while parsing the response. I tried to do parallel processing which improved performance. But is there any other way. Appreciate your help.
I need some more information
what is the complete version of DS
is the response big (how many levels of nesting) ?
are you un-nesting the complete response of doing it in multiple queries ?
are you doing any other transformation in the query that is doing the un-nesting ?
what is the performance if you don’t un-nest the response and directly dump to a XML file ?
DS other 3rd party libs to process the response and build the XML, I am try to isolate where the problem is
do you have a Support case for this ? if yes, what is the case/Incident # ?
Thanks for your response Manoj. We are using DS 4.1. Response is little bit big. When I make SOAP call response is coming with in 150milliseconds and the response data is 2500 bytes. It has 6 nested levels including Envelope and Body. I am not doing any thing with response. All I am doing is capturing webservice call response as success are failure based on the fault code (Error code). We haven’t raised any support case. We are using webservices provided by third party. Also when I looked at DB level it looks DS is sending requests sequentially. For each request its taking 200milli seconds. and for 50000 requests job is running for 3hours. Is there any configuration with in DS to send multiple webservice calls.
