connect to SSL webservice that requires certificate exchange

system · January 7, 2016, 9:22am

Hello.

I am new to working with web services.

Is it possible that the certificates automatically supplied with Data Services 4.2 will be sufficient to connect with an external web service that requires mutual certificate authentication?

The target web service states it requires certificates that comply with X.509 standard - is this the case with the default Data Services certificates (not sure how I check this)?

Also can anyone tell me what Certificate Authority (CA) is used by the default Data Services certificates?
Should the CA make a difference as long as it is in itself valid?

thanks for any help or comments on this.

icytrue (BOB member since 2006-04-10)

system · January 15, 2016, 9:30am

I have moved on a bit and can clarify the question to this:

Jonathan Fraser 15-Jan-2016 10:09

I have a requirement where we need BODS connect to a web service as a datastore which requires mutual authentication via CA approved security certificates.

Does BODS support this?

All answers appreciated.

icytrue (BOB member since 2006-04-10)

system · January 18, 2016, 1:17am

if the webserver you are connecting to requires certificate then you will have to get the server certificate and set the SERVER_CERT parameter in axis2.xml file

have you tested this webservice using SOAPUI ? if yes, what are the setting s you are doing in SOAPUI

manoj_d (BOB member since 2009-01-02)

system · January 18, 2016, 9:56am

Hi Manoj,

Thanks for the reply. We are in the awkward position of waiting on another party to make available the web service that we are to interact with. We know from their docs that it will require mutual authentication and we are striving to understand this in prep.

Since my original post I have made good progress and have BODS working with a non-secure web service through an Adapter data store. I notice that the advanced properties of this include the “keystore path”

From your experience is this an alternative to modifying the axis2.xml or does it have to be set in conjunction with the axis2.xml or have I misunderstood its purpose and it is an irrelevance?

It is all a bit theoretical at the moment until we have the test webservice available, but I trying to be as ready as possible in the meantime.

We will be using SOAP UI once the web service is up.

It would be nice if there was a public-use mutual certificate exchange test site, but that sounds like an awkward thing to exist. I guess the answer is, if you have the resource, to create an internal one?

thanks
icytrue

icytrue (BOB member since 2006-04-10)

system · January 18, 2016, 8:19pm

mutual authentication is nothing but 2-way SSL
don’t use Adapter Datastore, instead use Webservice Datastore for any kind of webservices (SSL or non-SSL)

I don’t think you can find a public service that you can use for testing, for one-way SSL you can use DS WSDL to run a batch job or even ping operation

for this you will have to setup DS tomcat for SSL and I guess you might be able to setup it for 2-way SSL as well, may have to do a bit a reading around for setting that up

setting up one way SSL should be easy for DS tomcat

manoj_d (BOB member since 2009-01-02)

system · January 19, 2016, 9:42am

Hi Manoj

Thanks very much for the advice. If it’s not too much trouble could you please summarise the reason to not use the adapter method?

Many thanks

Jonathan

icytrue (BOB member since 2006-04-10)

system · January 20, 2016, 4:52pm

if you are using webservice adapter then it requires additional setup/configuration and adapter instance run as a separate process, and requires process to process communication at run time via message broker for exchanging data which adds a minor overhead for both processing and maintenance. whereas the Webservice datastore is similar to any other datastore and doesn’t depend on the any other process to be available when running the job

other thing is webservice adapter also requires that you have DS WebServer (MC) running on the same server on which your Adapter Job Server is running else you will not be able to import the metadata

since there is webservice datastore all new functionality will be implemented in webservice datastore and not in Webservice Adapter

manoj_d (BOB member since 2009-01-02)

system · January 26, 2016, 4:54pm

Hi Manoj

We almost have sight of the web service now, in that we have WSDL but not actual access to the live web service.

The WSDL however has revealed an unwelcome aspect in that it features a payload defined as an xs:any


                                             <xs:complexType name="MessageContainer_Type">
                                                            <xs:sequence>
<xs:element name="DocumentReferenceNumber" type="b2b:DocumentReferenceNumber_Type"/>
<xs:element name="DocumentTransactionType" type="b2b:DocumentTransactionType_Type"/>
<xs:element name="DataTransactionFormat" type="b2b:DataTransactionFormat_Type"/>
<xs:element name="NumberOfDataTransactions" type="b2b:NumberOfDataTransactions_Type" minOccurs="0"/>
                                                                           <xs:element name="Payload" type="b2b:Payload_Type"/>
                                                            </xs:sequence>
                                             </xs:complexType>
                                             <xs:complexType name="Payload_Type">
                                                            <xs:sequence>
                                                                           <xs:any processContents="skip" namespace="##any"/>
                                                            </xs:sequence>
                                             </xs:complexType>

The expectation from the creators of the web service is that the payload will be a block of xml defined by a separate xsd (we don’t have control of this).
When I try and use this web service as a function in a data flow it does not allow anything to be passed into the payload. It errors on validation with a message like

Have you encountered a web service that is set up in this manner and have you found a way for Data Services to pass XML in to it?

many thanks for any help.

Jonathan

icytrue (BOB member since 2006-04-10)

system · January 27, 2016, 5:18am

for data services you will need fixed schema, WSDL can have element declared as of type “any” but if you want to use this in DS you will have to provide the schema of the data that you are going to process and depending on your requirement, you may have to create multiple WSDL with possible schema that you want to process

can you share the WSDL file ?

manoj_d (BOB member since 2009-01-02)

system · January 27, 2016, 4:05pm

thanks Manoj

We have “hacked” our local copy of the WSDL so that it asks for the content of the separate XSD that defines what the owners of the web service are expecting as the payload. Data Services then can pass the values in as appropriate.
The hope is that when BODS connects to the actual we service when it is available there will be no issue as the block of XML will just slot in to the XS:ANY portion of the actual web service.

Does that sound likely and practical?

Naturally we will have the extra burden of having to maintain our altered WSDL if the master WSDL or XSD files change.

Jonathan

icytrue (BOB member since 2006-04-10)

system · February 11, 2016, 11:42am

Hi Manoj and other BODS web service experts

The web service that I have been discussing is now available, however I am hitting a new issue that I am hoping someone can help me with.

When I run my job that calls the secure web service I have a problem that the BODS does not fire out a SOAP request at all (according to our network team there is no message from the server at all).
This is using a Web Service Data Store.

When I try the same but using an Adapter data store (which works fine for HTTP web services) I get the following error:

might this be the cause of the failure to launch for the Web Service Data Store version or is this a separate issue only experience by the Adapter Data Store version.

I don’t seem to be able to find a comprehensive and clear step by step guide for setting up BODS to work with a secure web service.

I have modified axis2.xml to enable https - is that used by both the Adapter Data Store method and WebService Data Store method?

Any help in progressing this by either Data Store type would be much appreciated.

thanks

Jonathan

icytrue (BOB member since 2006-04-10)

system · February 12, 2016, 5:05am

if you are using Adapter Datastore, then there is no need of making any changes to axis2.xml, it is not used by the Adapter datastore

for the Adapter Datastore, you will have to get the Server Certificate and import that into a key store and specify the location of the keystore in the datastore parameter (include the keystore file name also), this file should be on Adapter job server

manoj_d (BOB member since 2009-01-02)

system · February 17, 2016, 11:35am

Hi Manoj and other BODS / Web Service experts,

We have been working away and now have a much clearer picture of what is required and where we have questions regarding Data Services.

We have been pursuing a parallel approach of trying both the web service data store and the adapter data store method.

With each we have reached a blocker and so I have come back to this thread.

The goal is to have Data Services interact with a web service in a mutual authentication scenario.

Adapter

With the Adapter route we are seeing a 403 Access Denied error. This would fit with the mutual authentication scenario.
Currently the adapter points at a keystore that has the target web service certificate as well as our signed client certificates in, but my understanding now is that for the communication to work data services would need use the private key of our signed certificates. This resides in the .pfx file - can the Adapter work with a .pfx file?

Web Service Data Store

We seem to have reached the same point as the poster Clint in this thread

I am not sure whether this is a manifestation of a bug as suggested in the thread or more an expected behaviour with the current set up in this mutual authentication scenario, as data services may not be able to handshake correctly with the target web service.

So we need confirmation as to whether this is a bug associated with the version (We have 14.2.3.549) or not.
If not I then need to know if the web service data store method can work with a .pfx or not.

To summarise - can Data Services (whether web service data store or Adapter data store or any other method) use a pfx file to achieve mutual authentication required to interact with this target web service?

many thanks to any contributors who can provide answers to any of the questions posed.

Jonathan

icytrue (BOB member since 2006-04-10)

system · February 17, 2016, 12:11pm

Hi guys

we are facing a very similar issue at my organisation.

@Manoj. Any help or info you guys uncover will be a big help.

Thanks

JJ

jane123 (BOB member since 2006-02-22)

system · February 21, 2016, 10:26pm

there are option in axis2.xml if you are using webservices datastore which are for specifying the client certificate, but I haven’t worked with webservices requiring client certificate, but I think it is possible to make it work with Webservices Datastoer and I guess with Adpater too

do you have a case open for the issue that you are having with the webservices ? if not will it be possible for you to do that ? please open a case describing the error you are getting when using Adpater and also webservices data store, attach the axis2_log.txt or adapter error log to the case and let me know the incident #

manoj_d (BOB member since 2009-01-02)

system · February 22, 2016, 9:50am

Hey Manoj,

Great to hear from you again. We have raised an incident and it does have the axis2.xml and axis2_log.txt attached.

The incident number is 90000927496

We may have mentioned your name when we raised the call

thanks

Jonathan

icytrue (BOB member since 2006-04-10)

system · February 22, 2016, 6:52pm

whom did you open this case with ? this doesn’t looks like a SAP Message #
ok, found it, the number is 16176 / 2016

manoj_d (BOB member since 2009-01-02)

system · February 24, 2016, 10:39pm

Hi Manoj,

Thanks very much for the time that you spent with my colleague Mike yesterday. It has really moved us on and we now have the two way authentication working for both the Adapter Data Store method and the Web Service Data Store method.

For interested parties the answer was that there is indeed a bug with the Web Service Data Store method which is fixed in a later version.
Also you need to have your certificates in PEM format rather than CER, JKS or PFX.

We have now hit another issue, which might warrant another thread, but as we have already discussed it in this thread in earlier posts I will pose the questions here.

As stated earlier our target web service is using xs:any which is intended to allow any content for that element. However BODS does not work with that concept as it sees a nested level but no elements for it to pass anything into.
Manoj has suggested that we would need multiple WSDLs with the various structures that we will be passing defined within them.

This looks like the route we will take, but I have a few questions to help me be sure:

Is there any way that BODS can recognize where an element is optional (as defined in the WSDL/XSD)? A special secret setting somewhere? As that would allow us to have one WSDL containing all the possible content and just pass in the structures that are valid for that row of data.
Or a different approach - is there any way within BODS that we could pre-form the required XML for the xs:any part of the web service and then use some mechanism (in BODS or maybe a called script) to pass that XML to the web service and then capture its response?
I did wonder if I could tweak the WSDL/XSD so that instead of xs:any the element was defined as a string and that I would then pass the pre-formed XML into that element. When I try this I see the XML go through URL-encoding. Presumably this is because it recognizes the special characters would break a web service if they were values within an element and thus encodes them. Is there any way around this or is this an idea with no mileage?

Any answers or thoughts much appreciated.

Jonathan

icytrue (BOB member since 2006-04-10)