Hi,
Does anybody have any experience with receiving JSON messages through Data Services?
I can’t find much on the web and I want to see if it’s even possible.
Thanks,
b/
Bisquite (BOB member since 2009-07-15)
An update on this. I am thinking that we will need to write a custom adapter that converts the JSON messages to XML. I am not certain this is the correct approach but I have no alternative at the moment.
Bisquite (BOB member since 2009-07-15)
JSON messages are not supported in DS, yes, writing a custom Adapter to convert the JSON message to XML is the only approach for now
manoj_d (BOB member since 2009-01-02)
Just thinking loud…
What if we define a file format with one large string and as file source in the dataflow reader we define the http address. I have never done that but was told it is supported in the meantime. Then we have one row/one column with the entire JSON string and using this technique break it apart.
https://wiki.sdn.sap.com/wiki/pages/viewpage.action?pageId=49414399
Werner Daehn 
(BOB member since 2004-12-17)
Thanks for the reply guys, and the confirmation that the custom adapter is the correct approach.
Werner, I’m not sure I understand your idea entirely. How would I incorporate the file formats into a real-time job. I thought every source for a real-time job had to be XML, or is my understanding wrong?
Bisquite (BOB member since 2009-07-15)
My mistake. I thought you want to use a batch process to grab JSON data once a while.
Werner Daehn (BOB member since 2004-12-17)
Thinking about this some more. Could we take the source class files for the existing JMS adapter, copy them and change them as required?
Is there any existing documentation for the JMS adapter?
Thanks,
b/
Bisquite (BOB member since 2009-07-15)
JMS Adapter and other Adpater shipped with DS are not shipped along with the source code, you can use the testAdapter source code to implement your own JASON Adapter
do you know any publicly available services that can send and receive jason messages ?
manoj_d (BOB member since 2009-01-02)
As a quick&dirty alternative to writing an adapter (which would require learning Java from me, for example), I did the following:
- Use CURL utility to get the file in JSON format on the hard drive.
- Create a file source and parse it in a Data Flow using custom functions.
Works fine for me.
Codehopper (BOB member since 2010-05-14)
Hi.
Yes, this is certainly an option as we already have some java code that will pull the messages from the queue. However operationally we really want to have the real-time jobs running in Data Services so it will make administering the whole of the data warehouse much easier.
b/
Bisquite (BOB member since 2009-07-15)
Update:
I wonder if we could take the installed built-in JMS adapter, decompile the class files and change the code so that we can build DTDs for the JSON objects.
Bisquite (BOB member since 2009-07-15)
The JMS Adapter is just a Java code written with the Adapter SDK, which is included in DS.
Werner Daehn (BOB member since 2004-12-17)
You still can use “custom transfer” option (say, a .bat file) together with a file and run curl right inside your data flow, which may be a part of realtime job then.
You can even create a preparatory data flow that could generate that .bat file with runtime-defined parameters inside.
well, if you look to reuse the existing java code it’s a different story…
Codehopper (BOB member since 2010-05-14)
creating files, having a table with delete-data-before load,… all should be avoided in Realtime dataflows. You might have multiple instances of a realtime job deleeting each others data, the point in time when the delete-event is called is not that obvious,… etc. Be careful!
Werner Daehn (BOB member since 2004-12-17)
I beg the topicstarter’s pardon for moving offtopic, but for the temporary storage file a unique name could be generated using a (global?) variable (not that I tried it myself, though).
It all started for me when I realised that the standard HTTP Adapter has limited capabilities etc.
Creation of a JSON adapter sounds like a nice idea anyway. Has SAP any plans of making one itself? I just don’t want to make duplicate effort…
Codehopper (BOB member since 2010-05-14)
Hey B,
by the way, from what source did you plan to receive JSON messages?
Codehopper (BOB member since 2010-05-14)
It is a JMS queue. Active MQ is the broker being used.
Bisquite (BOB member since 2009-07-15)
There is a wiki page about custom adapters :
http://wiki.sdn.sap.com/wiki/display/EIM/Integration+of+Adapters+into+DI
Werner, it would be great if you could provide a custom adapter for Twitter and Facebook on that wiki page. There is one example for e-mail adapter already.
aderici (BOB member since 2012-01-13)
how to create an custom adapter and where i should write the codes for creating custom adapter and bringing the data into DS?
saphod (BOB member since 2012-07-24)