BusinessObjects Board

JASON, SOCIAL, SENTIMENTS,TWITTER, FACEBOOK

dataservices
#1

http://scn.sap.com/thread/3204013

http://scn.sap.com/community/data-services/blog/2012/07/31/using-sap-bo-data-services-text-data-processing-for-entities-extraction-from-rss-feeds
http://scn.sap.com/community/data-services/blog/2012/07/27/on-using-data-services-for-twitter-data-sentiment-analysis
http://scn.sap.com/community/data-services/blog/2012/07/17/jsonadapter-for-data-services-helps-to-bring-unstructured-text-data-into-sap-ecosystem

https://ideaplace.brightidea.com/ct/ct_a_view_idea.bix?c=34503CB9-F213-4EF9-8603-E500CB16D712&idea_id=3568C85D-392C-4A64-9048-35553A2F8E06#C820CC4D0

sjain :india: (BOB member since 2009-04-17)

#2

That Jason guy must be really popular :rotf:

Leigh Kennedy :australia: (BOB member since 2012-01-17)

#3

Yes.

However, we want to hear from SAP/BusinessObjects that it is readily available even for XI3.2.

sjain :india: (BOB member since 2009-04-17)

#4

Twitter, Facebook, Social I understand.

JSON does not belong to above group as it is a TCP/IP based protocol only. It’s like saying you want Twitter and HTTP.

But I keep hearing JSON often, so what are the concrete sources that speak this protocol?

Werner Daehn :de: (BOB member since 2004-12-17)

#5

MY understanding was JSON was like XML, but less verbose, not a protocol so much as a files/stream format.

Leigh Kennedy :australia: (BOB member since 2012-01-17)

#6

Okay, then let’s say JSON is a file format like object including a custom transfer.

What I am struggling with is

  • should we support JSON as we support file formats - all flexible but you have to build it?
  • should we support the actual sources like we support SAP ERP although it is using files under the cover, we just hide that?
  • do we need both? Which one is more important?
  • if we simplify writing new Java adapters, would that be the most acceptable solution?

Take Twitter as example with above cases

  • We could support JSON and you have to deal with the oddities and details on how to use JSON with Twitter, every single user
  • We could have a Twitter connector that you simply use as source
  • We could do both to give you both options
  • We could enable you writing a Twitter Adapter with a few lines of code

Long term we need most options anyhow.

The other aspect is the call-direction with JSON. At first sight it is simple, you use JSON like calling a remote procedure: “Dear Twitter, return all Tweets about xyz”. In DS this would look like calling an e.g. lookup_ext() function that returns multiple rows.

A Twitter Stream API would not be possible with this approach. There you would tell: “Twitter, please start sending Tweets, I am listening here”. So not the function gets back the data, you need to have a second listener running, a realtime dataflow in DS terminology. see https://dev.twitter.com/docs/streaming-apis

For an Adapter, no problem at all.

Werner Daehn :de: (BOB member since 2004-12-17)

#7

From my point of view, Id prefer you take the simplify building adapters route. And not having to use Java would be a good start. If you could use Perl, Python or Ruby to write adapters (maybe you could use jruby or jython to wrap the existing java ones), I think you would find a vibrant community would spring up around this.

Nothing wrong with Java, its just that it is rare for ETL people to have Java skills, where as almost all of them have scripting skills.

Leigh Kennedy :australia: (BOB member since 2012-01-17)

#8

At the moment if you want to use the Adapter SDK you need to be an expert in Java, in DataServices and have used the Adapter SDK for a week about. For some reason it is hard to find somebody with that skill set. :wink:

My goal with the Adapter SDK is that every average Java developer can write new Adapters, no need to know the SDK that much with all its options and you don’t need to be an ETL person.
In future one person would extend the methods needed for the Adapter e.g. produce a list of source tables, what columns does a table have. And you would call the Adapter from DataServices to see if all works. So both people can stick to their natural environment, then one looking at Java only, the other using Designer only.

I will not be able to achieve that goal completely but with some documentation we are almost there.

Werner Daehn :de: (BOB member since 2004-12-17)

#9

Would you consider wrapping the java so it can be used with Jruby or JPython ?

Leigh Kennedy :australia: (BOB member since 2012-01-17)

#10

not sure.

Werner Daehn :de: (BOB member since 2004-12-17)

#11

Well we are a Microsoft shop, therefore we don’t have any Java developers. Even though we have some senior people who have done java before and could probably wing it, management would not allow us to use java for support and religious reasons.

Why not do something like this, but release it as open source / unsupported ? Microsoft have a community called Codeplex which is basically a combination of free, open source and unsupported extensions to SQL server tools. You could do something similar (perhaps as an offshoot of the wiki).
This would also encourage people to contribute the adaptors , functions, etc they have developed to be used by others. It would probably take at most 1/2 headcount for you to manage.

Just a thought…

Leigh Kennedy :australia: (BOB member since 2012-01-17)