Here is something to put on the radar. If SAP/BO implements in-memory analytics across the BO suite next year outside of the SAP BW architecture, will this impact your current data warehouse architecture and /or your data warehouse plans?
Thanks for the link. It didn’t really help. She mentions in-memory technology, and even has a link to another article she has written about it. That article talks about the benefits, but still doesn’t explain what it is. Now I’m really curious.
This link to the SAP Netweaver Business Intelligence Accelerator on the community website may help explain SAP’s approach to in-memory processing across BW.
Thanks, daiban. That really helps. Since the database resides entirely in RAM, its size is quite limited. The technology, it seems to me, would be primarily limited to datamarts. However, this could be perfect for a substantial amount of reporting.
I dont think so InMemory can required any additional or changes in existing architecture. Its has to do more with processing rather than anything else. Its more of data processing and i dont think it impact the existing architecture.
Size may be going away as an issue Michael. In the 1960’s IBM dreamed of having memory be so cheap that everything would be stored in memory. I worked at Moody’s about 10 years ago and they had the entire Ratings database in memory on an RS6000 back then. That is why when you call for a rating the customer service people can look it up and give it to you almost instantly.
Yes, there are still some limitations but we keep getting closer to the IBM idea from the '60’s every day.
Hasso used his now-familiar blackboard style to educate the audience on the revolutionary promise of in memory analytics. As an interesting contrast, he pointed out that the much hyped-advances of cloud computing and virtualization are valuable, but are just fundamentally delivering existing solutions in a new way.
Hasso laid out a simple vision, even if the math makes it a bit complicated. He believes that an individual computing blade will soon have 8 CPUs with 16 cores each. Each of these blades will be able to hold up to 500GB of memory. With 10 blades, you would have 5 TBs and 1280 computing elements.
Using column store compression, raw transactions can be compressed by a fact of 10. This means that this 10 blade system can hold the equivalent of 50 TB of data. To put this in context, this is approximately the size of the printed collection of the U.S. Library of Congress. Or, large enough to run the combined financials of 70 companies.
The column store compression makes querying the data fast, as well. Hasso again demonstrated SAP BusinessObjects Explorer, reinforcing the fact that it can access 1B records in less than a second. In a twist, he showed the analogous demo from within Microsoft Excel, pointing out the limitation to the current non-accelerated version of Excel.
10 years from now, we will have maybe 10x or 100x times more RAM and much faster and many more CPUs tha now but also the volume of data will increase in the same factor. So still, technology could be behind the requirements…
I am not skeptical, not at all. Just realistic
The concept of working from RAM rather than physical disk is not new (RAM disk anyone? from the DOS days? ). Clearly RAM is going to respond faster than a physical device like a disk. However, RAM has historically been volatile, meaning if you turn off your box the contents of memory are lost. Disk is non-volatile meaning (in theory, anyway) when you turn your box back on it’s still there.
There were quite a few mentions of the “accelerated” version of Explorer at SAPPHIRE. It should be clear that Explorer (renamed from Polestar) can be purchased as a separate product, without the acceleration. It’s only with the hardware that it gets really sexy. Can I say sexy? I guess I can, I just did.
I want to do more reading about the core technology behind the in-memory bits. I don’t really care about the Explorer front end, because frankly any front-end that you put on an in-memory database is going to fly in terms of performance. I’m curious about things like…
Is the in-memory database truly non-volatile? If not, how is it reloaded, and at what cost, when the system is restarted / rebooted?
How does the data get there to begin with? From the sessions that I have seen, you don’t do any aggregate tables or partitioning of any kind, you just load the data. But how does that process work, and how long does it take?
Is the data reliable? For example, with disk we have RAID where there is built-in redundancy. If one disk fails, the system is still up. How does this affect the in-memory database?
What is the compression and how effective is it? I was told it’s a fairly substantial number (meaning if you have a 2TB database you don’t need 2TB of RAM to store it in memory) but I can’t remember the numbers right now.
And if you want to take an interest in the concept but don’t want to use Explorer… what about building your own memory device and pointing Oracle to it? Is that even possible?
Good points Dave, I was also thinking about few of them, mainly about how all data gets initially to RAM and how to add new data, like daily increments, or maybe the whole data set needs to be loaded again?
I would also like to read about how it works when many concurrent users access the same data set, is the performance still OK?
Then what if I have few smaller, independent data marts - is it possible to load all of them into RAM, into independent “structures” so users could use any of them based on their needs?
SAP did mention that they plan to have an agnostic accelerator to interface with disparate data sources next near. That can be used for BO client tools besides Explorer
My Sinclair ZX Spectrum was powered on and didn’t have a hard drive. It ran on 48k of RAM and software was loaded from a cassette tape and took, typically, four to five minutes. That was in 1982.
14 years later I was using BusinessObjects on a Pentium 75 running Windows 3.11 for Workgroups with a 40MB hard disk
14 years later still I’m using a P4 with 3.4Gb of RAM and a 100Gb hard drive running Windows XP and grumbling about the performance of it!
14 years from now I expect solid state storage to be the norm, probably with RISC processors and greater use of thought transmission as a user input device.