BO XI 3.1 SP3 FP 3.7 WIS30280

system · January 31, 2014, 4:23pm

Windows 2008 R2 STD
IIS web
Oracle repo
16GB RAM

Everything was fine till december.
Than a problem with one report started.
The main problem is that it returns WIS30280:
The Web Intelligence server memory is full. Please log off and try to
connect later. If the problem persists, contact your Business Objects
administrator. (WIS 30280)

as i can see in a wireportserver log file looks like:

cefactory.cpp:163:class CORBA_Object *__thiscall CDZServerFactory::newService(const unsigned short *,const unsigned short *): TraceLog message 1
2014/01/31 13:23:37.515|>=|E|X| 5672|16092| |||||||||||||||Uncaught Exception in SilentScope: CDZServerFactory::newService: 0
cefactory.cpp:163:class CORBA_Object *__thiscall CDZServerFactory::newService(const unsigned short *,const unsigned short *): TraceLog message 2
2014/01/31 13:23:39.527|>=|E|X| 5672|12692| |||||||||||||||Uncaught Exception in SilentScope: CDZServerFactory::newService: 0
kc3allocator.cpp:2528:void *__cdecl c3_alloc(unsigned int,class c3_allocator *,const char *,const char *): TraceLog message 3
2014/01/31 15:47:26.726|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_ALLOCATOR:No memory left during allocation class: : cdbStateValue [kc3allocator.cpp;2528]
kc3cdbNodeSort.cpp:1178:long __thiscall Cube::CDB::cdbSortNode::KeyOn(const class std::vector<struct Cube::CDB::Query::SortItem,class std::allocator > &): TraceLog message 4
2014/01/31 15:47:26.726|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_Calculator:(A) *** Unexpected exception caught *** : (false) [kc3cdbNodeSort.cpp;1178]
kc3allocator.cpp:2528:void *__cdecl c3_alloc(unsigned int,class c3_allocator *,const char *,const char *): TraceLog message 5
2014/01/31 15:49:00.326|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_ALLOCATOR:No memory left during allocation class: : cdbStateValue [kc3allocator.cpp;2528]
kc3cdbNodeSort.cpp:1178:long __thiscall Cube::CDB::cdbSortNode::KeyOn(const class std::vector<struct Cube::CDB::Query::SortItem,class std::allocator > &): TraceLog message 6
2014/01/31 15:49:00.326|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_Calculator:(A) *** Unexpected exception caught *** : (false) [kc3cdbNodeSort.cpp;1178]
kc3allocator.cpp:2528:void *__cdecl c3_alloc(unsigned int,class c3_allocator *,const char *,const char *): TraceLog message 7
2014/01/31 15:49:56.190|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_ALLOCATOR:No memory left during allocation class: : cdbStateValue [kc3allocator.cpp;2528]
kc3cdbNodeSort.cpp:1178:long __thiscall Cube::CDB::cdbSortNode::KeyOn(const class std::vector<struct Cube::CDB::Query::SortItem,class std::allocator > &): TraceLog message 8
2014/01/31 15:49:56.190|>>|E| | 5672|5076| |||||||||||||||**ERROR:C3_Calculator:(A) *** Unexpected exception caught *** : (false) [kc3cdbNodeSort.cpp;1178]
kdgWICDZProc.cpp:735:void __thiscall WICDZProcBasic::generateResult(class OBFixSeq *&,const char *,const char *,const char *,const unsigned char *,const unsigned char *,class std::basic_istream<char,struct std::char_traits > *,int,bool &): TraceLog message 9
2014/01/31 15:49:56.299|>>|E| | 5672|5076| |||||||||||||||**ERROR:WICDZ_i:Error stream :

<ERROR COMPONENT=“WIS” ERRORCODE=“30280” ERRORTYPE=“USER” MESSAGE=Pamięć serwera Web Intelligence jest pełna. Wyloguj się i spróbuj połączyć się później. Jeżeli problem nie ustąpi, skontaktuj się z administratorem oprogramowania Business Objects. (WIS 30280)" PREFIX=“ERR”>

[kdgWICDZProc.cpp;735]

I was looking at System Manager while the report was scheduled and running and I didn’t see any problem with memory of the Web Intelligence Processing Server that was running the script… and with any other server too.

The report was only report running at that time.
I wounder what is the problem. Why it was running with no problems, and than started to fail with no reason. It was not edited or changed it is just as is.

Adam924 (BOB member since 2011-01-13)

system · January 31, 2014, 4:29pm

Hi,

Maybe an obvious question:
Have you tried restarting the BOBJ server? Did it release memory and help at least temporarily?

Marek Chladny (BOB member since 2003-11-27)

system · January 31, 2014, 4:34pm

Yes, I did try with no result.

Adam924 (BOB member since 2011-01-13)

system · January 31, 2014, 4:42pm

Can you check the report that’s causing this problem? How many queries are there? How many rows do they fetch from the database?

Marek Chladny (BOB member since 2003-11-27)

system · January 31, 2014, 4:51pm

So, there have been no upgrades then?

Have you tried opening the report, purging and saving as a different name? This will generate a new report ID, sometimes this helps.

Mak 1 (BOB member since 2005-01-06)

system · January 31, 2014, 5:01pm

It was developed by the 3rd party firm, and it’s a query with context.
There are also some parameters couse it returns data for a month period.
It is rather complex query.

Adam924 (BOB member since 2011-01-13)

system · January 31, 2014, 6:55pm

Hi,

It does not matter how complex the query is. That’s what a database should be able to handle.

What’s important is the number of records retrieved by the query from the database. If the query pulls enormous number of records back into the WEbI report then the server can have problems to process that volume and that’s where the troubles with memory could be originated.

Marek Chladny (BOB member since 2003-11-27)

system · February 3, 2014, 11:26am

Ok, I’ve managed to check, it is about 18milions (18 000 000) of records returned from orcale to BO.
WiReportServer*32.exe as I recoginse WebIntelligenceProcessingServer was growing to about 1,2-1,3GB of RAM
and than report failed.

Metrics for the server are:
Registered Name com.seagatesoftware.img.osca.webiserver.“WAR01BOGIF2”-WAR01BOGIF2.WebIntelligenceProcessingServer1
Operating System Windows .NET 6.1
CPU Type Pentium (II/Pro)
CPUs 24
RAM (MB) 16 374,988
Local Time poniedziałek, 3 luty 2014 12:25:37 CET
Disk Size (GB) 720,205
Used Disk Space (GB) 30,723
Version 12.3.8.1380
Start Time piątek, 31 styczeń 2014 14:23:36 CET
PID 5672
Host Name XXX
Host IP Address XXX
Request Port 62466
Open Connections 0
Active Requests 0
Cache In Use (KB) 89764
Virtual Memory Size (MB) 204
Maximum Idle Time (seconds) 1200
Idle Document Timeout (seconds) 300
Polling Interval (seconds) 120
Maximum Documents Per User 5
Opened Documents Before Server Recycle 50
Errors On Maximum Map Size Enabled 1
Cache Timeout (seconds) 4370
Document Cache Cleanup Interval (minutes) 120
Idle Connection Timeout (minutes) 20
Maximum Connections 25
Cache Enabling State 1
Real-time Caching Enabled 1
Cache Timeout (seconds) 4370
Maximum Cache Size (KB) 1000000
Maximum Document Cache Reduction Space (%) 70
List Of Values Caching Enabled 1
List Of Values Batch Size (entries) 1000
Maximum Entries for Custom Sort 100
Maximum Character Stream Size (MB) 5
Maximum Binary Stream Size (MB) 250
Maximum Universes in Universe Cache 20
Maximum Documents in Document Cache 0
Memory Watcher Enabled 1
Lower Memory Threshold (MB) 1000
Upper Memory Threshold (MB) 1500
Maximum Memory Threshold (MB) 1800
Secured Cache enabled 0
Maximum List Of Values Size (entries) 100000

Adam924 (BOB member since 2011-01-13)

system · February 3, 2014, 11:43am

I would suggest that this is too many, especially with only 16GB of Ram. I expect that you are only just facing issues as the data volumes have gone up over time.

Mak 1 (BOB member since 2005-01-06)

system · February 3, 2014, 11:51am

I was a bit struggled by this value.
I don’t think tha the 16GB of RAM is a problem.
Bigger problem is 32bit BO that will not hold on more data than 1,8-2GB for single process.

Any suggestion how Can I work this around or any SAP related info about max limit for returned rows ?
I need something official to show for developers and can’t find it.

Adam924 (BOB member since 2011-01-13)

system · February 3, 2014, 11:56am

Agreed, this is what I believe would cause the issue here.
18 Million rows is more of a data dump than a report, this sort of thing should be generated by the DB directly, IMO.

Its not just about the number of rows returned. The complexity of the report or how many rows / columns you are actually displaying has a bearing here.

Mak 1 (BOB member since 2005-01-06)

system · February 3, 2014, 12:28pm

That’s what I feared. It’s too many detailed records to process on the report side.

What you should ask the developer(s) of this report as well as developer(s) of a universe that this report is built on top is why detailed records are fetched from the database and whether any sort of aggregation can be done on the database side (using a SQL that contains GROUP BY clause) so the number of records retrieved from the database would be much much smaller.

Marek Chladny (BOB member since 2003-11-27)

system · February 3, 2014, 1:48pm

OK Guys, Thank You very much, for Your effort and Your knowledge,
You’re Great.
If any one have something to add feel free.

PS. Wonder what SAP would say to this issue

Adam924 (BOB member since 2011-01-13)