Administrators Cockpit Error: Unable to Generate Data...

system · October 1, 2018, 2:39pm

Good morning All,

Recently we ran into the dreaded error: Unable to Generate Data: Service URL must start with service.jmx: (RWS 00030) Error, but only in QA. I thought this may have been caused by a Windows update after checking our Production environment.

However; today, after reboot, our Production environment is now doing the same thing, after being up for 8-days and working prior to reboot.

All the articles talk about redirect, which we do not have, and the service itself does indeed start with service:jmx:boeiiop. At this point, I am at a loss. The APS is running and jobs are running. Probes are working, but the monitoring service is not.

Any help or a directions for a jumping off point is much appreciated.

hmnathanstotts (BOB member since 2018-03-08)

system · October 1, 2018, 2:41pm

APS logs might have something?

Nick Daniels (BOB member since 2002-08-15)

system · October 1, 2018, 3:22pm

Looking in the text file in Notepad, all I see are Allocation Failures.

6.404: [GC (Allocation Failure) [PSYoungGen: 131584K->17496K(153088K)] 131584K->17504K(502784K), 0.1447698 secs] [Times: user=0.34 sys=0.00, real=0.14 secs]
28.736: [GC (Allocation Failure) [PSYoungGen: 149080K->21499K(284672K)] 149088K->24772K(634368K), 0.0613770 secs] [Times: user=0.14 sys=0.03, real=0.06 secs]
28.865: [GC (Metadata GC Threshold) [PSYoungGen: 25346K->20517K(284672K)] 28619K->23799K(634368K), 0.0343126 secs] [Times: user=0.00 sys=0.02, real=0.03 secs]
28.900: [Full GC (Metadata GC Threshold) [PSYoungGen: 20517K->0K(284672K)] [ParOldGen: 3281K->20826K(349696K)] 23799K->20826K(634368K), [Metaspace: 20355K->20355K(282624K)], 0.1383893 secs] [Times: user=0.33 sys=0.01, real=0.14 secs]
35.297: [GC (Allocation Failure) [PSYoungGen: 263168K->18343K(534528K)] 283994K->39234K(884224K), 0.7647694 secs] [Times: user=0.78 sys=0.02, real=0.77 secs]
45.781: [GC (Allocation Failure) [PSYoungGen: 534439K->21495K(544768K)] 555330K->72999K(894464K), 0.6030766 secs] [Times: user=0.78 sys=0.05, real=0.61 secs]
52.404: [GC (Allocation Failure) [PSYoungGen: 544759K->45561K(633856K)] 596263K->100185K(983552K), 0.1310435 secs] [Times: user=0.28 sys=0.02, real=0.13 secs]
63.566: [GC (Allocation Failure) [PSYoungGen: 633849K->27352K(643584K)] 688473K->81984K(993280K), 0.4454065 secs] [Times: user=0.22 sys=0.00, real=0.45 secs]
70.175: [GC (Allocation Failure) [PSYoungGen: 615640K->31011K(636416K)] 670272K->91749K(986112K), 0.1477436 secs] [Times: user=0.23 sys=0.00, real=0.16 secs]
77.563: [GC (Allocation Failure) [PSYoungGen: 622371K->24836K(641536K)] 683109K->86084K(991232K), 0.1307544 secs] [Times: user=0.11 sys=0.02, real=0.14 secs]
84.969: [GC (Allocation Failure) [PSYoungGen: 616196K->34727K(647680K)] 677444K->97748K(997376K), 0.0537340 secs] [Times: user=0.17 sys=0.00, real=0.06 secs]
90.072: [GC (Allocation Failure) [PSYoungGen: 634279K->28861K(648704K)] 697300K->93186K(998400K), 1.1330542 secs] [Times: user=0.17 sys=0.00, real=1.13 secs]
95.405: [GC (Allocation Failure) [PSYoungGen: 628413K->47072K(631296K)] 692738K->125494K(980992K), 0.1047028 secs] [Times: user=0.20 sys=0.02, real=0.11 secs]
98.110: [GC (Allocation Failure) [PSYoungGen: 631264K->54955K(639488K)] 709686K->165185K(989184K), 1.0009300 secs] [Times: user=0.44 sys=0.00, real=1.00 secs]
108.246: [GC (Allocation Failure) [PSYoungGen: 639147K->24257K(634368K)] 749377K->136726K(984064K), 0.0930505 secs] [Times: user=0.14 sys=0.00, real=0.09 secs]
115.386: [GC (Allocation Failure) [PSYoungGen: 599233K->43980K(619008K)] 711702K->165841K(968704K), 0.0465284 secs] [Times: user=0.20 sys=0.00, real=0.05 secs]
127.669: [GC (Allocation Failure) [PSYoungGen: 618956K->33737K(638464K)] 740817K->155868K(988160K), 0.0546586 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
133.444: [GC (Allocation Failure) [PSYoungGen: 609225K->43552K(635904K)] 731356K->182071K(985600K), 0.0703237 secs] [Times: user=0.22 sys=0.02, real=0.06 secs]
136.254: [GC (Allocation Failure) [PSYoungGen: 619040K->44848K(635392K)] 757559K->206607K(985088K), 0.0804058 secs] [Times: user=0.23 sys=0.03, real=0.09 secs]
138.142: [GC (Allocation Failure) [PSYoungGen: 619312K->26161K(636416K)] 781071K->209591K(986112K), 0.0549870 secs] [Times: user=0.25 sys=0.00, real=0.06 secs]
140.780: [GC (Allocation Failure) [PSYoungGen: 600625K->46768K(634368K)] 784055K->243599K(984064K), 0.0662959 secs] [Times: user=0.25 sys=0.00, real=0.06 secs]
143.517: [GC (Allocation Failure) [PSYoungGen: 619696K->41871K(635904K)] 816527K->238829K(985600K), 0.0549866 secs] [Times: user=0.16 sys=0.02, real=0.06 secs]
145.942: [GC (Allocation Failure) [PSYoungGen: 614799K->33322K(632320K)] 811757K->231165K(982016K), 0.0409242 secs] [Times: user=0.20 sys=0.03, real=0.05 secs]
148.550: [GC (Allocation Failure) [PSYoungGen: 607274K->24769K(634368K)] 805117K->223252K(984064K), 0.0316504 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
150.798: [GC (Allocation Failure) [PSYoungGen: 598721K->17703K(636928K)] 797204K->217015K(986624K), 0.0472219 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]
153.525: [GC (Allocation Failure) [PSYoungGen: 594727K->57881K(635392K)] 794039K->258354K(985088K), 0.0962003 secs] [Times: user=0.28 sys=0.00, real=0.09 secs]
159.632: [GC (Allocation Failure) [PSYoungGen: 634905K->23955K(630272K)] 835378K->224751K(979968K), 0.0228060 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
161.459: [GC (Allocation Failure) [PSYoungGen: 590739K->40459K(632832K)] 791535K->248548K(982528K), 0.0763582 secs] [Times: user=0.20 sys=0.00, real=0.06 secs]
162.968: [GC (Allocation Failure) [PSYoungGen: 607243K->35863K(625664K)] 815332K->267477K(975360K), 0.0608297 secs] [Times: user=0.25 sys=0.02, real=0.06 secs]
166.681: [GC (Allocation Failure) [PSYoungGen: 603671K->26205K(629760K)] 835285K->278827K(979456K), 0.0615485 secs] [Times: user=0.16 sys=0.00, real=0.06 secs]
169.406: [GC (Allocation Failure) [PSYoungGen: 594013K->22019K(634880K)] 846635K->285600K(984576K), 0.0432724 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
171.408: [GC (Allocation Failure) [PSYoungGen: 598019K->49151K(625152K)] 861600K->315645K(974848K), 0.1002145 secs] [Times: user=0.25 sys=0.00, real=0.11 secs]
175.799: [GC (Allocation Failure) [PSYoungGen: 625151K->48278K(630784K)] 891645K->344646K(980480K), 0.1465095 secs] [Times: user=0.34 sys=0.00, real=0.16 secs]
179.906: [GC (Allocation Failure) [PSYoungGen: 615062K->39968K(632832K)] 911430K->360584K(982528K), 0.0866886 secs] [Times: user=0.33 sys=0.03, real=0.09 secs]
179.993: [Full GC (Ergonomics) [PSYoungGen: 39968K->0K(632832K)] [ParOldGen: 320616K->102400K(349696K)] 360584K->102400K(982528K), [Metaspace: 34009K->34009K(294912K)], 0.2080833 secs] [Times: user=0.89 sys=0.00, real=0.20 secs]
182.295: [GC (Allocation Failure) [PSYoungGen: 566784K->50832K(629760K)] 669184K->153233K(979456K), 0.0617394 secs] [Times: user=0.23 sys=0.00, real=0.06 secs]
185.421: [GC (Allocation Failure) [PSYoungGen: 616592K->25575K(632320K)] 718993K->156642K(982016K), 0.0495181 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]
187.990: [GC (Allocation Failure) [PSYoungGen: 591335K->52813K(626688K)] 722402K->192138K(976384K), 0.0766662 secs] [Times: user=0.28 sys=0.00, real=0.06 secs]
191.986: [GC (Allocation Failure) [PSYoungGen: 613453K->19157K(629760K)] 752778K->158578K(979456K), 0.0645906 secs] [Times: user=0.17 sys=0.00, real=0.06 secs]
193.793: [GC (Metadata GC Threshold) [PSYoungGen: 475456K->14183K(628736K)] 614877K->162438K(978432K), 0.0292751 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
193.822: [Full GC (Metadata GC Threshold) [PSYoungGen: 14183K->0K(628736K)] [ParOldGen: 148254K->70301K(349696K)] 162438K->70301K(978432K), [Metaspace: 56215K->56215K(313344K)], 0.2950660 secs] [Times: user=0.98 sys=0.00, real=0.30 secs]
199.880: [GC (Allocation Failure) [PSYoungGen: 559616K->24912K(584704K)] 629917K->95221K(934400K), 0.0837153 secs] [Times: user=0.31 sys=0.00, real=0.09 secs]
203.529: [GC (Allocation Failure) [PSYoungGen: 584528K->29376K(631296K)] 654837K->107590K(980992K), 0.0391559 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
205.578: [GC (Allocation Failure) [PSYoungGen: 592576K->45280K(628224K)] 670790K->131705K(977920K), 0.0467999 secs] [Times: user=0.22 sys=0.00, real=0.05 secs]
207.820: [GC (Allocation Failure) [PSYoungGen: 608480K->50224K(627200K)] 694905K->159953K(976896K), 0.0762139 secs] [Times: user=0.20 sys=0.00, real=0.08 secs]
209.400: [GC (Allocation Failure) [PSYoungGen: 610864K->30050K(629760K)] 720593K->167340K(979456K), 0.0542348 secs] [Times: user=0.19 sys=0.00, real=0.05 secs]
211.556: [GC (Metadata GC Threshold) [PSYoungGen: 238348K->13812K(629760K)] 375638K->163225K(979456K), 0.0522159 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]
211.608: [Full GC (Metadata GC Threshold) [PSYoungGen: 13812K->0K(629760K)] [ParOldGen: 149413K->90963K(349696K)] 163225K->90963K(979456K), [Metaspace: 93451K->93451K(348160K)], 0.2134428 secs] [Times: user=0.59 sys=0.00, real=0.22 secs]
595.460: [GC (Allocation Failure) [PSYoungGen: 562688K->14274K(630272K)] 653651K->105245K(979968K), 0.0290223 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
1141.404: [GC (Allocation Failure) [PSYoungGen: 576962K->9764K(631296K)] 667933K->108906K(980992K), 0.0310868 secs] [Times: user=0.08 sys=0.00, real=0.03 secs]
1694.154: [GC (Allocation Failure) [PSYoungGen: 573988K->67052K(631296K)] 673130K->186045K(980992K), 0.0661241 secs] [Times: user=0.31 sys=0.00, real=0.08 secs]
1877.683: [GC (Allocation Failure) [PSYoungGen: 631276K->67552K(537088K)] 750269K->247868K(886784K), 0.1510114 secs] [Times: user=0.45 sys=0.00, real=0.16 secs]
2179.320: [GC (Allocation Failure) [PSYoungGen: 537056K->5312K(459264K)] 717372K->253243K(808960K), 0.1249371 secs] [Times: user=0.41 sys=0.00, real=0.13 secs]
2476.643: [GC (Allocation Failure) [PSYoungGen: 458944K->5536K(534016K)] 706875K->253467K(883712K), 0.0148269 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2716.979: [GC (Allocation Failure) [PSYoungGen: 443808K->4992K(428544K)] 691739K->252923K(778240K), 0.0225192 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2956.887: [GC (Allocation Failure) [PSYoungGen: 428416K->5504K(501248K)] 676347K->253435K(850944K), 0.0173298 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
3201.562: [GC (Allocation Failure) [PSYoungGen: 415104K->6016K(402432K)] 663035K->253955K(752128K), 0.0199471 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
3442.640: [GC (Allocation Failure) [PSYoungGen: 402304K->6304K(466944K)] 650243K->254243K(816640K), 0.0173700 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
3736.255: [GC (Allocation Failure) [PSYoungGen: 389792K->6944K(378368K)] 637731K->254883K(728064K), 0.0200973 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]

hmnathanstotts (BOB member since 2018-03-08)

system · October 1, 2018, 3:43pm

Wrong file. It should have a.glf extension.

Nick Daniels (BOB member since 2002-08-15)

system · October 1, 2018, 4:50pm

Funny story, Production fixed itself while I was perusing the logs. But, QA is still doing the same thing. I am going to restart the APS and kick off the logging process to generate a log.

hmnathanstotts (BOB member since 2018-03-08)

system · October 1, 2018, 5:01pm

Nick Daniels (BOB member since 2002-08-15)

system · October 1, 2018, 7:05pm

Alright, here is the most recent log file for our QA environment.

Still, full of errors.
APS_Log.zip (157.0 KB)

hmnathanstotts (BOB member since 2018-03-08)

system · October 2, 2018, 8:10am

I would start by picking out the errors.

Nick Daniels (BOB member since 2002-08-15)

system · October 2, 2018, 12:36pm

The whole file is errors. Any idea what I should be looking for?

I don’t see any errors saying a component failed, just Java type errors.

hmnathanstotts (BOB member since 2018-03-08)