Cannot Login (HTTP Status 500) Tomcat/Apache

Following a reboot, a webi server is returning a HTTP Status 500 error when you try to login.

Version 6.5 (SP2), I have stopped / restarted the apache server and tomcat (do these need to be done in a specific order?)

I havent pasted the full page (I can if it means anything to anyone) but do these lines indicate the problem to anyone?

exception

org.apache.jasper.JasperException: WebIntelligence SDK / JSP Exception — Number :19002 — javaError : java.lang.RuntimeException: Exception while binding APIBROKER (min load search policy) : com.bo.asf.proxy.ASFSmartProxyException: ERROR : unable to bind the object (IDL:w2.bo/wiapibroker/serverobject:1.0) : org.omg.CORBA.BAD_INV_ORDER: attempt to establish binding after ORB shutdown vmcid: OMG minor code: 4 completed: No — Description : UNKNOWN EXCEPTION

root cause

com.bo.wibean.WIException: WebIntelligence SDK / JSP Exception — Number :19002 — javaError : java.lang.RuntimeException: Exception while binding APIBROKER (min load search policy) : com.bo.asf.proxy.ASFSmartProxyException: ERROR : unable to bind the object (IDL:w2.bo/wiapibroker/serverobject:1.0) : org.omg.CORBA.BAD_INV_ORDER: attempt to establish binding after ORB shutdown vmcid: OMG minor code: 4 completed: No — Description : UNKNOWN EXCEPTION


rob_bo :uk: (BOB member since 2005-08-04)

Since this is not a report-building question, I have moved your topic from “Building Reports” --> WebI v. 6 sub-forum to “Server Administration, Installation, Upgrades” --> WebI 6 Servers sub-forum.


Bobarella (BOB member since 2004-02-12)

What was changed?
Without getting into the specific error details - i’m guessing that your reboot did not shut down the server apps!
Try this manually - shut down webi, then Apache, the Tomcat.
Wait a while or check for exe’s - then, start Tomcat, then Apache, then webi.
That’s the correct order.
Have you got them all to start up as a service?
Have you installed Tomcat Manager and Admin?
What happens if you only use the web and app server urls?

If still experiencing problems - recreate the application and web server components for Infoview - hopefully the ORB is still ok.


MikeD :south_africa: (BOB member since 2002-06-18)

Thanks for the update.

They are all set up as services, but my thoughts are that they are not starting in the right order, so i will try this.

Thanks

Rob


rob_bo :uk: (BOB member since 2005-08-04)

AArrggghh!! Business Objects is driving me NUTS!!!

I’ve tried doing as you suggested, and now we are getting a different message:

HTTP Status 503 -

type Status report
message
description The requested service () is not currently available.

I’ve checked all the services are running. All the services were started in the order you specified (Tomcat, Apache then Webi) and using the proper shortcuts from the appropriate menus (i.e. not in the control panel/services window)

I’ve searched for this error but there doesnt seem to be anything that is relevant to the problems I have


rob_bo :uk: (BOB member since 2005-08-04)

Are you able to login to the Admin Console from the Server PC?


Sridharan :india: (BOB member since 2002-11-08)

I didn’t think that it would resolve the issue, but did you try the http://server:8001 and http://server:8080 or whatever ports you assign to Apache and Tomcat ie Do you get the correct site displays for these products?
Have you checked the logs - sometimes you reach space thresh holds on the server and the auto recycle flush gets all messed up?

What versions of Ap/Tom? Did you download and install the Tomcat Manager and Admin modules?

The problems you’re experiencing are typical for when an OS SP has been applied or when a service/exe did not shut down cleanly before a sever shutdown.
Check that no patches where applied to the OS.
Then - hopefully you don’t have corrupted COM’s hanging around, but I’d suggest deleting the cluster - back up doc storage 1st!!
Take some time to do it in reverse stages and delete the app / web definitions, then Infoview, then the ORB, the mycluster.
Sure, you could just delete mycluster, but I’ve learnt not to rely on high level removals.
Remove the webi status icon from the desktop display.
Reboot.
Shut down Tomcat / Apache then:
Recreate the cluster (remember the order of application then web for Infoview etc!!)
Recreate the bomain.key - repoint/restore storage.
Manually Start Tomcat, Apache and then Webi.
If this works - try an auto shutdown and restart.

If this still doesn’t work - mail me at Business_Intelligence@hotmail.com during working hours (US EST) and I will ask you for some logs and maybe get you to check the COM definitions - i don’t pay for calls & don’t mind helping.


MikeD :south_africa: (BOB member since 2002-06-18)

This goes against what MikeD said, but I’ve seen BO suggest in a few places that you’ll want to start everything in this order:

  1. Webi
  2. Tomcat
  3. Apache

The places I’m referencing:

Edit: please note the 2nd reference is for Solaris and BO Version 6.0. However, I included this reference because it’s the only explanation as to why services should be started in a particular order.

We have Tomcat and Apache on separate servers, using the mod_jk. Because of this, we usually wait 5 minutes after Webi’s up to start Tomcat, and then wait 5 minutes after Tomcat starts before we bring up Apache. This seems to give us the greatest stability.

In addition, if you’re using Application Foundation, it can be more sensitive to things starting up in the proper order…

HTH -

DJ


DJ06482 :us: (BOB member since 2002-11-22)

I’ve seen enough strange server behaviour to know not to defend my suggestion, but would just like to point out that your ref was for Solaris and 6.1.
The Tomcat then Apache order is the key!
But - from my experience http 500 issues generally occur due to a previous bad shutdown - so whatever sequence works - stick with it, and in my case I prefer to know that the web and app servers are running before firing webi up and likewise when shutting down.
I also always stop webi before rebooting the system.


MikeD :south_africa: (BOB member since 2002-06-18)

We are experiencing similar problems with Apache/Tomcat. I have a case open with BO tech support which hasn’t helped much, hence the need for the BOB forum.
Here’s the symptom: Sporadically, users cannot login. It happens once then may not occur for several weeks, then twice in one week. Very random.
Our solution for a 7 months now has been to reboot all 3 servers in our cluster. First the primary then the secondary nodes. After reboot we stop Apache then Tomcat then restart in reverse order and all is well for a time.
BO support says they cannot help us further unless we turn on trace logging. This creates such a load on our services that we cannot just turn it on. Also, to turn it off we have to reboot the server anyway, thus eliminating the original problem for several weeks at a time. We have reviewed Apach/Tomcat logs as well as the Task Manager during the condition but nothing relevant has developed.
In June 2006 Hotfix 461 was installed to eliminate a different problem with report names in Infoview. When we had BO 5x the servers were on a weekly reboot schedule. Since BO 6.5 was installed, we were told we did not have to reboot anymore. Yet the problem has returned.
I am not that familiar with Apache/Tomcat more with IIS. I don’t have the Apach/Tomact Manager installed.
If anyone has any suggestions, please post them.
We are on WINNT servers, BO 6.5.1, Apache-Tomcat 4.1 and Apache 2.0.46(win32)mod_jk 1.2.4.

Thanks to everyone for past support!


DaveMC :us: (BOB member since 2002-08-20)

MikeD -

I agree with you 100% that there’s no “correct” way to bring up the servers, as each installation is different. The best way is whatever way gives you the best performance and stability, and as you can see, it works differently for everyone. Between the versions of BO, Apache, and Tomcat as well as deployment-specific considerations (AF vs. no AF, clustering, putting Apache and Tomcat on separate servers, etc.), it’s impossible to generate any hard and fast rules in terms of startup order.

I’ve edited my original post to point out the platform and version differences in the 2nd reference. I realized that the second reference wasn’t a 1 to 1 match, but I thought it was still relevant because it explained the timing of the connectivity between Tomcat and Apache. As far as I know, the basic mechanisms of that communication haven’t changed with the incremental version upgrades from 6.0 to 6.5, but if anyone knows differently, please speak up!

I decided to include the second reference because it was consistent with the first (from the 6.5.1 Documentation for Windows), and because it’s the only place where I’ve seen BO explain the reasoning behind the startup order of Webi, Tomcat, and Apache. It’s too bad that they generally don’t do a better job of explaining why things should be done in a specific order.

I agree that the Tomcat / Apache order is definitely the key, and I think our installation is more sensitive to when Webi is started because we’re also using AF. (AF with Apache/Tomcat has had all kinds of stability issues that are well-documented here on BOB :)) In our particular deployment, Tomcat and Webi are on seperate servers, which also increases the complexity of the communication between the two.

When I’ve personally administered servers, I’ve always found it a good practice to bring everything (Apache, Tomcat, and Webi) down before a reboot. I just wish those people who are currently administering our servers would do the same!

Thanks,

DJ


DJ06482 :us: (BOB member since 2002-11-22)

Dave -

Rebooting 6.5.1 servers on (at least) a weekly basis seems to be the norm. If you search under the 6.x Knowledge Base (on Business Objects’ Tech Support website) for “mod_jk”, you’ll see some suggestions for upgrading Apache/Tomcat, as well as some tuning suggestions. Another solution is to copy everything from your Apache directory over to the wijsp directory in Tomcat and run Tomcat in standalone mode (no Apache). It might be something worth testing if the issues are causing major problems for your users. The best solution is probably to upgrade to XIR2, which utilizes newer versions of Apache/Tomcat. However, that might not be practical, depending on your deployment and upgrade strategy.

HTH -

DJ


DJ06482 :us: (BOB member since 2002-11-22)

I recently convinced the US arm of our global org to migrate from IIS to Apache/Tomcat and since then we have never had any issues with connectivity, BUT - I reboot all the nodes weekly or bi-weekly during a regular maint window on Friday afternoons - or during the weekend.

As to the 6.0 to 6.5 mention - work on the premise that version 6 only somewhat stablised from 6.1.3 onwards! I should know as I was leapfrogging right behind the releases to resolve some major webi issues :yesnod:

Your mod_jk mention is wise as this handles the comms, but once configured generally does not cause any issues (the .14 version is a good choice).
I have used the autoconnect function and it works like a bomb.
Apache/Tomcat minimum any releases above the initial v5.0 and v2.0 versions (i.e never use the 1st major release).

As to webi - all the issues i’ve dealt with at our various sites have been due to the configuration or Corba.
And this typically occurs when people deviate from the default suggestions/values/installation guide without understanding what they’re doing.
Or they try re-installs / re-configs to correct an issue without knowing what the actual cause was, but it generally helps resolve most of the common problems.
This can mess the DCOM’s / Corba up.
To resolve webi issues
I start with the Global Report and all the logs - i.e. I normally enforce a period of enabling all possible logging methods on webi and the web and application servers - 90% of the time, the answer is in there.

And - typically, people forget to constantly tweak the system as loads increase, so the config settings don’t use the default memory flush cycles as often as they should or they conflict with the increased activity.

As to removing Apache - yes and no = Apache IS faster serving up web content and I prfer dealing with web and app issues separately.


MikeD :south_africa: (BOB member since 2002-06-18)