I’m getting a strange problem with our BO Server. It’s only happened in the last few days. To my knowledge, nothing has changed on this server.
The server gets rebooted daily and a script runs to restart the crystal services. The CMS one often doesn’t start up first time and we have to manually do it which is why we created a script to restart all services on boot up. This has been working fine for months now.
However, over the last few days, if we look in the BO Servers section, the services are showing that they are down (red arrow) but when we look in control panel/services.msc they are all started. When we run the reports, they are all fine and everything works ok.
Restarting all the services via the script makes BO recognise that they are up and running. If we try and restart them from BO, it gives an error saying that the service is already in the state requested.
Be very carefull with automating reboots and restarts etc.
Always apply a delay and separate out the CMS and FRS’s from the rest as there are a few dependancies.
it has been suggested that to do this properly, one should first disable the servers prior to stopping them - and then in the correct order so as to not have any orphan objects or objects out of synch - but the newer XI versions are more amenable to tackling these issues.
Then as far as windows services go - I had a fun time at one stage with using Perl to auto test all XI services and restart those that were down … and learnt the hard way that you don’t get a true service state in the windows service monitor:
A service can exist in one of three basic states: Running, Paused, or Stopped AND ContinuePending, PausePending, StartPending, or StopPending - which are typically displayed as Stopped
Thanks Mike, very helpful. This was going to be my first suggestion.
Looking through the event logs, there’s no consistency in how they start up each day and other windows services are getting mixed up in between so I’m going to delay it by about 30 minutes to isolate it and see if that makes a difference.
I am using XIR2.
You mentioned to disable first and then stop. I guess we can’t disable the services without stopping in the CCM. Right?
If we need to restart the services in CCM manually, what is the order to follow? I guess CMS needs to be stopped first and then followed by tomcat and then the rest. Is that right?
Disabling first via CCM - this does not stop anything - it simply allows current processes to complete and disallows any new requests.
Typical for a place where you don’t have a maint window and cannot determine if everyone has indeed logged off.
I know that you can check if there are any users logged on via the CMC - but i think I had issues with that info.
Tomcat admin is probably more trustworthy - but it’s the idle users that tend to catch one and here it depends on how you set your idle time outs etc.
The DISABLE suggestion was initiated at a time when XI was pretty new and it was almost impossible to identify and possible repair unsyched entries in the FRS(s) and the CMS. I heard that was resolved.
NOTE: CMS has to be running for the enable / disable function.
As to stopping things - one would assume that it’s a logical top down and bottom up process i.e dependancies / core applications / web&application servers. And if you look at the dependancies on the services - there are almost none - Event Log being the common factor for all.
I then think one has to consider monitoring and logging - as certain instances could trigger events or alarms - depending on your internal monitoring methods.
And not to forget that there are is some periodic system cycling occuring on certain XI servers - I’ve never had an issue with this, but it’s worth keeping in mind if wanting a clean shut down - check your logs to see if you ever interrupt these processes when having strange behaviour.
But yeah, for a relatively clean process:
Shutting Down:
Disable everything except the CMS
Stop everything except the CMS.
Stop the CMS
Stop Tomcat.
Reboot
Starting up:
Tomcat
CMS
FRS(s)
Everything else …
…And I can’t recall if restarting then resets the disabled status …
But - last I heard, one could not automate the disabling / enabling part …
There is no need to delay by 30 minutes - you could probably get away with about 5 to 10 minutes - and can re-issue stop commands but not start commands.
But we all have our own opinions and experiences, so make sure you read up anything you can on this:
… And keep in mind that XI 3.0 only displays Tomcat and SIA in the CCM window and you have to click on Manage Servers to see the list, so maybe don’t spend too much time on AI type automation scripts etc
(BOB member since 2008-09-09)
(BOB member since 2002-06-18)
(BOB member since 2006-09-15)
(BOB member since 2005-06-27)