Quick assist on managing multiple WLS domains on a single Linux box and Server Manager

It's becoming more common to have a number of weblogic (WLS) domains on your web servers.  This is because that every component that is being released requires it's own J2EE container - and you might want a separate one for ADF and traditional web.  So, you create a couple of domains.

This is a good idea, but there are a couple of things to remember - each domain generally has an adminServer, and this can use quite a bit of memory - especially when you start creating multiple domains with multiple admin servers.

The dir structure generally looks like this:

[root@E1WEBv3 domains]# ls -l

total 20

drwxr-x---. 20 Oracle Oracle 4096 Dec 14 14:59 ADF_Apps

drwxrwxrwx. 18 Oracle root   4096 Oct 27 23:31 E1_Apps

drwxrwxrwx. 12 Oracle root   4096 Oct  7  2015 x_medrec

drwxrwxrwx. 12 Oracle root   4096 Oct  7  2015 x_medrec-spring

drwxrwxrwx. 13 Oracle root   4096 Oct  7  2015 x_wl_server

[root@E1WEBv3 domains]# pwd

/U01/Oracle/Middleware/user_projects/domains

You can see that I have ADF_Apps and E1_Apps in my home.  These can also be seen in server manager:

You cannot really tell from the front screen how many domains you have.  You should name things properly so it's obvious.  If you are introducing and naming a new domain, put that word into the text description!

This is a much better view and is hierarchal, we can see our domains and our admin servers and the various JDE components that we have running in each.

The main purpose of this post is to show you what you need running to be able to manage your  managed servers and managed instances.

If you click on the domain items, you'll see a table like:

Cluster Name	Managed Server	Machine	Listen Port	Related Managed Instances
	{not clustered}	AdminServer  (Running )	SampleMachine1	7001	None
	{not clustered}	myAccess  (Failed )	SampleMachine1	9204	myAccess_9204 (EnterpriseOne HTML Server)
	{not clustered}	myAIS_TR  (Stopped )	SampleMachine1	9224	myAIS_TR (EnterpriseOne Application Interface Services)
	{not clustered}	myAIS_MYDEMO  (Running )	SampleMachine1	9223	myAIS_MYDEMO (EnterpriseOne Application Interface Services)
	{not clustered}	MYDEMO_9203  (Running )	SampleMachine1	9303	MYDEMO_9203 (EnterpriseOne HTML Server)
	{not clustered}	myAIS_PY  (Failed )	SampleMachine1	9221	myAIS_PY (EnterpriseOne Application Interface Services)
	{not clustered}	myAIS_DEMO  (Running )	SampleMachine1	9222	myAIS_PD (EnterpriseOne Application Interface Services)
	{not clustered}	ADFServer  (Running )	SampleMachine1	9266	None
	{not clustered}	SMC_Server_EOne_ManagementConsole_Console  (Running )	SampleMachine1	8999	None
	{not clustered}	JPY920_9202  (Running )	SampleMachine1	9202	JPY920_9202 (EnterpriseOne HTML Server)

None of these links will work, don't worry.  but you can see the ports for the domain's admin server

You'll also see the root install dir for WLS in /U01/Oracle/Middleware

I'm getting the following, what do I do?

Error

1. Server 'IOT_Server' could not be started: Start task failed: Connection refused. Could not connect to NodeManager. Check that it is running at /192.168.204.238:5556.
   Failure during invoke:invoke(, , start) javax.management.MBeanException : Server 'IOT_Server' could not be started: Start task failed: Connection refused. Could not connect to NodeManager. Check that it is running at /192.168.204.238:5556.
2. 
   

I can see that this is for the ADF_Apps domain, so let's start the nodeAgent - or at least check that it's running:

[root@E1WEBv3 ADF_Apps]# pwd

/U01/Oracle/Middleware/user_projects/domains/ADF_Apps

Above is the root dir.  You can check for the process running.

[Oracle@E1WEBv3 ~]$ cd /U01/Oracle/Middleware/user_projects/domains/ADF_Apps/bin

[Oracle@E1WEBv3 bin]$ pwd

/U01/Oracle/Middleware/user_projects/domains/ADF_Apps/bin

[Oracle@E1WEBv3 bin]$ user

-bash: user: command not found

[Oracle@E1WEBv3 bin]$ whoami

Oracle

Make sure you are logged in as the user that owns the installation, generally oracle.

[Oracle@E1WEBv3 bin]$ ./setDomainEnv.sh

Ensure that you run the above.

[Oracle@E1WEBv3 bin]$ ps -ef |grep -i "weblogic.NodeManager -v" |more

Oracle    1690  1127  0 Jul24 ?        11:01:38 /usr/java/jdk1.8.0_60/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/U

01/Oracle/Middleware/wlserver/../coherence -Dbea.home=/U01/Oracle/Middleware/wlserver/.. -Dweblogic.RootDirectory=/U01/Oracle/Middleware/user_pr

ojects/domains/E1_Apps -Xverify:none -Djava.endorsed.dirs=/usr/java/jdk1.8.0_60/jre/lib/endorsed:/U01/Oracle/Middleware/wlserver/../oracle_commo

n/modules/endorsed -Djava.security.policy=/U01/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/usr/java/j

dk1.8.0_60 weblogic.NodeManager -v

Oracle    8435  8394 19 10:57 pts/2    00:00:29 /usr/java/jdk1.8.0_60/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/U

01/Oracle/Middleware/wlserver/../coherence -Dbea.home=/U01/Oracle/Middleware/wlserver/.. -Doracle.security.jps.config=/U01/Oracle/Middleware/use

r_projects/domains/ADF_Apps/config/fmwconfig/jps-config-jse.xml -Dcommon.components.home=/U01/Oracle/Middleware/oracle_common -Dopss.version=12.

1.3 -Dweblogic.RootDirectory=/U01/Oracle/Middleware/user_projects/domains/ADF_Apps -Xverify:none -Djava.endorsed.dirs=/usr/java/jdk1.8.0_60/jre/

lib/endorsed:/U01/Oracle/Middleware/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/U01/Oracle/Middleware/wlserver/server/lib

/weblogic.policy -Dweblogic.nodemanager.JavaHome=/usr/java/jdk1.8.0_60 weblogic.NodeManager -v

Oracle    8484  8307  0 10:59 pts/2    00:00:00 grep --color=auto -i weblogic.NodeManager -v

[Oracle@E1WEBv3 bin]$ 

Above will show you that there are two actually running

1 for E1_apps and 1 for ADF_Apps (see in the command for RootDirectory)

[Oracle@E1WEBv3 bin]$ ps -ef |grep -i "weblogic.NodeManager -v" |more

Oracle    1690  1127  0 Jul24 ?        11:01:39 /usr/java/jdk1.8.0_60/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/U

01/Oracle/Middleware/wlserver/../coherence -Dbea.home=/U01/Oracle/Middleware/wlserver/.. -Dweblogic.RootDirectory=/U01/Oracle/Middleware/user_pr

ojects/domains/E1_Apps -Xverify:none -Djava.endorsed.dirs=/usr/java/jdk1.8.0_60/jre/lib/endorsed:/U01/Oracle/Middleware/wlserver/../oracle_commo

n/modules/endorsed -Djava.security.policy=/U01/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/usr/java/j

dk1.8.0_60 weblogic.NodeManager -v

Oracle    8435     1 11 10:57 pts/2    00:00:30 /usr/java/jdk1.8.0_60/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/U

01/Oracle/Middleware/wlserver/../coherence -Dbea.home=/U01/Oracle/Middleware/wlserver/.. -Doracle.security.jps.config=/U01/Oracle/Middleware/use

r_projects/domains/ADF_Apps/config/fmwconfig/jps-config-jse.xml -Dcommon.components.home=/U01/Oracle/Middleware/oracle_common -Dopss.version=12.

1.3 -Dweblogic.RootDirectory=/U01/Oracle/Middleware/user_projects/domains/ADF_Apps -Xverify:none -Djava.endorsed.dirs=/usr/java/jdk1.8.0_60/jre/

lib/endorsed:/U01/Oracle/Middleware/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/U01/Oracle/Middleware/wlserver/server/lib

/weblogic.policy -Dweblogic.nodemanager.JavaHome=/usr/java/jdk1.8.0_60 weblogic.NodeManager -v

Oracle    8510  8307  0 11:01 pts/2    00:00:00 grep --color=auto -i weblogic.NodeManager -v

Interestingly when you kill the shell that started the nodemanager with nohup, the parent process moved to root (1).  That is nice, you nodeManager does not die.

[Oracle@E1WEBv3 bin]$ nohup ./startNodeManager.sh > /tmp/startNodeManager.log 2>&1 &

Above is a proper start command.  This will write logs to /tmp/startNodeManager.log

If you do this, it's terrible:

Oracle@E1WEBv3 bin]$ ./startNodeManager.sh &

[1] 8692

[Oracle@E1WEBv3 bin]$ NODEMGR_HOME is already set to /U01/Oracle/Middleware/user_projects/domains/ADF_Apps/nodemanager

CLASSPATH=/usr/java/jdk1.8.0_60/lib/tools.jar:/U01/Oracle/Middleware/wlserver/server/lib/weblogic_sp.jar:/U01/Oracle/Middleware/wlserver/server/lib/weblogic.jar:/U01/Oracle/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/U01/Oracle/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar::/U01/Oracle/Middleware/wlserver/..

Firstly all of the output goes to your shell

I thought it was going to kill nodeManager when I logged out of the session, but it remains!

Note that the domain based startNodeManager actually calls the server/bin startNodeManager.sh script!

You can kill the nodeAgent as much as you like and it will not affect any of your J2EE servers running.

If you kill the admin servers, nothing runs either (control commands from SM), but you can just run the start commands at the commandline for the current domain to get things running.

Error

1. Server 'IOT_Server' could not be started: Failed to connect to WebLogic. Make sure the domain administration server is running.
   Failure during invoke:invoke(, , start) javax.management.MBeanException : Server 'IOT_Server' could not be started: Failed to connect to WebLogic. Make sure the domain administration server is running.

Note that killing the admin Server also does not affect your existing instances running under the control of the admin server.

You cannot just start the admin server as a managedServer, see errors below

[Oracle@E1WEBv3 bin]$ pwd

/U01/Oracle/Middleware/user_projects/domains/E1_Apps/bin

[Oracle@E1WEBv3 bin]$ nohup ./startManagedWebLogic.sh AdminServer  > /tmp/startAdminServer.log 2>&1 &

[1] 9268

[Oracle@E1WEBv3 bin]$ 

[Oracle@E1WEBv3 bin]$ 

[Oracle@E1WEBv3 bin]$ 

[Oracle@E1WEBv3 bin]$ ps -ef |grep AdminServer

Oracle    9268  8768  0 11:20 pts/2    00:00:00 /bin/sh ./startManagedWebLogic.sh AdminServer

Oracle    9323  9269 99 11:20 pts/2    00:00:09 /usr/java/jdk1.8.0_60/bin/java -server -Xms512m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/U01/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/U01/Oracle/Middleware/wlserver/server/lib/cacerts -Djava.endorsed.dirs=/usr/java/jdk1.8.0_60/jre/lib/endorsed:/U01/Oracle/Middleware/wlserver/../oracle_common/modules/endorsed -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.1.3 -Digf.arisidbeans.carmlloc=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/config/fmwconfig/carml -Digf.arisidstack.home=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/U01/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_12.1.3,/U01/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_12.1.3,/U01/Oracle/Middleware/oracle_common/modules/oracle.jps_12.1.3 -Dweblogic.jdbc.remoteEnabled=false -Dcommon.components.home=/U01/Oracle/Middleware/oracle_common -Djrf.version=12.1.3 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/U01/Oracle/Middleware/user_projects/domains/E1_Apps -Doracle.server.config.dir=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/U01/Oracle/Middleware/user_projects/domains/E1_Apps/config/fmwconfig -Doracle.mds.filestore.preferred= -Dadf.version=12.1.3 -da -Dwls.home=/U01/Oracle/Middleware/wlserver/server -Dweblogic.home=/U01/Oracle/Middleware/wlserver/server -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -Dem.oracle.home=/U01/Oracle/Middleware/em -DINSTANCE_HOME=/U01/Oracle/Middleware/user_projects/domains/E1_Apps -Djava.awt.headless=true -Doracle.sysman.util.logging.mode=dual_mode -Dweblogic.management.server=http://E1WEBv3.mits.local:7001 -Djava.util.logging.manager=oracle.core.ojdl.logging.ODLLogManager -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server

Oracle    9344  8768  0 11:20 pts/2    00:00:00 grep --color=auto AdminServer

You need to actually start weblogic with startWebLogic.sh

startManagedWeblogic.sh only works when startWebLogic.sh has completed.

Note that you also want to nohup this bad boy, as it's pretty vocal.

But, now that the adminserver is running and the node manager is running, you can start things with SM.

Hopefully this gives you some skills in running commands from the command line to get your service processes up and running.