This post started life as a description of oakcli commands and where to run them. Pretty simple… It ended up with me really trying to understand the virtualised ODA architecture and where it might be letting me down.
All my research came down to a couple of salient facts:
· Start and stop VM’s from oakcli on ODA_BASE
· Start and stop ODA_BASE from Dom0
First, you should do with through EM12C if you can, but I had a pile of problems with that too.
So, you need to use oakcli – nice… go old school on it
oakcli is a command that seems to do anything for your VM’s on your ODA.
A good reference is here: https://docs.oracle.com/cd/E22693_01/doc.12/e55580/oakcli.htm
Before you start hacking away, it’s nice to understand what you are dealing with when you have a virtualized ODA.
What are all these machines?
Each “Dom X” is a VM, where Dom0 is the management domain and Dom 1 is the ODA_BASE domain (a Dom U with special privs). All of these Dom’s sit above the hypervisor.
Each node is a physical machine
So, you’ve probably called your host – ronin… After the singer!
Dom0 gives you control over the hypervisor (ronin0_net1)
Dom1 is ODA_BASE (which is a glorified Dom U), it has direct access to the storage shelf.
Dom U is user domains, for each VM essentially
Then you have ronin0 and ronin1 which are hostnames for each of the ODA_BASE machines on your physical “ronin” ODA. You then have hypervisors (essentially another VM) known as ronin0_net1 and ronin1_net1.
Oakcli from Dom0
So you need to login as root to ronin0_net1 to use the oakcli command interface:
if you are in Dom0, you have a limited number of objects for oakcli, but you still have the same command:
[root@ronin1-net1 ~]# oakcli show oda-base Usage: oakcli  : [show | configure | deploy | start | stop | restart | create | delete]: [firstnet | additionalnet | ib2fiber | oda_base | env_hw | vlan | network] : [force] : [-changeNetCard | -publicNet] available with configure network
[root@buttons-net1 ~]# oakcli show oda_base
ODA base domain
ODA base CPU cores :8
ODA base domain memory :384
ODA base template :/OVS/oda_base_18.104.22.168.tar.gz
ODA base vlans :['net1', 'net2']
ODA base current status :Running
Oakcli from ODA_BASE
You get a few more options from ODA_BASE, you can actually control your VMs. Now, this is strange, as the Dom0 machine actually has them running within it… Very strange to me.
So, when you are on ODA_BASE, you can run vm commands:
[root@ronin0 tmp]# oakcli show vm
NAME NODENUM MEMORY VCPU STATE REPOSITORY
bear 1 8192M 2 ONLINE jderepotst
chinapeak 0 8192M 2 ONLINE jderepotst
georgetown 0 8192M 2 ONLINE jderepotst
homewood 0 8192M 2 ONLINE jderepotst
jdessosvr 0 4096M 2 ONLINE jderepotst
mountshasta 1 8192M 2 ONLINE jderepotst
Note also that the database and grid run in ODA_BASE too:
oracle 57905 1 0 Apr15 ? 00:55:14 ora_ckpt_JDETEST1
oracle 57909 1 0 Apr15 ? 00:18:57 ora_smon_JDETEST1
oracle 57913 1 0 Apr15 ? 00:01:06 ora_reco_JDETEST1
oracle 57917 1 0 Apr15 ? 00:35:03 ora_mmon_JDETEST1
oracle 57921 1 0 Apr15 ? 06:05:21 ora_mmnl_JDETEST1
oracle 57925 1 0 Apr15 ? 00:01:33 ora_d000_JDETEST1
oracle 57929 1 0 Apr15 ? 00:01:27 ora_s000_JDETEST1
oracle 58081 1 0 Apr15 ? 06:44:49 ora_lck0_JDETEST1
oracle 58089 1 0 Apr15 ? 00:06:48 ora_rsmn_JDETEST1
oracle 58287 1 0 Apr15 ? 00:02:24 ora_arc0_JDETEST1
oracle 58291 1 0 Apr15 ? 00:03:50 ora_arc1_JDETEST1
See above there are a heap of processes for the JDETEST1 RAC instance of the database
When logged into Dom0, you actually see the ODA_BASE VM and the other VM’s if you PS for a command.
Note also that your VM’s don’t run in ODA_BASE, they run in Node 0
[root@ronin1-net1 ~]# ps -ef | grep domain
root 6590 10362 0 21:38 pts/2 00:00:00 grep domain
root 15970 15027 0 Mar04 ? 04:58:17 /usr/lib64/xen/bin/qemu-dm -d 1 -domain-name oakDom1 -videoram 4 -vnc 0.0.0.0:0 -vncunused -vcpus 16 -vcpu_avail 0xffff -boot c -serial pty -acpi -net none -M xenfv
root 31107 15027 2 16:26 ? 00:08:16 /usr/lib/xen/bin/qemu-dm -d 7 -domain-name bear -videoram 4 -vnc 0.0.0.0:10 -vncunused -vcpus 2 -vcpu_avail 0x3L -boot c -acpi -usbdevice tablet -net none -M xenfv
root 49452 15027 2 Apr07 ? 3-17:47:50 /usr/lib/xen/bin/qemu-dm -d 5 -domain-name mountshasta -videoram 4 -vnc 0.0.0.0:10 -vncunused -vcpus 2 -vcpu_avail 0x3L -boot c -acpi -usbdevice tablet -net none -M xenfv
Note that an interesting point is that the first vm above is actually ODA_BASE or oakDom1. So the _net1 is the bare metal host.
If you run this command from the dom0 on the other node,
[root@ronin0-net1 ~]# ps aux |grep domain
root 5674 2.5 0.0 64744 2944 ? SLl Aug01 876:22 /usr/lib/xen/bin/qemu-dm -d 16508 -domain-name chinapeak -videoram 4 -vnc 0.0.0.0:10 -vncunused -vcpus 2 -vcpu_avail 0x3L -boot c -acpi -usbdevice tablet -net none -M xenfv
root 15964 0.2 0.0 56552 996 ? SLl Mar04 573:03 /usr/lib64/xen/bin/qemu-dm -d 1 -domain-name oakDom1 -videoram 4 -vnc 0.0.0.0:0 -vncunused -vcpus 16 -vcpu_avail 0xffff -boot c -serial pty -acpi -net none -M xenfv
root 39236 0.0 0.0 61208 736 pts/5 S+ 22:38 0:00 grep domain
root 60399 4.0 0.1 64744 5100 ? SLl Apr07 8210:40 /usr/lib/xen/bin/qemu-dm -d 12 -domain-name georgetown -videoram 4 -vnc 0.0.0.0:10 -vncunused -vcpus 2 -vcpu_avail 0x3L -boot dc -acpi -usbdevice tablet -net none -M xenfv
root 61443 2.6 0.0 64744 956 ? SLl Apr07 5231:00 /usr/lib/xen/bin/qemu-dm -d 13 -domain-name homewood -videoram 4 -vnc 0.0.0.0:10 -vncunused -vcpus 2 -vcpu_avail 0x3L -boot c -acpi -usbdevice tablet -net none -M xenfv
You see another ODA_BASE (of course, because we are running RAC), but you see a different list of Dom0’s
This is a great diagram of how disk works for OVM guest OS’s on the ODA:
What a mess! The hypervisor exposes disk to ODA_BASE a little more natively, but VM disks are exposed as a NFS share to an ACFS volume. The above is stolen from http://blog.dbi-services.com/oda-vms-possibilities-a-performances/ . So you have VMs running on the hypervisor which are stored on an NFS share itself exported by a VM……not the most efficient architecture.