Infocus and Summit–what did I learn?

Firstly it was great to see so many familiar faces, and meet some new people.  I had a number of people come and say hi at the conference and said thanks for blogging – that may have been my favourite part of the conference, for the people that did come and say hi – I say thanks!

Infocus is a quest run customer event, specific for JD Edwards clients and partners.   This was followed by a 2 day partner conference, known as summit.  Both conferences were in Denver Colorado, who spoilt attendees with 30 degree (celsius) days with 0 humidity.

I must admit that when I went into the conference, I was worried.  I was a little worried about investing strategically into JD Edwards.  My thoughts were somewhat reiterated when I did not hear about any huge announcements.  But, then I heard Lyle at summit (a lot of clients will not hear this), and I was once again excited about what was to come.  I don’t know what specifically Lyle did say to change my opinion, but it was made up of the following:

• Acknowledgement of the formation of a specific sales team at oracle that are rewarded and driven to sell JDE – cool
• Premiere support of JD Edwards 9.2 to at least 2030 – amazing!
• That NetSuite and cloud ERP are going to be the dominant SaaS offerings, but there is a lot of whitespace between these two products
• Lyle showed passion for JD Edwards and wanting to “protect his turf” when it came to SaaS sales trying to take away the JDE opportunities
• reiteration of what JDE is and what it can do for an organisation -
• awesome integration is easy now with AIS –> orchestration
• awesome integration with Cafe1
• useability through the roof with all of the UDO’s that are supporting citizen development
• Remember that ERP’s (especially mature ones) are going to create sales orders, track stock and do all of the basic transactions that you need.  JD Edwards especially is a mature product that will do ALL of these things easily.  Now you need to look at how you are going to augment this base functionality.  You’ll do that by embracing all mega-trends – vis-a-vis this diagram:

In this diagram I’ve tried to show that JD Edwards is awesome and does what it is told.  Great security, great database, stable, single source of truth.  As clients we need to respect master data and expose JDE securely using great integration (AIS / Orchestration) to enable amazing technology to be used and consumed side by side with your ERP.  You do not need to wait for AI enabled ERP – you can have it today!

This is being put into action today,

• Fusion5 are already using AI to interpret images attached to media objects and indexing and reporting on these.  This is a SIMPLE integration that can be plugged into any public cloud ERP implementation.  And this is just the beginning.
• Fusion5 are integrating IOT devices and raising work orders when things get too hot or too humid – this is being done today.
• Any many more examples of using the strength of your core ERP – but being innovative about hooking into new technology.

But, how can everyone do this? I personally think that you need 3 capabilities to deliver continuous innovation.

Both summit and infocus reiterated the need for all clients to get to 9.2 and start to embrace configuration not code.  reduce your technical debt by retiring modifications, which will allow you to embrace continuous innovation.

Get a partner to help your (like fusion5) adopt continuous innovation, but keeping you code current.  Fusion5 have CD3 (continuous development, continuous deployment, continuous delivery) which allows continuous innovation.  It’s important to start to get economies of scale by getting a partner to help you stay code current and allow you to focus on business while we take are of the platform.

If you are looking for ways that you can reinvigorate your JD Edwards and your passion for innovation, think about having a hackathon, or perhaps an innovation bootcamp.  We run these internally and for clients and they are a great way of coming up with ideas that could make a difference to your business or perhaps change an industry.

I attended a couple of great interactive sessions on 1 click provisioning and 64bit JD Edwards – two things that are going to affect you and your JD Edwards installation positively.  I also attended a great session on containers and JDE, if you are not looking into this technology (as I have said before) you should be.  This is going to change the way we think about large systems.

9.2.3 is going to contain some great enhancements (especially in orchestration) and you should look closely for the next large oracle conference (Open World?) for this to be announced.

My summary

What it lacked in large announcements, it made up for in core messaging and core product acknowledgment.  Embrace continuous delivery and find a partner to help you.  Embrace innovation and find a partner to help you.  JD Edwards is here for the long haul, use it’s strengths and augment and extend to the cloud – pugging in AI, ML and other megatrends to improve your decision making capabilities.  Configure your ERP, don’t modify it.

Use Google to Find My Oracle Support Content

What??  This could be great.  Use google search for https://support.oracle.com

You can now use Google and other search engines to find My Oracle Support Content! Go to Google and search using all of the Google capabilities you have come to rely on. Even better, bookmark site:support.oracle.com and search Oracle content exclusively. Nice announcement in Nov last year, but does not really provide the results that I need. For example: site:support.oracle.com e1 jvm size Wow, how cool! or is it?  I'm going to need to fine tune my searching Not really what I want - not enough information but... I get a lot more information from the actual support site.  I get a lot more suggestions and quality documentation. Can I find anything in particular?  Let's search for exact knowledge documents, cannot find them... I cannot find this for example: A bit of a toy, a bit hit an miss for me.

powershell command line options starter kit

I don’t know how I’ve been able to do this until now, but I’ve avoided powershell until last week.

I might quickly become addicted to it though, after my brief introduction.

I have a requirement to upload files into sharepoint from a UBE, simple enough.

I got some amazing fusion5 people to cut me a script, and it’s great for 90% of scenarios, but I need to do a couple of mods for 100% of scenarios.

My modifications revolve around the ability to process parameters, $? $# etc – I miss ksh.

So, here is some tips for me in 3 months time when I need to do this again.

First and foremost, your Param function needs to be 1st line of the script!

Param (
     [string]$filetoupload,
     [string]$username = $( Read-Host "Input username, please" ),
     [string]$password = $( Read-Host "Input password, please" ),
     [switch]$force = $false,
     [switch]$run = $false
)

This terseness is AWESOME!

When calling this function with the following, it assigns all of the variables

PS C:\fusion5> .\uploadContentsToSP_BM_Test_singlefile.ps1 -filetoupload c:\shannon.pd -username shannon.moir@fusion5.com.au -password hello

So, if I did a:

write-host $username

in the script, it’d tell me the username.  The other nice thing is that if you do not specify the parameter and there is a $read-Host directive, it’s going to prompt only for the items that have not been entered.  So cool!

Also, you’ll see that I’ve used a mix of [switch] and [string] parameters.  string of course is a variable, but switch is cool – it’s binary

if your code, you can then use:

if ($force) {
write-host "file to upoload: " $filetoupload
}

So therefore, if the script is called with –force then it’ll execute the write-host function – easy

Param (

    [Parameter(Mandatory=$true)]
    [string]$filetoupload,
     [string]$username = $( Read-Host "Input username, please" ),
     [string]$password = $( Read-Host "Input password, please" ),
     [switch]$force = $false,
     [switch]$run = $false
)

Calling this:

PS C:\fusion5> .\uploadContentsToSP_BM_Test_singlefile.ps1 -filetouplad c:\shannon.pd -username shannon.moir@fusion5.com.au -password hello
C:\fusion5\uploadContentsToSP_BM_Test_singleFile.ps1 : A parameter cannot be found that matches parameter name
'filetouplad'.
At line:1 char:45
+ .\uploadContentsToSP_BM_Test_singlefile.ps1 -filetouplad c:\shannon.p ...
+                                             ~~~~~~~~~~~~
     + CategoryInfo          : InvalidArgument: (:) [uploadContentsT..._singleFile.ps1], ParameterBindingException
     + FullyQualifiedErrorId : NamedParameterNotFound,uploadContentsToSP_BM_Test_singleFile.ps1

So if you do not specify the mandatory parameter (see that I have a spelling mistake) we get the error that the parameter has not been specified

It could be more graceful if you manage it (less big red writing)

The directive only affects the following parameter, not all of them.

SQL Server miracles

I have not used SQL Server for ages, in fact – it’s been spotty throughout my career.

But, I was just doing some hectic Media Object massaging (F00165) – I’m converting them from type 1 (physical files) to type 5 HTML links.

My SQL caused a duplicate for a large insert, but look at what SQL tells me!!!

It tells me the bad record!!!

How many group by queries has this just saved me, so awesome.

Oracle SE2 license clarification for oracle technology foundation

I think it's really important to know your rights when it comes to database licensing and the cloud.

I'm only going to talk about the database here, that it to say - oracle SE2.

What can I do with SE2, are there going to be significant performance issues if I used SE2? Perhaps, or perhaps not...

Description of support for SE2:In September, 2015 Oracle announced the withdrawal of Oracle Database Standard Edition and the availability of a new product, Oracle Database Standard Edition 2. This announcement is relevant for all JD Edwards customers who have licensed Oracle Technology Foundation for JD Edwards EnterpriseOne because this product includes a limited use license for Oracle Database Standard Edition.In light of the new Oracle Database Standard Edition 2 product offering, Oracle Technology Foundation for JD Edwards EnterpriseOne has been updated to include a restricted use license of Oracle Database Standard Edition 2. Refer to the JD Edwards EnterpriseOne Licensing Information User Manual for a detailed description of the restricted use licenses provided in the Oracle Technology Foundation for JD Edwards EnterpriseOne product.

augmented with

Licence informationOracle Technology Foundation for JD Edwards EnterpriseOne may be licensed instead of EnterpriseOne Core Tools and Infrastructure for customers wanting the Oracle components but are not currently licensed for EnterpriseOne Core Tools and Infrastructure. The Oracle components included with Oracle Technology Foundation for JD Edwards EnterpriseOne are listed under "Entitled Products and Restricted User Licenses" below.Oracle Technology Foundation for JD Edwards EnterpriseOne would cover the JD Edwards EnterpriseOne Core Tools and Infrastructure prerequisite requirement.
Entitled Products and Restricted Use LicensesA license for Oracle Technology Foundation for JD Edwards EnterpriseOne includes the restricted-use licenses of: Oracle Database Standard Edition 2; Oracle Internet Application Server Standard Edition; Oracle WebLogic Server Standard Edition; JRockit JVM; Oracle Application Server Portal; Oracle WebCenter Services; Oracle BPEL Process Manager; Oracle Business Activity Monitoring; Oracle Application Server Single Sign-On; Oracle Access Manager Basic; Oracle Application Server Web Cache; and Business Intelligence Publisher (formerly XML Publisher).
As noted in the preceding paragraph, a license for Oracle Technology Foundation for JD Edwards EnterpriseOne includes a restricted-use license for Oracle Database Standard Edition 2. Oracle Database Standard Edition 2 may be used solely in conjunction with any and all JD Edwards EnterpriseOne programs licensed under your agreement, including third party programs licensed for use with JD Edwards EnterpriseOne programs. Oracle Database Standard Edition 2 may be installed on an unlimited number of processors. When used with Oracle Real Application Clusters, Oracle Database Standard Edition 2 may be installed on any number of RAC nodes. If you require features and functions beyond those included with the Oracle Database Standard Edition 2, or if you require use of Oracle Database beyond your JD Edwards EnterpriseOne implementation, you may purchase a non-limited use license by contracting directly with Oracle or one of its authorized distributors.
A license for Oracle Technology Foundation for JD Edwards EnterpriseOne also includes a restricted-use license for the following components of Oracle Fusion Middleware: Oracle Application Server Standard Edition or Oracle WebLogic Server Standard Edition (either of these products may be used, but both products cannot be used for the same function); Oracle Application Server Portal; Oracle WebCenter Services; Oracle BPEL Process Manager; Oracle Business Activity Monitoring; Oracle Application Server Single Sign-On; Oracle Access Manager Basic; Oracle Application Server Web Cache; and Business Intelligence Publisher. These components may be used solely in conjunction with any and all JD Edwards EnterpriseOne programs licensed under your agreement, including third party programs licensed for use with JD Edwards EnterpriseOne programs. These components may be installed on an unlimited number of processors. If you require use of these components beyond your JD Edwards EnterpriseOne implementation you may purchase a non-limited use license for any of the Oracle components by contracting directly with Oracle or one of its authorized distributors.
As noted in the preceding paragraph, a license for Oracle Technology Foundation for JD Edwards EnterpriseOne includes a restricted-use license for Oracle Business Intelligence Publisher. Oracle Business Intelligence Publisher may be used to create or modify reports that use the Oracle supplied database schema, or modifications to that schema done to support modifications to supplied Oracle JD Edwards EnterpriseOne programs. For the avoidance of doubt, examples of uses that are not permitted include, but are not limited to, the following: adding new reports that support different applications or database schemas other than JD Edwards EnterpriseOne.
Summary

Point 1: Oracle Database Standard Edition 2 may be used solely in conjunction with any and all JD Edwards EnterpriseOne programs licensed under your agreement

Q:  Does this mean things like DSI or reportsnow? I'm told YES


Point 2: Oracle Database Standard Edition 2 may be installed on an unlimited number of processors

Wow, so if you use RAC, you can have unlimited nodes for JDE with standard edition.

Point 3: Okay, you cannot used statistics and performance packs, that's a shame. But if you have a mature environment - you'll get away with it.

Point 4: data-guard. RDS is pretty much looking after this for you (and all the complexity). They are maintaining and shipping your logs to a remote replica and doing all of the network aliasing behind the scenes. This is SO easy to implement.

AWS summary:

This is where things get a little exciting. Lets look at RDS options for Se2





AWS tells us (and I believe their lawyers have done the work), that they can commission a 16 way SE2 machine with BYOL. And as clearly stated about, you have BYOL for as many cores as you can point your SE2 at... but limited to 16 because SE2 does that.

You could be addressing 16 cores and 488 GB of RAM under your standard JD Edwards agreement. For a multi AZ deployment in this situation, you'd be paying approximately 12,000 US per month, but a more modest 16 way with 122GB of ram is 3500 a month - not bad. 

With my load testing experience in JDE, 16 cores goes a LONG way - you don't need parallel either for JDE (IMHO).

Remember that this is Highly available (without RAC), so you actually have two machines ready to process your requests if AWS lose an availability zone.




Above you can see all of the server classes that you can address for SE2.

I appreciate that oracle will probably have a different view on this and I recommend that you seek specific advice before acting on the above recommendation / opinion.

Load testing JD Edwards–some load testing tips from the field?

I’ve said it before and I’ll say it again, load testing must be one of my favourite and rewarding consulting gigs.  I really like it, probably revealing a little too much about myself here.

I like constantly pursuing bottlenecks and trying to give clients confidence that the changes are going to make a difference and that they are pushing their hardware and software to the limit – getting the best bang for buck.

The age old question is – how hard do you push?

Here are a couple of things that I live by when load testing:

• choose everyday transactions – sales order entry, PO entry, address book find, launch a simple job.  Do not just hit the system with all of the slow and hard transactions, you are not going to test anything
• Do not go to the end of large grids
• know the type of transactions that you are performing, are they logic intensive or are they data intensive and understand their individual timings and more importantly their consistency.
• get your users waiting more than 3 seconds per interaction with the browser, you would be surprised when you see actually how hard your JDE is hit
• Include a mix of batch and interactive
• make sure that you count records before and after so that you know transactions are hitting the database
• make everything repeatable.  Add a PO, approve a PO, print a PO – so you do not run out of data
• dates should be variables
• Don’t start near the end of a month, you’ll have date problems and warning problems before you know it
• Use OATS – it’s great for JDE load testing
• get a specialist (like me) in to do the job.  It’s a tough gig to set this up and run it yourself.

But, back to the question, how many users?

Here are some stats that you may or may not believe.

For a site that has about 400 active JDE users a week approximately.

This is cookie based.

They’ve loaded 1.4 million pages and server over 37000 logins

Wait, that is cool, that means 37 pages per second – wait ERP analytics tells us this!

And we can see that the timeout is probably 30 minutes!

But, we can also tell the peak periods of usage.

A simple custom report can tell me pageviews per minute, at a peak of 117

and pageviews per hour, at a peak of 3405

This is really good data for load testing.

If we looked at this basically we’d want to load test 1394000 million pages and try and divide out a value for # days, 10 hours a day etc etc…

1394000/13x5 (weeks work days)/10 hours per date = 2144 an hour – but the peak is 3405 using data (1.5 times the estimate)

Taking this to another level 2144/60=35 pages a minute, which is more than 3 times less than the peak!

Load testing for this client could take many different forms, but if I’m loading more than 117 pages a minute – then we are net positive.

What am I testing you ask?  21 a second… or 1260 pages a minute!  so I’m actually load testing at 10 times their maximum load for the last 3 months.  I think that I can wind back the testing wait time and sit back and relax!

A long was to tell a short story–HAFS mounts on an ODA for PrintQueue

Everyone wants to create a disposable compute environment, it’s the right thing to do.

IF your machines / servers are stateless, then this is the first step to being elastic and more portable – think containers…  I know that I’m talking about an ODA here, but you can still put constructs into the design that allow you to be more flexible for HA and DR…  that is taking stateful data from your machines and creating a level of abstraction.

So, helping out with JDE, PrintQueue needs to go!

If you want to make your environment elastic, then eventually you need to put printqueue somewhere else and mount the location on your enterprise server.

Imagine that you did all of this, and then when the filesystem was mounting on boot, it wanted to FSK the mounted drive, and perhaps you do not have a network yet…  Guess what happens – NOTHING

Imagine if this was a ODA and the VM did not really give you great access to grub – wow – you got a problem!

Welcome to my world!

/etc/fstab looked something like:

10.255.252.180:/u01/app/sharedrepo/printqdv /mnt/printqueue  nfs nfsvers=3,rw,bg,intr,0 0

easy – leaving this automatic.  the use of hard is implied if not specified, so we do not need this in the mount options.

We created this FS on ODA_BASE as a repo

oakcli create repo printqdv -size 200G -dg DATA

Then on ODA_BASE created the NFS server using grid based srvctl (HAFS) which

srvctl stop exportfs -name printqdv
srvctl remove exportfs -name printqdv
  srvctl add exportfs -name printqdv -id havip_1 -path /u01/app/sharedrepo/printqdv -clients 10.255.252.150 –options rw,no_root_squash"
srvctl start exportfs -name printqdv

So we have a shared printqueue as a repo on ODA_BASE that all of the guests can mount using NFS.  Therefore when using WSJ, all jobs are in a single location and we can support seamless augmentation of logic hosts.

But, when automounting on guests (enterprise servers), we found though is when the machine needed to FSK, it wanted to do this to the printQueue and would not boot.

This is highly risky, we implemented the following:

10.255.252.180:/u01/app/sharedrepo/printqdv /mnt/printqueue  nfs nfsvers=3,rw,bg,intr,_netdev 0 0

adding _netdev to try and tell the boot sequence to only attempt this if there was a network, that should be much nicer.  Though I still am a little worried that this is a massive FS and I don’t want to wait for an FSK EVER!

But, could I risk this?  I want to do noauto and have a systemctl command mount the printqueue manually after boot.

go to this dir as root

/etc/systemd/system

create a new file (use a name that you want for your service)

vi jdePrintQ.service

Add the following contents

[Unit]
Description=JD Edwards PrintQueue
After=network.target

[Service]
#these need a full path
ExecStart=/usr/bin/mount /mnt/printqueue
ExecStop=/usr/bin/umount /mnt/printqueue

[Install]
WantedBy=multi-user.target

Now chmod

chmod 755 ./jdePrintQ.service

you can now use the command systemctl start jdePrintQ and she’ll start – easy

You can stop it too

systemctl stop jdePrintQ

get the status

systemctl status jdePrintQ

Aug 08 12:41:29 bear. systemd[1]: Started JD Edwards PrintQueue.
Aug 08 12:41:29 bear. systemd[1]: Starting JD Edwards PrintQueue...
Aug 08 12:41:29 bear. mount[3351]: mount.nfs: /mnt/printqueue is busy or already mounted
Aug 08 12:41:29 bear. systemd[1]: jdePrintQ.service: main process exited, code=exited, status=32/n/a
Aug 08 12:41:29 bear. systemd[1]: Unit jdePrintQ.service entered failed state.
Aug 08 12:41:29 bear. systemd[1]: jdePrintQ.service failed.

I guess that this is belt and braces, but rescuing a non booting VM on an ODA is not the most fun job in the world.

Just ensure that fstab now looks like this too:

10.255.252.180:/u01/app/sharedrepo/printqdv /mnt/printqueue  nfs nfsvers=3,rw,bg,intr,_netdev,noauto 0 0

The addition of noauto

Make sure you set your service to start on boot:

[root@bear system]# chkconfig jdePrintQ on
Note: Forwarding request to 'systemctl enable jdePrintQ.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/jdePrintQ.service to /etc/systemd/system/jdePrintQ.service.

It’s really important that you are ready to rescue a VM’s boot disk or any disk for that matter on the ODA.  Remember that if a repo runs out of space, or there is an ACFS problem (which we had lots of) that prevents ODA_BASE seeing / writing to the repo.  You are probably going to get corrupt machines – or at least the need to FSK drives.  Make sure that you have the ability to clone a boot disk and mount it to a temporary guest so that you can fix any problems with the /etc/fstab or other files that might be giving you problems on boot.  Perhaps you can stop services too.  I did NOT have a problem getting to the console of the machine.

Garbage collection and tuning the JVM for JDE

This starts to get pretty complex pretty quickly.

We’ve noticed that a JDE web install on Azure feels slow, something feels wrong.

You coad load up 20 users with OATS and watch weblogic 12.2.1.2.0 start to slow down

Actually, speed is not the problem, the machine grinds to a halt with CPU usage.

We are using a super simple JD Edwards test, just basic navigation around P01012 and WSJ.  Could not get easier – you’d think

The machine is on struggle street.

We are seeing the GC’s on the machine:

Look at the GC’s!

Too bouncy.  This is 25 users doing the same thing and logging in at different times.  There should be completely flat lines.

Wow, we are getting a full GC which is a total lockup of the JVM pretty much ALL of the time.  Taking up to 9 seconds at times.

We need to get a fix for this.

What you quickly learn about GC in JVM’s is that there are 10000 parameters and without knowing how the objects are allocated by JDE, JDBC etc it’s hard to know which levers you should be pulling.

The tip here it, what levers is java pulling by default?

Let’s find out.

create a file called hello.java and paste in the following:

public class hello {

    public static void main(String[] args) {

        // Prints "Hello, World" to the terminal window.

        System.out.println("Hello, Garbage");

    }

}

Then javac hello.java

Do this from the jdk that is running your JVM.  you might need to ps or task manage your way to finding this part of the command line.

C:\Program Files\Java\jdk1.8.0_162\bin>javac hello.java

C:\Program Files\Java\jdk1.8.0_162\bin>java -Xloggc:c:/GC/%t_gclogger.log -XX:+PrintGC -XX:-PrintGCDetails hello

Hello, Garbage

You are telling java to be verbose and print you what it’s starting with, nice.

So now you can goto c:\GC and find out what (windows 2016 in my case) is defaulted to run with java

Java HotSpot(TM) 64-Bit Server VM (25.162-b12) for windows-amd64 JRE (1.8.0_162-b12), built on Dec 19 2017 20:00:03 by "java_re" with MS VC++ 10.0 (VS2010)

Memory: 4k page, physical 16776756k(7382136k free), swap 20970920k(5873592k free)

CommandLine flags: -XX:InitialHeapSize=268428096 -XX:MaxHeapSize=4294849536 -XX:+PrintGC -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC 

So this is the naked truth for what the windoze JVM thinks that it should start with.

you can find all of the options listed here http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Using this iknowledge for JDE WLS is easy, at least you are going to know what the default params are.  The other nice this is that if you enable verbose GC logging, you’ll be able to check the JVM history nice and easy

Java HotSpot(TM) 64-Bit Server VM (25.92-b14) for windows-amd64 JRE (1.8.0_92-b14), built on Mar 31 2016 21:03:04 by "java_re" with MS VC++ 10.0 (VS2010)

Memory: 4k page, physical 16776756k(11436092k free), swap 20970920k(14323404k free)

CommandLine flags: -XX:+AggressiveOpts -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:+PrintGC -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TraceClassLoading -XX:+TraceClassResolution -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC

9.772: [GC (Metadata GC Threshold)  356941K->37709K(2010112K), 0.0497603 secs]

9.822: [Full GC (Metadata GC Threshold)  37709K->36962K(2010112K), 0.1157298 secs]

12.303: [GC (Metadata GC Threshold)  104455K->51591K(2010112K), 0.0146142 secs]

12.318: [Full GC (Metadata GC Threshold)  51591K->32982K(2010112K), 0.0572060 secs]

12.490: [GC (System.gc())  56999K->33214K(2010112K), 0.0015203 secs]

12.492: [Full GC (System.gc())  33214K->23886K(2010112K), 0.1942664 secs]

24.818: [GC (Metadata GC Threshold)  518419K->68032K(2010112K), 0.0430348 secs]

24.862: [Full GC (Metadata GC Threshold)  68032K->66171K(2010112K), 0.2901091 secs]

77.256: [GC (Allocation Failure)  590971K->231300K(2010112K), 0.1385489 secs]

85.555: [GC (Allocation Failure)  756100K->287925K(1813504K), 0.1240276 secs]

88.482: [GC (Allocation Failure)  616117K->264082K(1911808K), 0.0430451 secs]

91.890: [GC (Allocation Failure)  592274K->262682K(1938944K), 0.0406183 secs]

95.013: [GC (Allocation Failure)  633882K->264546K(1927680K), 0.0386120 secs]

97.406: [GC (Allocation Failure)  635746K->261578K(1955840K), 0.0398701 secs]

104.722: [GC (Metadata GC Threshold)  564677K->278930K(1948672K), 0.0511449 secs]

104.773: [Full GC (Metadata GC Threshold)  278930K->126063K(1948672K), 0.2996287 secs]

271.829: [GC (Allocation Failure)  535151K->179467K(1964032K), 0.0409892 secs]

1712.011: [GC (Allocation Failure)  606987K->143383K(1958912K), 0.0272208 secs]

2124.193: [GC (Allocation Failure)  570903K->188328K(1969152K), 0.0613237 secs]

2130.849: [GC (Allocation Failure)  624552K->211110K(1962496K), 0.0687304 secs]

2140.347: [GC (Allocation Failure)  647334K->246768K(1937920K), 0.1185877 secs]

2145.322: [GC (Allocation Failure)  655856K->299962K(1952256K), 0.1423171 secs]

2152.052: [GC (Allocation Failure)  709050K->347231K(1825280K), 0.1426725 secs]

2153.875: [GC (Metadata GC Threshold)  460486K->366841K(1845248K), 0.1227049 secs]

2153.997: [Full GC (Metadata GC Threshold)  366841K->275362K(1845248K), 1.1694144 secs]

2158.558: [GC (Allocation Failure)  557474K->297258K(1872384K), 0.0199824 secs]

2161.004: [GC (Allocation Failure)  547626K->304734K(1872896K), 0.0478987 secs]

2165.078: [GC (Allocation Failure)  555102K->336275K(1885696K), 0.0419868 secs]

2167.680: [GC (Allocation Failure)  602515K->398773K(1875968K), 0.1051175 secs]

2171.856: [GC (Allocation Failure)  665013K->435986K(1853440K), 0.1449460 secs]

2175.736: [GC (Allocation Failure)  677138K->464323K(1828352K), 0.1165887 secs]

2180.276: [GC (Allocation Failure)  705475K->492642K(1864192K), 0.1381040 secs]

2183.488: [GC (Allocation Failure)  725602K->511599K(1864192K), 0.1451506 secs]

2187.965: [GC (Allocation Failure)  744559K->535714K(1864192K), 0.1557600 secs]

2191.799: [GC (Allocation Failure)  768674K->549443K(1864192K), 0.1302633 secs]

2195.937: [GC (Allocation Failure)  782403K->576326K(1864192K), 0.1069931 secs]

2198.253: [GC (Allocation Failure)  809286K->594966K(1864192K), 0.0552789 secs]

2201.184: [GC (Allocation Failure)  827926K->614134K(1864192K), 0.0333154 secs]

I still have this to deal with for 20 users:

Wish me luck!