Orchestration caching frustration - Cross Reference

 Perhaps this is just me and my dodgy use case.

I'm using jitterbit and JD Edwards orchestration to synchronise the address book between MSFT CEC and JD Edwards.  Nice and simple.  

I have a JB developer doing his thing and I said that I would donate my time to do the orchestration development - coz I'm a nice guy.

All started well and is not ending well.

A super simple start...  well...  There are no fields in the addressbook that are long enough to take the 40ish character CEC unique ID. I thought that an easy thing to use would be cross reference.  This is designed for this purpose... right?

So I start checking for my new AIS cross reference, if it does not exist - create it with the AN8 record

Easy hey?  Then I I do a similar check for edit and I delete both on a delete!  The perfect crime.

And finally delete

It's all fine until you string it all together.

Add:

Works fine.  Cross reference says it aborted, but it's configured to continue.  This is dodgy.  We can see that the AN8 has been created and now there is a value in the cross reference.

and AN8

I can now run all of my edits, all work well.  Address changes and cross reference picks up the add without an issue.

I run my delete:

I can see that the cross reference has been deleted and the AN8 is gone

See above that 06 is now missing.  BUT - if I try and create it again...  Everything craps out, because the value is cached in AIS

I think personally that is a bug.  If we are using highly dynamic reference data [which is reasonable], then I believe that the cache should stay current with the list.

You need to clear the JAS cache for this to work - crazy I feel.  

Anyway, this took me quite a while to find and makes me think I cannot use cross reference.

Yay - we went live... But how can I measure success?

Go-lives are exciting, but I'm going to cover off go-lives for upgrades - which can be more exciting in some respects.  Why, well everyone knows how to use the system, so it's generally going to be hit pretty hard pretty early.  You users will also have their specific environment that they want to test, and when you mess up their UDO's and grid formats - you are going to hear about it.

I'm focused on, as a management team, how do you know that you upgrade is a success.

I have a number of measures that I try to use to indicate whether a go-live has been successful:

1. Number of issues is important, but often they do not get raised.
2. Interactive system usage
1. how many users, applications, modules and jobs are being run and how that compares with normal usage.
2. Performance; I want to know what is working well and what needs to be tuned
3. Time on page - by user and by application
4. Batch and queue performance and saturation - to ensure that my jobs are not queuing too much

Here is some example reports of these metrics which allow you to define success, measure and improve.

I find the above report is handy, as this is able to show me a comparison of the last week to the last 50 days...  So I get instant trend information.  You can see that I can drill down to ANY report to quickly see what it's runtime was over the last 50 days.  The other nice thing is that the table shows you exactly what is slower and by how much.  I can spend 5 minutes and tell a client all of the jobs that are performing better and which ones they need to focus on.

The next place I look for a comparison is here:

Above I'm able to see my busy jobs and how many times they are running, how much data they are processing or how many seconds they are running for.  I know my go-live date, so that makes things easy too.

I can do this on a queue by queue basis if needed too.

We can also quickly see that users are using all parts of the application that they should be, comparing the previous period - this is user count, pages loaded and time on page.

From a webpage performance point of view, we can report over the actual user experience.  This is going to tell us the page download time, the server response time and the page load time - all critical signals for a healthy system.

All of these reports can be sent to you on a weekly or monthly cadence to ensure that you know exactly how JDE is performing for you.

Load Testing JD Edwards orchestrations

I've been tasked to help a very large client work out whether they can use JD Edwards orchestration to facilitate transactions from their ecommerce solution.  This is a very common question that I'm being asked more and more.  Customers want to connect their ecommerce solution to JDE for stock availability and also for pricing. 

The problem is that pricing is complex for B2B and a little less complex for B2C - as there is generally only a single price.  When you need to price on volume and customer - then you need to tap into the JD Edwards advanced pricing rules.

How are you going to do this effectively?

My choice is calling the JD Edwards BSFN's for pricing via orchestration.  This is a super easy process.  There are some nasty product out there that masquerade as middleware and expose BSFN's - but you can do this through AIS simply and free.  The other advantages are that it'll upgrade with your JD Edwards application stack.  I would go native AIS / orchestration every time.

I could do a blog (and might) on how easy it is to create an orchestration that allows you to call a BSFN.  It takes minutes.

My highly complex orchestration in 9.2.5.3

?xml version='1.0' encoding='UTF-8'?>

<ServiceRequest>

  <omwObjectName>SRE_2108230001F5</omwObjectName>

  <studioVersion>9.2.5.3</studioVersion>

  <name>request_CallB4201500</name>

  <shortDesc>call B4201500</shortDesc>

  <productCode>55</productCode>

  <locale>en</locale>

  <updateTime>1629696722845</updateTime>

  <description></description>

  <group>0</group>

  <appStack>false</appStack>

  <returnFromAllForms>false</returnFromAllForms>

  <bypassER>false</bypassER>

  <serviceRequestSteps>

    <serviceRequestSteps type="customServiceRequest">

      <scriptLanguage>groovy</scriptLanguage>

      <objectName>B4201500</objectName>

      <functionName>CalculateSalesPricesAndCosts</functionName>

      <dataStructureName>D4201500</dataStructureName>

      <bsfnInputs>

        <bsfnInputs>

          <name>szAdjustmentSchedule</name>

          <input>szAdjustmentSchedule</input>

          <defaultValue></defaultValue>

          <controlID>9</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnAddressNo</name>

          <input>mnAddressNo</input>

          <defaultValue></defaultValue>

          <controlID>10</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnShipToNo</name>

          <input>mnShipToNo</input>

          <defaultValue></defaultValue>

          <controlID>11</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnShortItemNo</name>

          <input>mnShortItemNo</input>

          <defaultValue></defaultValue>

          <controlID>12</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szBaseCurrencyCode</name>

          <input></input>

          <defaultValue>USD</defaultValue>

          <controlID>13</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szCustomerCurrencyCode</name>

          <input></input>

          <defaultValue>USD</defaultValue>

          <controlID>14</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>cCurrencyConversionMethod</name>

          <input></input>

          <defaultValue>Z</defaultValue>

          <controlID>18</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szBranchPlantDtl</name>

          <input>szBranchPlantDtl</input>

          <defaultValue>M30</defaultValue>

          <controlID>34</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szCompany</name>

          <input>szCompany</input>

          <defaultValue>00411</defaultValue>

          <controlID>43</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnQtyShipped</name>

          <input>mnQtyShipped</input>

          <defaultValue></defaultValue>

          <controlID>51</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnQtyOrdered</name>

          <input></input>

          <defaultValue>1</defaultValue>

          <controlID>54</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnConvFactorTransToPrim</name>

          <input></input>

          <defaultValue>1</defaultValue>

          <controlID>55</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>mnConvFactorPricingToPrim</name>

          <input></input>

          <defaultValue>1</defaultValue>

          <controlID>56</controlID>

          <type>Numeric</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szTransactionUom</name>

          <input></input>

          <defaultValue>EA</defaultValue>

          <controlID>66</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szPricingUom</name>

          <input></input>

          <defaultValue>EA</defaultValue>

          <controlID>67</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>jdPriceEffectiveDate</name>

          <input>jdPriceEffectiveDate</input>

          <defaultValue>20210823</defaultValue>

          <controlID>68</controlID>

          <type>Date - yyyyMMdd</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szCustomerPricingGroup</name>

          <input>szCustomerPricingGroup</input>

          <defaultValue></defaultValue>

          <controlID>73</controlID>

          <type>String</type>

        </bsfnInputs>

        <bsfnInputs>

          <name>szLineType</name>

          <input>szLineType</input>

          <defaultValue></defaultValue>

          <controlID>87</controlID>

          <type>String</type>

        </bsfnInputs>

      </bsfnInputs>

      <bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>20</controlID>

          <variable>mnUnitPrice</variable>

          <title>mnUnitPrice</title>

          <type>Numeric</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>21</controlID>

          <variable>mnExtendedPrice</variable>

          <title>mnExtendedPrice</title>

          <type>Numeric</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>24</controlID>

          <variable>mnListPrice</variable>

          <title>mnListPrice</title>

          <type>Numeric</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>26</controlID>

          <variable>szListPriceUOM</variable>

          <title>szListPriceUOM</title>

          <type>String</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>44</controlID>

          <variable>jdTransactionDate</variable>

          <title>jdTransactionDate</title>

          <type>Date - yyyy-MM-dd</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>66</controlID>

          <variable>szTransactionUom</variable>

          <title>szTransactionUom</title>

          <type>String</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>67</controlID>

          <variable>szPricingUom</variable>

          <title>szPricingUom</title>

          <type>String</type>

        </bsfnReturnControls>

        <bsfnReturnControls>

          <controlID>68</controlID>

          <variable>jdPriceEffectiveDate</variable>

          <title>jdPriceEffectiveDate</title>

          <type>Date - yyyy-MM-dd</type>

        </bsfnReturnControls>

      </bsfnReturnControls>

      <isAsynch>false</isAsynch>

    </serviceRequestSteps>

  </serviceRequestSteps>

</ServiceRequest>

Above is the code to the SR, as you might want to know the parameters that you need to fill out to get this working

I therefore have the ability to run this with the following active parameters:

{

  "szAdjustmentSchedule": "string",

  "mnAddressNo": "string",

  "mnShipToNo": "string",

  "mnShortItemNo": "string",

  "szBranchPlantDtl": "string",

  "szCompany": "string",

  "mnQtyShipped": "string",

  "jdPriceEffectiveDate": "string",

  "szCustomerPricingGroup": "string",

  "szLineType": "string"

}

Awesome and easy so far.

I can use curl to run this and put a bunch of &'s at the end of the queries to try and fudge some performance statistics, but that is not really going to help anyone.  I want to do a proper job of testing something that is massively parallel - as we also know that advanced pricing - even in a simple form - is going to take too long as a linear transaction.

Here is a complete script that will allow you to call the orchestration that I have created above.

There are a number of nuances in this that you need to get right and took me quite a while (lucky you).  The use of compressed as native compression at my tools level.  The use of insecure because I was using the JMeter proxy and did not load the certificate properly.  Using curlFormat for some nice timing information for your load testing.  And the use of a here file to load the JSON into a variable.

Note that the first command is not proxied and the second command is proxied.

data=$(cat <<EOF
{
  "mnAddressNumber": "4242",
  "jdDateEffective": "1/1/2021",
  "mnQtyOrdered": "12",
  "szUnitOfMeasure": "EA",
  "szCostCenter": "M30",
  "szItemNo": "220"
}
EOF
)
echo $data
#there is some native compression there, need to turn that off / account for it
#-w "@curl-format.txt" -o /dev/null
set -x
#curl -v --output - --compressed --request POST \
  #--url https://f5dv.fusion5.cloud/jderest/orchestrator/orch_getPrice2 \
  #--header 'Accept: application/json' \
  #--header 'Authorization: Basic Something==' \
  #--header 'Cache-Control: no-cache' \
  #--header 'Connection: keep-alive' \
  #--header 'Content-Type: application/json' \
  #--header 'Host: f5dv.fusion5.cloud' \
  #--header 'accept-encoding: gzip, deflate' \
  #--header 'cache-control: no-cache' \
  #--data "${data}"
countMax=5
i=0
while [ $i -lt ${countMax} ]
do
curl --proxy localhost:8888 --insecure -o /dev/null -w "@curlFormat.txt" --compressed --request POST \
  --url https://f5dv.fusion5.cloud/jderest/orchestrator/orch_getPrice2 \
  --header 'Accept: application/json' \
  --header 'Authorization: Basic Something==' \
  --header 'Cache-Control: no-cache' \
  --header 'Connection: keep-alive' \
  --header 'Content-Type: application/json' \
  --header 'Host: f5dv.fusion5.cloud' \
  --header 'accept-encoding: gzip, deflate' \
  --header 'cache-control: no-cache' \
  --data "${data}" &
  i=$(($i+1))
done

Cool - so I can now do rough testing.

I also created some additional scripts that ripped out the auth token and saved more time there.

Even if I can get 20 items on a page, and my amazing JDE can return a price in .4 of a second...  I need to wait 20x.4 or 8 seconds for the thing to complete - it is not going to fly.  We need to move on from a linear equation.

JMeter to the rescue

Unfortunately (or fortunately) OATS is nearing end of life.  unfortunately as I understand the platform and limitation well and have executed MANY load testing scenarios very successfully.  Fortunately we were given enough notice and I was able to not pay the maintenance bill this year for the companies subscription and find an alternative solution.

JMeter is totally awesome and nerdy - I'm loving it.  It does everything that I need and I can load test JD Edwards [with a number of tweaks] and also importantly orchestrations.

There is a lot of steep learning for JMeter, but it's not my first rodeo.

Above are the results for a 20 thread linear load test, finishing in a eye watering 12 seconds.  I know my internet can be slow, but a customer waiting 12 seconds to render a 20 item page might be too much.

Let's try another step.  How about caching and handle reuse (token specifically)

WOW = 2.8 seconds for 20 calls.  Look at the session initialisation overhead for what we are doing - that is crazy.  It's good to know how long the actual BSFN is taking, that is .12 of a second.  So we are dealing with .28 of overhead.  Fair enough I say.

Let's start to look into parallel processing:

What we are doing here is breaking down the AIS calling into 20 separate HTML calls or threads.  If there is one thing that the internet is pretty good at - that is threading.

We have 2.05 seconds for the 20 threads to complete.  You can see the distinctions in the start times for the above two scenarios.  Immediately above we can see all 20 threads leave JMeter at the same time [but interestingly seem to come back one at a time [seems like we are single threaded somewhere]?].  Although we have all fire off at the same time, their responses are only retrieved at about .12 seconds [magic number] after the previous one... Hmmm - might need to look closer at the parallel processing in WLS and also my server.

But, the theory is good (if not great) when I have lots of servers.  So imagine that I had 20 AIS servers, the I'm only going to wait as long as the longest request.  I'm going to use JSESSIONID in my load balancer and not need stateless load balancing in 9.2.5.3 and I'm going to maintain open connections to the AIS servers.  So, when my ecommerce solution requests a lot of pricing or a lot of availability - I'm going to reply in spades!

I'll write more about this load testing exercise and the use of JMeter for your load testing needs.  I don't see a lot of point in having additional tools. 

AIS information generation / Form Compare / JDE environment compare

I want to talk you through a utility that I'm starting to use more and more, that is the AIS information generation node application. Wow, that is almost poetic.  

The application itself is fairly simple, but it has some really powerful uses.  I have blogged about this a long time ago - but the usage was a little unclear - so I thought that I would try agian.

I'll let you know specifically what is does, and then talk you through the use cases.

The java script iterates through all of the controls of a form (that are exposed via AIS) and compares them to a second environment...  So you can configure two JDE environments in a config file, run this code and it will tell you what controls are the same and what controls are different...   It will also tell you all of the controls on a form...  Wait... this sounds really handy.

So... Let me get this straight.  If I wanted to compare two JD Edwards environments (PY and PD) and see of all of the controls on a form were the same - at a very detailed level - I could do this?  YES!

Could I also use this to compare a complete system code - lots of forms - YES you could...

Do you mean that if I ran this after a full package deploy, this would actually generate all of the serialized objects for the JDE forms that I choose - YES - it could even do that for you.

What if I wanted to test column security - well, yes - this is going to tell you all the controls that the user can see - it's going to do that also!

Oh dear, this sounds like a very useful utility - YES it is.

I do a lot of debugging and troubleshooting (yes still, and yes I enjoy it) and this has helped me in so many situations.  Another use case is when I'm doing any AIS direct integrations (not orchestrations), I have a really quick way of getting all of the form controls and their internal AIS ID's into a CSV file.  This is really handy when I'm coding and hacking around.

How does this magic work you ask?

Firstly, you need to reach out and ask me for the file - I don't want to make it publicly available.  Sure, that is nasty - but I'm pretty friendly.

You unzip the contents of the file onto your machine:

restore the package from the zip file

install node - from here https://nodejs.org/en/download/

goto your restore dir from before with command prompt

 

C:\temp\ais-compare-cli-master

>npm i 

--This will install dependencies

Once this is complete 

C:\temp\ais-compare-cli-master>node app --help

  Usage: app [options] [command]

  Commands:

    compare <formName> [additionalFormNames...]  Compare two different forms.

    Example compare P4210_W4210A --format csv --out P4210_W4210A.csv

  Options:

    -h, --help         output usage information

    -V, --version      output the version number

    --format <format>  Specify the output format. Allowed: "csv,json" (default: json)

    --out <file>       Write to a file instead of the command line

 Config is in

C:\temp\ais-compare-cli-master\config\default.yaml

The config is really simple 

C:\temp\ais-compare-cli-master\config>type default.yaml

servers:

  - url: https://f5play.fusion5.cloud

    username: SX00001

    password: myPAssW0rd

  - url: https://f5dv.fusion5.cloud 

    username: SX00001

    password: myPAssW0rd

What the above is telling you is that it's going to compare f5dv with f5play using the credentials that you specify.

It'll provide you with output like this:

P42101_W42101A,101[],title,true,Deliver To,Deliver To

P42101_W42101A,101[],presence,true,true,true

P42101_W42101A,101[],dataType,true,9,9

P42101_W42101A,101[],editable,true,false,false

P42101_W42101A,101[],longName,true,mnDeliverTo_101,mnDeliverTo_101

P42101_W42101A,101[],visible,true,true,true

P42101_W42101A,102[],title,true,Pull Signal,Pull Signal

P42101_W42101A,102[],presence,true,true,true

P42101_W42101A,102[],dataType,true,2,2

P42101_W42101A,102[],editable,true,false,false

P42101_W42101A,102[],longName,true,sPullSignal_102,sPullSignal_102

P42101_W42101A,102[],visible,true,true,true

P42101_W42101A,103[],title,true,Ship To Attention,Ship To Attention

P42101_W42101A,103[],presence,true,true,true

P42101_W42101A,103[],dataType,true,2,2

P42101_W42101A,103[],editable,true,false,false

P42101_W42101A,103[],longName,true,sShipToAttention_103,sShipToAttention_103

P42101_W42101A,103[],visible,true,true,true

P42101_W42101A,104[],title,true,Ship To Contact Sequence ID,Ship To Contact Sequence ID

P42101_W42101A,104[],presence,true,true,true

P42101_W42101A,104[],dataType,true,9,9

P42101_W42101A,104[],editable,true,false,false

P42101_W42101A,104[],longName,true,mnShipToContactSequenceID_104,mnShipToContactSequenceID_104

P42101_W42101A,104[],visible,true,true,true

P42101_W42101A,261[],title,true,Pending Order Status,Pending Order Status

P42101_W42101A,261[],presence,true,true,true

P42101_W42101A,261[],dataType,true,2,2

P42101_W42101A,261[],editable,true,false,false

P42101_W42101A,261[],longName,true,sPendingOrderStatus_261,sPendingOrderStatus_261

P42101_W42101A,261[],visible,true,true,true

Which you can sed and awk and coerce into the value that you need.

I'm going to put this on a website soon, so that you don't need to download it - you can just run it free!

ps. I did not write any of this code, I needed a lot of help.

Another Day another upgrade - msgType=916

 Working with another great JDE upgrade.

Part of what I like to do is look through the logs on a nightly basis and see if anything strange occurred.  I have a couple of stories from the trenches on this.

Firstly, some logs 

057         Wed Jun  2 09:35:11.080962         ipcmisc.c348

IPC3700004 - sendMessage timed out, idx=246, errno=4.

3057       Wed Jun  2 09:35:11.081207         extmsgq.c1194

Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),Message aborted for msgType=924.

3057       Wed Jun  2 09:35:11.081261         extmsgq.c1271

NET2000002:  Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),  Message aborted, type=924

 

Wed Jun  2 10:01:38.787813         extmsgq.c1271

NET2000002:  Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),  Message aborted, type=924

2981       Wed Jun  2 13:08:03.750311         ipcmisc.c348

IPC3700004 - sendMessage timed out, idx=246, errno=4.

2981       Wed Jun  2 13:08:03.750469         extmsgq.c1194

Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),Message aborted for msgType=924.

2981       Wed Jun  2 13:08:03.750520         extmsgq.c1271

 

2614       Wed Jun  2 02:17:35.323798         ipcmisc.c348

IPC3700004 - sendMessage timed out, idx=246, errno=4.

2614       Wed Jun  2 02:17:35.323955         extmsgq.c1194

Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),Message aborted for msgType=916.

2614       Wed Jun  2 02:17:35.324004         extmsgq.c1271

NET2000002:  Could not pass message to jdenet_k kernel queue=<Krnl18952ReqQ> (lastIPCError=<eIPCTimedOut>),  Message aborted, type=916

2614       Wed Jun  2 02:17:35.324045         saw_mon.c1100

jdeSawImpGetKernelUserListV3:Error:JDENET_SendMsg:le_net_error 12:<JDEVPAPP02>,<6017>,18952,5

The important thing about logs like this is the msgType, because this tells you the type of message being dropped.  Let's be honest, if it's serverManager related stats, we do not care!

opening msgtpye.h in netmessaging dir from system/include on the server, shows me the message that this is occurring for.  But you also need to look at the kernel.

You do not need to be a programmer to work out the moderate description from the comments in the code.

typedef enum tagJDENETCallObjectMsgType {
   JDEMSGCALLOBJ = CALLOBJECT_RANGE_START,
   JDEMSGINITREMOTEUSER,     /* 902 */
   JDEMSGFREEREMOTEUSER,     /* 903 */
   JDEMSGINITREMOTEENV,      /* 904 */
   JDEMSGFREEREMOTEENV,      /* 905 */
   JDEMSGGETSZUSERLIST,      /* 906 */
   JDEMSGCUSTOMRUN,          /* 907 */
   JDEMSGCUSTOMLOAD,         /* 908 */
   JDEMSGCUSTOMUNLOAD,       /* 909 */
   JDEMSGNOTUSED,            /* 910 */ /* Previously JDEMSGTAMSEND */
   JDEMSGREFRESHCACHE,       /* 911 */
   JDEMSGREFRESHPOCACHE,     /* 912 */
   JDEMSGCOMMITREMOTEUSER,   /* 913 */
   JDEMSGROLLBACKREMOTEUSER, /* 914 */
   JDEMSGPREPAREREMOTEUSER,  /* 915 */
   JDEMSGGETUSERLIST,        /* 916 */
   JDEMSGCLEARCACHE,         /* 917 */
   JDEMSGSETUSEROPTION,      /* 918 */
   JDEMSGGETMSDTCWHEREABOUTS,/* 919 */
   JDEMSGXMLCALLOBJ,         /* 920 */
   JDEMSGBULKTREELOADFROMJDECACHE, /* 921 */
   JDEMSGGETZEVENTTEMPLATE, /* 922 */
   JDEMSGCONFIGURATIONCHANGE, /* 923 */

So we can see that 916 is JDEMSGGETUSERLIST.  We can assume that this is server manager asking the kernel who is connected - so that it can report back to eager CNC people.  Let's be honest, I do nto care if this is having problems.

Remember that the jdenet kernel is the magician that is passing messages internally via IPC and externally via TCP/IP.  You can see where they are going to though, just need to read.

At the end of the day, this is a good tip for working back from your logs to deciphering the IPC errors on your server.  Generally if you are hitting OS limits, the JDE logs are going to get pretty verbose.  So in the instance above, we do not need to increase any of the OS limits because of the IPC errors

remember on Linux as the jde user, you can run ipcs and see all of the shared memory and IPCs that JDE is using in kernel world.