Thursday, 26 August 2021

Load Testing JD Edwards orchestrations

I've been tasked to help a very large client work out whether they can use JD Edwards orchestration to facilitate transactions from their ecommerce solution.  This is a very common question that I'm being asked more and more.  Customers want to connect their ecommerce solution to JDE for stock availability and also for pricing. 

The problem is that pricing is complex for B2B and a little less complex for B2C - as there is generally only a single price.  When you need to price on volume and customer - then you need to tap into the JD Edwards advanced pricing rules.

How are you going to do this effectively?

My choice is calling the JD Edwards BSFN's for pricing via orchestration.  This is a super easy process.  There are some nasty product out there that masquerade as middleware and expose BSFN's - but you can do this through AIS simply and free.  The other advantages are that it'll upgrade with your JD Edwards application stack.  I would go native AIS / orchestration every time.

I could do a blog (and might) on how easy it is to create an orchestration that allows you to call a BSFN.  It takes minutes.

My highly complex orchestration in 9.2.5.3

?xml version='1.0' encoding='UTF-8'?>
<ServiceRequest>
  <omwObjectName>SRE_2108230001F5</omwObjectName>
  <studioVersion>9.2.5.3</studioVersion>
  <name>request_CallB4201500</name>
  <shortDesc>call B4201500</shortDesc>
  <productCode>55</productCode>
  <locale>en</locale>
  <updateTime>1629696722845</updateTime>
  <description></description>
  <group>0</group>
  <appStack>false</appStack>
  <returnFromAllForms>false</returnFromAllForms>
  <bypassER>false</bypassER>
  <serviceRequestSteps>
    <serviceRequestSteps type="customServiceRequest">
      <scriptLanguage>groovy</scriptLanguage>
      <objectName>B4201500</objectName>
      <functionName>CalculateSalesPricesAndCosts</functionName>
      <dataStructureName>D4201500</dataStructureName>
      <bsfnInputs>
        <bsfnInputs>
          <name>szAdjustmentSchedule</name>
          <input>szAdjustmentSchedule</input>
          <defaultValue></defaultValue>
          <controlID>9</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnAddressNo</name>
          <input>mnAddressNo</input>
          <defaultValue></defaultValue>
          <controlID>10</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnShipToNo</name>
          <input>mnShipToNo</input>
          <defaultValue></defaultValue>
          <controlID>11</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnShortItemNo</name>
          <input>mnShortItemNo</input>
          <defaultValue></defaultValue>
          <controlID>12</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szBaseCurrencyCode</name>
          <input></input>
          <defaultValue>USD</defaultValue>
          <controlID>13</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szCustomerCurrencyCode</name>
          <input></input>
          <defaultValue>USD</defaultValue>
          <controlID>14</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>cCurrencyConversionMethod</name>
          <input></input>
          <defaultValue>Z</defaultValue>
          <controlID>18</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szBranchPlantDtl</name>
          <input>szBranchPlantDtl</input>
          <defaultValue>M30</defaultValue>
          <controlID>34</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szCompany</name>
          <input>szCompany</input>
          <defaultValue>00411</defaultValue>
          <controlID>43</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnQtyShipped</name>
          <input>mnQtyShipped</input>
          <defaultValue></defaultValue>
          <controlID>51</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnQtyOrdered</name>
          <input></input>
          <defaultValue>1</defaultValue>
          <controlID>54</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnConvFactorTransToPrim</name>
          <input></input>
          <defaultValue>1</defaultValue>
          <controlID>55</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>mnConvFactorPricingToPrim</name>
          <input></input>
          <defaultValue>1</defaultValue>
          <controlID>56</controlID>
          <type>Numeric</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szTransactionUom</name>
          <input></input>
          <defaultValue>EA</defaultValue>
          <controlID>66</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szPricingUom</name>
          <input></input>
          <defaultValue>EA</defaultValue>
          <controlID>67</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>jdPriceEffectiveDate</name>
          <input>jdPriceEffectiveDate</input>
          <defaultValue>20210823</defaultValue>
          <controlID>68</controlID>
          <type>Date - yyyyMMdd</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szCustomerPricingGroup</name>
          <input>szCustomerPricingGroup</input>
          <defaultValue></defaultValue>
          <controlID>73</controlID>
          <type>String</type>
        </bsfnInputs>
        <bsfnInputs>
          <name>szLineType</name>
          <input>szLineType</input>
          <defaultValue></defaultValue>
          <controlID>87</controlID>
          <type>String</type>
        </bsfnInputs>
      </bsfnInputs>
      <bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>20</controlID>
          <variable>mnUnitPrice</variable>
          <title>mnUnitPrice</title>
          <type>Numeric</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>21</controlID>
          <variable>mnExtendedPrice</variable>
          <title>mnExtendedPrice</title>
          <type>Numeric</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>24</controlID>
          <variable>mnListPrice</variable>
          <title>mnListPrice</title>
          <type>Numeric</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>26</controlID>
          <variable>szListPriceUOM</variable>
          <title>szListPriceUOM</title>
          <type>String</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>44</controlID>
          <variable>jdTransactionDate</variable>
          <title>jdTransactionDate</title>
          <type>Date - yyyy-MM-dd</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>66</controlID>
          <variable>szTransactionUom</variable>
          <title>szTransactionUom</title>
          <type>String</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>67</controlID>
          <variable>szPricingUom</variable>
          <title>szPricingUom</title>
          <type>String</type>
        </bsfnReturnControls>
        <bsfnReturnControls>
          <controlID>68</controlID>
          <variable>jdPriceEffectiveDate</variable>
          <title>jdPriceEffectiveDate</title>
          <type>Date - yyyy-MM-dd</type>
        </bsfnReturnControls>
      </bsfnReturnControls>
      <isAsynch>false</isAsynch>
    </serviceRequestSteps>
  </serviceRequestSteps>
</ServiceRequest>



Above is the code to the SR, as you might want to know the parameters that you need to fill out to get this working

I therefore have the ability to run this with the following active parameters:

{

  "szAdjustmentSchedule": "string",

  "mnAddressNo": "string",

  "mnShipToNo": "string",

  "mnShortItemNo": "string",

  "szBranchPlantDtl": "string",

  "szCompany": "string",

  "mnQtyShipped": "string",

  "jdPriceEffectiveDate": "string",

  "szCustomerPricingGroup": "string",

  "szLineType": "string"

}

Awesome and easy so far.

I can use curl to run this and put a bunch of &'s at the end of the queries to try and fudge some performance statistics, but that is not really going to help anyone.  I want to do a proper job of testing something that is massively parallel - as we also know that advanced pricing - even in a simple form - is going to take too long as a linear transaction.

Here is a complete script that will allow you to call the orchestration that I have created above.

There are a number of nuances in this that you need to get right and took me quite a while (lucky you).  The use of compressed as native compression at my tools level.  The use of insecure because I was using the JMeter proxy and did not load the certificate properly.  Using curlFormat for some nice timing information for your load testing.  And the use of a here file to load the JSON into a variable.

Note that the first command is not proxied and the second command is proxied.

data=$(cat <<EOF
{
  "mnAddressNumber": "4242",
  "jdDateEffective": "1/1/2021",
  "mnQtyOrdered": "12",
  "szUnitOfMeasure": "EA",
  "szCostCenter": "M30",
  "szItemNo": "220"
}
EOF
)
echo $data
#there is some native compression there, need to turn that off / account for it
#-w "@curl-format.txt" -o /dev/null
set -x
#curl -v --output - --compressed --request POST \
  #--url https://f5dv.fusion5.cloud/jderest/orchestrator/orch_getPrice2 \
  #--header 'Accept: application/json' \
  #--header 'Authorization: Basic Something==' \
  #--header 'Cache-Control: no-cache' \
  #--header 'Connection: keep-alive' \
  #--header 'Content-Type: application/json' \
  #--header 'Host: f5dv.fusion5.cloud' \
  #--header 'accept-encoding: gzip, deflate' \
  #--header 'cache-control: no-cache' \
  #--data "${data}"
countMax=5
i=0
while [ $i -lt ${countMax} ]
do
curl --proxy localhost:8888 --insecure -o /dev/null -w "@curlFormat.txt" --compressed --request POST \
  --url https://f5dv.fusion5.cloud/jderest/orchestrator/orch_getPrice2 \
  --header 'Accept: application/json' \
  --header 'Authorization: Basic Something==' \
  --header 'Cache-Control: no-cache' \
  --header 'Connection: keep-alive' \
  --header 'Content-Type: application/json' \
  --header 'Host: f5dv.fusion5.cloud' \
  --header 'accept-encoding: gzip, deflate' \
  --header 'cache-control: no-cache' \
  --data "${data}" &
  i=$(($i+1))
done

Cool - so I can now do rough testing.

I also created some additional scripts that ripped out the auth token and saved more time there.

Even if I can get 20 items on a page, and my amazing JDE can return a price in .4 of a second...  I need to wait 20x.4 or 8 seconds for the thing to complete - it is not going to fly.  We need to move on from a linear equation.

JMeter to the rescue

Unfortunately (or fortunately) OATS is nearing end of life.  unfortunately as I understand the platform and limitation well and have executed MANY load testing scenarios very successfully.  Fortunately we were given enough notice and I was able to not pay the maintenance bill this year for the companies subscription and find an alternative solution.

JMeter is totally awesome and nerdy - I'm loving it.  It does everything that I need and I can load test JD Edwards [with a number of tweaks] and also importantly orchestrations.

There is a lot of steep learning for JMeter, but it's not my first rodeo.


Above are the results for a 20 thread linear load test, finishing in a eye watering 12 seconds.  I know my internet can be slow, but a customer waiting 12 seconds to render a 20 item page might be too much.

Let's try another step.  How about caching and handle reuse (token specifically)

WOW = 2.8 seconds for 20 calls.  Look at the session initialisation overhead for what we are doing - that is crazy.  It's good to know how long the actual BSFN is taking, that is .12 of a second.  So we are dealing with .28 of overhead.  Fair enough I say.

Let's start to look into parallel processing:

What we are doing here is breaking down the AIS calling into 20 separate HTML calls or threads.  If there is one thing that the internet is pretty good at - that is threading.




We have 2.05 seconds for the 20 threads to complete.  You can see the distinctions in the start times for the above two scenarios.  Immediately above we can see all 20 threads leave JMeter at the same time [but interestingly seem to come back one at a time [seems like we are single threaded somewhere]?].  Although we have all fire off at the same time, their responses are only retrieved at about .12 seconds [magic number] after the previous one... Hmmm - might need to look closer at the parallel processing in WLS and also my server.

But, the theory is good (if not great) when I have lots of servers.  So imagine that I had 20 AIS servers, the I'm only going to wait as long as the longest request.  I'm going to use JSESSIONID in my load balancer and not need stateless load balancing in 9.2.5.3 and I'm going to maintain open connections to the AIS servers.  So, when my ecommerce solution requests a lot of pricing or a lot of availability - I'm going to reply in spades!

I'll write more about this load testing exercise and the use of JMeter for your load testing needs.  I don't see a lot of point in having additional tools. 




















Friday, 6 August 2021

AIS information generation / Form Compare / JDE environment compare

I want to talk you through a utility that I'm starting to use more and more, that is the AIS information generation node application. Wow, that is almost poetic.  

The application itself is fairly simple, but it has some really powerful uses.  I have blogged about this a long time ago - but the usage was a little unclear - so I thought that I would try agian.

I'll let you know specifically what is does, and then talk you through the use cases.

The java script iterates through all of the controls of a form (that are exposed via AIS) and compares them to a second environment...  So you can configure two JDE environments in a config file, run this code and it will tell you what controls are the same and what controls are different...   It will also tell you all of the controls on a form...  Wait... this sounds really handy.

So... Let me get this straight.  If I wanted to compare two JD Edwards environments (PY and PD) and see of all of the controls on a form were the same - at a very detailed level - I could do this?  YES!

Could I also use this to compare a complete system code - lots of forms - YES you could...

Do you mean that if I ran this after a full package deploy, this would actually generate all of the serialized objects for the JDE forms that I choose - YES - it could even do that for you.

What if I wanted to test column security - well, yes - this is going to tell you all the controls that the user can see - it's going to do that also!

Oh dear, this sounds like a very useful utility - YES it is.

I do a lot of debugging and troubleshooting (yes still, and yes I enjoy it) and this has helped me in so many situations.  Another use case is when I'm doing any AIS direct integrations (not orchestrations), I have a really quick way of getting all of the form controls and their internal AIS ID's into a CSV file.  This is really handy when I'm coding and hacking around.

How does this magic work you ask?

Firstly, you need to reach out and ask me for the file - I don't want to make it publicly available.  Sure, that is nasty - but I'm pretty friendly.

You unzip the contents of the file onto your machine:

restore the package from the zip file

install node - from here https://nodejs.org/en/download/

goto your restore dir from before with command prompt

 


C:\temp\ais-compare-cli-master

>npm i 

--This will install dependencies

Once this is complete 

C:\temp\ais-compare-cli-master>node app --help

  Usage: app [options] [command]

  Commands:

    compare <formName> [additionalFormNames...]  Compare two different forms.

    Example compare P4210_W4210A --format csv --out P4210_W4210A.csv

  Options:

    -h, --help         output usage information

    -V, --version      output the version number

    --format <format>  Specify the output format. Allowed: "csv,json" (default: json)

    --out <file>       Write to a file instead of the command line


 Config is in

C:\temp\ais-compare-cli-master\config\default.yaml


The config is really simple 

C:\temp\ais-compare-cli-master\config>type default.yaml

servers:

  - url: https://f5play.fusion5.cloud

    username: SX00001

    password: myPAssW0rd


  - url: https://f5dv.fusion5.cloud 

    username: SX00001

    password: myPAssW0rd


What the above is telling you is that it's going to compare f5dv with f5play using the credentials that you specify.

It'll provide you with output like this:
P42101_W42101A,101[],title,true,Deliver To,Deliver To
P42101_W42101A,101[],presence,true,true,true
P42101_W42101A,101[],dataType,true,9,9
P42101_W42101A,101[],editable,true,false,false
P42101_W42101A,101[],longName,true,mnDeliverTo_101,mnDeliverTo_101
P42101_W42101A,101[],visible,true,true,true
P42101_W42101A,102[],title,true,Pull Signal,Pull Signal
P42101_W42101A,102[],presence,true,true,true
P42101_W42101A,102[],dataType,true,2,2
P42101_W42101A,102[],editable,true,false,false
P42101_W42101A,102[],longName,true,sPullSignal_102,sPullSignal_102
P42101_W42101A,102[],visible,true,true,true
P42101_W42101A,103[],title,true,Ship To Attention,Ship To Attention
P42101_W42101A,103[],presence,true,true,true
P42101_W42101A,103[],dataType,true,2,2
P42101_W42101A,103[],editable,true,false,false
P42101_W42101A,103[],longName,true,sShipToAttention_103,sShipToAttention_103
P42101_W42101A,103[],visible,true,true,true
P42101_W42101A,104[],title,true,Ship To Contact Sequence ID,Ship To Contact Sequence ID
P42101_W42101A,104[],presence,true,true,true
P42101_W42101A,104[],dataType,true,9,9
P42101_W42101A,104[],editable,true,false,false
P42101_W42101A,104[],longName,true,mnShipToContactSequenceID_104,mnShipToContactSequenceID_104
P42101_W42101A,104[],visible,true,true,true
P42101_W42101A,261[],title,true,Pending Order Status,Pending Order Status
P42101_W42101A,261[],presence,true,true,true
P42101_W42101A,261[],dataType,true,2,2
P42101_W42101A,261[],editable,true,false,false
P42101_W42101A,261[],longName,true,sPendingOrderStatus_261,sPendingOrderStatus_261
P42101_W42101A,261[],visible,true,true,true


Which you can sed and awk and coerce into the value that you need.

I'm going to put this on a website soon, so that you don't need to download it - you can just run it free!

ps. I did not write any of this code, I needed a lot of help.


Extending JDE to generative AI