Friday, 24 May 2019

Cool SQL queries for rows and indexes

always forget some of the basic SQL Server catalogs, so this is going to help me next time:

Query to get the current index definitions from the database.  Remember that you can cross reference this with some of this knowledge:  https://shannonscncjdeblog.blogspot.com/2017/06/jde-slow-missing-indexes-find-it-fast.html (namely F98712 & F98713)

Remember that all index definitions do not need to be in specs (really – sorry if you disagree), performance based indexes can sit happily in the database and do not need to be put through the SDLC.  Let's be honest, the creation process is horrendous in JDE!

The query below when cross referenced with F98712 and F98713 could tell you missing or incorrect indexes with a single statement.  Though you could also run R9698711 and R9698713 to help you out.


select i.[name] as index_name,
    substring(column_names, 1, len(column_names)-1) as [columns],
    case when i.[type] = 1 then 'Clustered index'
        when i.[type] = 2 then 'Nonclustered unique index'
        when i.[type] = 3 then 'XML index'
        when i.[type] = 4 then 'Spatial index'
        when i.[type] = 5 then 'Clustered columnstore index'
        when i.[type] = 6 then 'Nonclustered columnstore index'
        when i.[type] = 7 then 'Nonclustered hash index'
        end as index_type,
    case when i.is_unique = 1 then 'Unique'
        else 'Not unique' end as [unique],
    schema_name(t.schema_id) + '.' + t.[name] as table_view,
    case when t.[type] = 'U' then 'Table'
        when t.[type] = 'V' then 'View'
        end as [object_type]
from sys.objects t
    inner join sys.indexes i
        on t.object_id = i.object_id
    cross apply (select col.[name] + ', '
                    from sys.index_columns ic
                        inner join sys.columns col
                            on ic.object_id = col.object_id
                            and ic.column_id = col.column_id
                    where ic.object_id = t.object_id
                        and ic.index_id = i.index_id
                            order by col.column_id
                            for xml path ('') ) D (column_names)
where t.is_ms_shipped <> 1
and index_id > 0
order by i.[name]

Provices something like
index_name columns    index_type unique     table_view     object_type
F0000194_PK     SYEDUS, SYEDBT, SYEDTN, SYEDLN   Clustered index     Unique     NULL Table
F0002_PK   NNSY Clustered index Unique     NULL Table
F00021_PK  NLKCO, NLDCT, NLCTRY, NLFY Clustered index Unique     NULL Table
F00022_PK  UKOBNM     Clustered index Unique     NULL Table
F0004_2    DTSY, DTRT, DTUSEQ    Nonclustered unique index     Unique     NULL Table
F0004_PK   DTSY, DTRT Clustered index Unique     NULL Table
F0004D_PK  DTSY, DTRT, DTLNGP    Clustered index Unique     NULL     Table
F0005_2    DRSY, DRRT, DRKY, DRDL02   Nonclustered unique index     Unique     NULL Table
F0005_3    DRSY, DRRT, DRDL01    Nonclustered unique index  Not unique     NULL Table
F0005_PK   DRSY, DRRT, DRKY Clustered index Unique     NULL     Table



SELECT
       sOBJ.name AS [TableName]
      , SUM(sPTN.Rows) AS [RowCount]
FROM
      [RRCDBSP01\JDE920].[JDE_PRODUCTION].[sys].[objects] AS sOBJ
      INNER JOIN [RRCDBSP01\JDE920].[JDE_PRODUCTION].[sys].[partitions] AS sPTN
            ON sOBJ.object_id = sPTN.object_id
WHERE
      sOBJ.type = 'U'
      AND sOBJ.is_ms_shipped = 0x0
      AND index_id < 2 -- 0:Heap, 1:Clustered
GROUP BY
      sOBJ.schema_id
      , sOBJ.name
ORDER BY [TableName]


Tables that are not in JDE, but are in the database:

       SELECT
       sOBJ.name AS [TableName]
      , SUM(sPTN.Rows) AS [RowCount]
FROM
      JDE_PRODUCTION].[sys].[objects] AS sOBJ
      INNER JOIN [JDE_PRODUCTION].[sys].[partitions] AS sPTN
            ON sOBJ.object_id = sPTN.object_id
WHERE
      sOBJ.type = 'U'
      AND sOBJ.is_ms_shipped = 0x0
      AND index_id < 2 -- 0:Heap, 1:Clustered
       AND not exists (select 1 from [JDE920].[OL920].[F9860] ol WHERE ol.siobnm = sOBJ.name and ol.sifuno = 'TBLE')
GROUP BY
      sOBJ.schema_id
      , sOBJ.name


Tables that are in:

       SELECT
       sOBJ.name AS [TableName]
      , SUM(sPTN.Rows) AS [RowCount]
FROM
      JDE_PRODUCTION].[sys].[objects] AS sOBJ
      INNER JOIN [JDE_PRODUCTION].[sys].[partitions] AS sPTN
            ON sOBJ.object_id = sPTN.object_id
WHERE
      sOBJ.type = 'U'
      AND sOBJ.is_ms_shipped = 0x0
      AND index_id < 2 -- 0:Heap, 1:Clustered
       AND exists (select 1 from [JDE920].[OL920].[F9860] ol WHERE ol.siobnm = sOBJ.name and ol.sifuno = 'TBLE')
GROUP BY
      sOBJ.schema_id
      , sOBJ.name



Stolen from:

curl an orchestration with environment variables in bash

It's so nice when you are old and get to feel productive.  This has been a phenomenon when doing lots of JDE on AWS.

AWS have an amazing cli interface in which you can script everything.

Orchestration in JD Edwards allows you to create an API out of anything you want to do in JDE…

Finally, cURL is an amazing utility in linux.

Oh, and have I mentioned, I have a black belt in ksh and awk ?

So, you add all of this together, and I'm getting productive.

We are doing some funky things in JDE to bring it out of the dark ages and into a CI/CD pipeline on AWS.  This does take time, but we there.

For one scenario we have created some highly available batch servers that can be replaced when there is a new package deploy, that's right – they are ephemeral.

So, to enable this (not to give too much away), there has been a lot of work by SvdS and myself – mainly SvdS – if you are in the game – you know who this is.

We have a fixed IP that we can attach to the server when it becomes the new server, but to make this smooth – we need to disable all of the queues on the original.  Make sure no batch jobs are running and then move them over.

So an orchestration to hold all the queues, that makes a lot of sense.  An orchestration to release all of the queues and an orchestration to run some batch jobs – just to make sure that things are cool (and reconnect the web server to the new machine).

Let's do it:

holdAllQueues is an orchestration that I wrote (I'll include it for download) that you can enter a hostname and it'll hold the queues, wow – simple.  But have you done this manually, painful!

If you want to call this in a ksh, then:

#!/usr/bin/ksh
set -x
if [ $# -ne 1 ]
  then
    echo "USAGE $0 SERVERNAME"
    exit 1
else
  serverName=$1
fi
echo ${serverName}
data='{"MYLOGICHOST":"'"${serverName}"'"}'
echo ${data}
#exit
curl -v --request POST \
  --resolve au.jde.something.com:443:10.116.23.100 \
  --url https://au.jde.something.com/jderest/orchestrator/orch_fusion5_holdAllQueues \
  --header 'Accept: */*' \
  --header 'Authorization: Basic WU9VQVJFOkFCVVRURkFDRQ==' \
  --header 'Cache-Control: no-cache' \
  --header 'Connection: keep-alive' \
  --header 'Content-Type: application/json' \
  --header 'Host: au.jde.something.com' \
  --header 'accept-encoding: gzip, deflate' \
  --header 'cache-control: no-cache' \
  --data "${data}"

The reason that this might help is the passing an environment variable (or parameter) into cURL can be curly (like that?).

So you'll see that this simple example conversion parameter 1 from the script input as the servername in the JSON body to the orchestration that I wrote.

The other thing I have in this example is avoiding certificate problems with the –resolve option, this is great because for many reasons, these addresses are no internally resolvable. (it's me not you).

So, this shows you how to use shell variables and create a really handy function to hold all of the batch queues – perhaps before a package deployment.

This allows me to move IP's to the new machine and then start loading it up with more batch jobs without an outage – YAY!

Friday, 17 May 2019

How I saved 1000s of dollars in one afternoon


I implemented a fairly basic schedule engine in AWS that works on specific tags to control when the machines are up or down.  The cost of our demo kit was starting to add up, so I looked into the console (which is great) and saw that the main costs were EC2 and RDS.  To fix this I followed a two prong approach:


  1. mandatory tagging
  2. schedule for shutdown


Kinda chicken or egg, because the solution I chose defined the tagging.


I basically followed the guides below:


https://docs.aws.amazon.com/solutions/latest/instance-scheduler/welcome.html

https://s3.amazonaws.com/solutions-reference/aws-instance-scheduler/latest/instance-scheduler.pdf


At Fusion5, we defined a single active Tag for schedules and a single schedule stack:


If you use the tag name ScheduleUptime on an EC2 or RDS instance, then this instance will be on a schedule – it's that simple.The fusion5 stack ID is f5sched

If you set your instance to have tag ScheduleUptime

 

The schedules rely on 2 building blocks, periods and schedules.

A period defines the hour day of starting and stopping, kinda like cron.  The schedule uses the period, but has more controls around the behaviour of the EC2 instance when there is an issue (and importantly uses TZ).

 

For Fusion5, we started with the following schedules.


  • NZ-office-hours
  • AU-office-hours
  • AU-7till7
  • NZ-7till7

 

I think that you can work out what they actually mean!

 

1.3      Periods:

 

command to create a period:

 

root@localhost ~]# scheduler-cli create-period --begintime 07:00 --description "7 till 7 baby" --endtime 19:00 --weekdays 0-4 --name 7till7Mon2Fri -s f5sched

{

   "Period": {

      "Description": "7 till 7 baby",

      "Weekdays": [

         "0-4"

      ],

      "Begintime": "07:00",

      "Endtime": "19:00",

      "Type": "period",

      "Name": "7till7Mon2Fri"

   }

}

 

 

1.4      Schedules:

 

 

command to create a schedule:


[root@localhost ~]# scheduler-cli create-schedule -s f5sched --description "Shannon TEsting 9 to 5" --timezone "Australia/Melbourne" --name AU-office-hours --periods "office-hours"

{

   "Schedule": {

      "RetainRunning": false,

      "Enforced": false,

      "Description": "Shannon TEsting 9 to 5",

      "StopNewInstances": true,

      "Periods": [

         "office-hours"

      ],

      "Timezone": "Australia/Melbourne",

      "Type": "schedule",

      "Name": "AU-office-hours"

   }

}

 

1.5      Viewing schedules and periods

1.5.1      Command line

Install the command line

wget https://s3.amazonaws.com/solutions-reference/aws-instance-scheduler/latest/scheduler-cli.zip

 

python setup.py install

 

then you can use this, but of course you need aws command line first (https://shannonscncjdeblog.blogspot.com/2017/06/move-tb-from-nz-to-aus-via-s3-bucket-of.html)

 

 

DynamoDB


There are two tables in DynamoDB

 

The top one is the only one to worry about (they are not really tables either)

 

Basically everything is saved as JSON document:

{

  "begintime": {

    "S": "07:00"

  },

  "description": {

    "S": "7 till 7 baby"

  },

  "endtime": {

    "S": "19:00"

  },

  "name": {

    "S": "7till7Mon2Fri"

  },

  "type": {

    "S": "period"

  },

  "weekdays": {

    "SS": [

      "0-4"

    ]

  }

}

 

 


 

 

Friday, 3 May 2019

More JDE scheduler frustrations

I have my 300 jobs, the password is right, but not I have to “reset schedule” on all of them.  when I open any of the jobs, there is no "next schedule / 5 rows apprearing in the grid".  A screen shot is below.

There is probably a PO or something much more simple than what I'm about to explain, but hang on!


See how when I look at the above, they are no listing of future scheduled jobs (or past).

So I need to open the job [as above], use the form exit of "reset schedule", then press okay and save for every one!!  This is going to take ages.  In IT, it's simple, I don't do repetitive tasks...  I automate them.

You cannot run the schedule app (P91300) from IV or fast path, otherwise I’d be tucking into a orchestration based solution – that’d be nice and easy.

So, I totally need to go old school on this, lucky I have some tricks for old school.  Lucky I am old school.

Firstly, I looked at the code of W91300B to see if I could attach that form exit to the main screen, then use repeat for grid.  The code looked bad, and there were a bunch of if statements and about 40 parameters to the BSFN that does the work to reset the schedule, I was not feeling that brave.

Secondly, I stopped and started and changed scheduler servers and reset the master, this did not help.  I did some https://support.oracle.com searches that revealed nothing.

So I bought out – captain vb script!

set objShell = wscript.createobject("WScript.Shell")

Do until success = True
  Success = objshell.AppActivate("Schedule Jobs - [Job Schedule--Canberra, Melbourne and Sydney]")
  wscript.sleep 1000
Loop

wscript.sleep 100
wscript.echo "Reset these schedules"
m=0
do until m = 300
      wscript.sleep 500
      objshell.sendkeys "%m"
      wscript.sleep 500
      objshell.sendkeys "t"
      wscript.sleep 500
      objshell.sendkeys "{Enter}"
      wscript.sleep 500
      objshell.sendkeys "%o"
      m=m+1
loop


And ran this at the command line

cmd> wscript resetSchedule.vbs

This then smashed through my 300ish jobs without a schedule and created them, nice.    There are probably much better ways of doing this.

Remember that you need to match the screen name exactly, once you select one of the records.  You don't want it working on the wrong screen.  My title once I select a row from the main screen is "Schedule Jobs - [Job Schedule--Canberra, Melbourne and Sydney" as seen below: 


Success = objshell.AppActivate("Schedule Jobs - [Job Schedule--Canberra, Melbourne and Sydney]")

What you need to do is match the number of records you are going to select with the main counter in the loop.  Highlight all of the records from P91300 then hit select.

Now, run the script at the command line.

JDE scheduler BULK password change with LDAP enabled

You have heaps of scheduled jobs and you need to change the password that is saved in the scheduler.  Easy, because there is a great function in admin password change, but you cannot use it when LDAP is enabled.  

This is a strange quandary too, as you need to save the LDAP password in the scheduler table.



What do you do when you need to change the password for 300 scheduled jobs?  No problems, I got you!

Just update a single record (as below, I changed the record for report 'R55HR002', version 'RBVS0010'.


Then run the following SQL:

update sy920.f91300 set sjschpwd = (
select sjschpwd from sy920.f91300 where sjschrptnm = 'R55HR002' and sjschver = 'RBVS0010')
where sjschuser like 'ZSCH%' and  sjschrptnm != 'R55HR002';
commit;

BOOM! 290 records updated. And schedules working.

Thanks to JDE for not using the job name or version name in the encryption.


Thursday, 2 May 2019

cheats guide to JDE and weblogic (in general) timeouts

On linux to change the web timeouts for JDE


Find where agent runs:


[oracle@jdepp1 SCFHA]$ ps -ef |grep java | grep -i scfagent



jde920    2554     1  0 Apr12 ?        00:08:38 /jde_home_32/SCFHA/jdk/jre/bin/java -classpath /jde_home_32/SCFHA /lib/scfagent.jar com.jdedwards.mgmt.agent.Launcher

oracle   31900     1  0 Apr26 ?        00:03:03 /jde_home/SCFHA/jdk/jre/bin/java -classpath /jde_home/SCFHA/lib/scfagent.jar com.jdedwards.mgmt.agent.Launcher



Easy, now find the relevant web.xml file:



Find /jde_home_32/SCFHA -name web.xml -print



[oracle@jdepp1 SCFHA]$ find /jde_home/SCFHA -name web.xml -print

/jde_home/SCFHA/targets/WEB_PP_JDEPP1/owl_deployment/webclient.ear/app/webclient.war/WEB-INF/web.xml



Back it up, then edit (vi) it

In the file, Find /web-app


...
        oracle/portal/provider/global/log/logLevel

        7

        java.lang.Integer

     
**  Modify here **


"targets/WEB_PP_JDEPP1/owl_deployment/webclient.ear/app/webclient.war/WEB-INF/web.xml" line 976 of 976 --100%-- col 3


Add this before



180



The results look like this

...
     

        oracle/portal/provider/global/log/logLevel

        7

        java.lang.Integer

     



180






The number 10,800,000 is for 3 hours

Do same for web.xml under user_projects

Ps -ef |grep



[oracle@jdepp1 SCFHA]$ ps -ef |grep java |grep WEB_PP_JDEPP1 | awk -F"-DINSTANCE_HOME=" '{print $2}' | awk '{print $1}'

/Oracle_Home/user_projects/domains/e1apps


Then same find

Nerdy use of flea to find the next web.xml


[oracle@jdepp1 SCFHA]$ find `ps -ef |grep java |grep WEB_PP_JDEPP1 | awk -F"-DINSTANCE_HOME=" '{print $2}' | awk '{print $1}'` -name web.xml |grep stage |grep WEB_PP



/Oracle_Home/user_projects/domains/e1apps/servers/WEB_PP_JDEPP1/stage/WEB_PP_JDEPP1/app/webclient.war/WEB-INF/web.xml



Vi that and fix it too

Finally JAS.INI

User session cache timeout

3,600,000 for 1 hour

Minutes to ms -> https://www.calculateme.com/time/minutes/to-milliseconds/

In my case 10,800,000

Bounce and done!

Extending JDE to generative AI