Thursday, 29 May 2014

Load testing webinar - RECORD | RUN | LEARN | IMPROVE

This is a bit of a promo, I’m hosting a load testing webinar next week.  If you’d like to attend, please use the following URL to register your interest:  http://myriad-it.com/load-testing-record-run-learn-improve/ 

WEBINAR DETAILS:
DATE:
Wednesday 4th June
TIME: 10 - 11am

PRESENTER:
Shannon Moir (Director, Oracle Consulting)
AGENDA:
Understanding Load Testing as a Service

> Architecture & software requirements

Load Testing demonstration

> How to record & run a JD Edwards testing script
Making the system faster
> Interpret Load Testing results
> Identify areas for improvement such as bottlenecks
> Benchmarking
> Monitoring & reporting

Tuesday, 27 May 2014

unix log file analysis

If you want to get gaps in your log file that are > 14000 (for example) miliseconds than the previous line, then this awk script is for you.  This does some gnarly calcs over the JD Edwards time stamps and tells you which two lines have a gap greater than X.  I know that you can do this with “performance workbench”, but that’s not as much fun.

cat R42565* | awk -F. '{if(substr($2,0,6)-prev>14000&&prev>0&&substr($2,0,6)*1==substr($2,0,6))printf("GAP(%d)\nPREV%s\nCURR%s\n",substr($2,0,6)-prev,prevline,$0);prev=substr($2,0,6);prevline=$0}' > Gaps.txt

GAP(15992)
PREVMay 27 09:21:58.293008 - 2388/4500 MAIN_THREAD                              Try to send the message to security server AUULTDWNWAD013
CURRMay 27 09:21:58.309000 - 2388/4500 MAIN_THREAD                              Try to free sec list
GAP(31911)
PREVMay 27 09:21:58.324089 - 2388/4500 WRK:Starting jdeCallObject               METADATA - Using Environment [JPY910] for user [00160531] in sendCnvMsgToMetadataServer()
CURRMay 27 09:21:58.356000 - 2388/4500 WRK:Starting jdeCallObject               Spec Data returned from Metadata server. size = 508.
GAP(15997)
PREVMay 27 09:21:58.371003 - 2388/4500 WRK:Starting jdeCallObject               METADATA - Using Environment [JPY910] for user [00160531] in sendCnvMsgToMetadataServer()
CURRMay 27 09:21:58.387000 - 2388/4500 WRK:Starting jdeCallObject               Spec Data returned from Metadata server. size = 732.

So the output is something like the above.  My threshold was 14000 ms.  Note that this is a common issue with my clients having a consistent wait of over 14000ms on traffic.  It’s pure network too – on windows 2008R2 – as some of the traffic is DB and some is local (to it’s own metadata kernel!)

If anyone has a fix for this 15000ms wait, I’d love to know!

Wednesday, 21 May 2014

Media Object File attachment is too slow

Have you ever gone to MO attachments and waited and waited for the FILE button to appear, like the below:

image

If you look at the frame source, you will work it out…  14MB in this clients case!

 

 

                <option value=14>Human Resources</option>

                                <option value=16>Manufacturing</option>

                                <option value=18>Misc Images</option>

                                <option value=20>Misc. Animations</option>

                                <option value=22>Miscellaneous</option>

                                </select>                             <input type=hidden name=queue value='0'>

                                <input type=hidden name=queuename value='BusinessFunctionDoc'>

                                </td></tr>

                                <tr><td>&nbsp;</td><td><LABEL FOR="IMAGE">File</LABEL></td>

                                <td>

                                <select name=IMAGE ID=IMAGE style="width:35em" onChange='if(document.MOImageList.preview.checked)show(options[selectedIndex].value);' >

                                <option value='\\SOP-JDEP\E910\PrintQueue\F0009_ALTERTABLE_D1130822_T123735.log' selected>F0009_ALTERTABLE_D1130822_T123735.log</option>

                                <option value='\\SOP-JDEP\E910\PrintQueue\F0009_ALTERTABLE_D1131009_T124312.log'>F0009_ALTERTABLE_D1131009_T124312.log</option>

                                <option value='\\SOP-JDEP\E910\PrintQueue\F0009_ALTERTABLE_D1131015_T164939.log'>F0009_ALTERTABLE_D1131015_T164939.log</option>

                                <option value='\\SOP-JDEP\E910\PrintQueue\F0009_ALTERTABLE_D1140128_T174906.log'>F0009_ALTERTABLE_D1140128_T174906.log</option>

 

The code above shows that it’s actually listing EVERY file that’s in the folder pointed to by the P98MOQUE entry for BusinessFunctionDoc.

In my case when I go to P98MOQUE and see where this points, it’s the deployment server printqueue folder (huh?).  But, this has lots of files.

image

So, Add AardvardAttachments as a file type attachment location and specify an empty dir, then your page is going to be faster than any other!

Friday, 16 May 2014

orphan full package - delete the record

Do you go into your "Package Assembly" screen and see some packages listed that are not actively deployed?  This occurs when you rename machines or re-purpose them (dev to prod etc).

All you need to do is carefully look through the data in F96511 and delete the records that are not needed.  You'll just need to delete the invalid record where or package name is in the SKSERSHP column.

Simple reference article today - OMW tables and descriptions


OMW Log Header                                               F98210  
OMW Log Detail                                               F98211  
OMW Project Master                                           F98220  
OMW Project Descriptions                                     F98220D  
OMW Project Users                                           F98221  
OMW Project Objects                                         F98222  
OMW Allowed Actions                                         F98223  
OMW Status Activity Rules                                   F98224  
OMW Transfer Activity Rules                                 F98225  
Source Management Access Configuration                       F98226  
OMW Source Control Path Configuration                       F98227  
OMW System Settings                                         F98230  
OMW Transfer Activity Rule Director - Detail                 F98231D  
OMW Transfer Activity Rule Director - Header                 F98231H  
LCM User Generated Contents Manager                         F982400  
LCM UGC Transfer Status Mapping                             F982401  
LCM/UGC Transfer and Package Header                         F982402  
LCM/UGC Transfer and Package Detail                         F982403  


Thursday, 15 May 2014

JD Edwards upgrade–make it faster

This is going to be a bit of a tease post, apologies.  Want to reduce your TC times from 50 hours to 2.5?

I’ve been recently engaged by a client to see what can be done with their upgrade window.  Everyone that has more than 1TB of data will start to push the upgrade window boundaries for their go-live weekend.  What I’m saying is, out of the box upgrading E1 with standard methodologies will take 12 –> 24+ hours with 1TB+ of data in normal circumstances.  Unless you have exadata, then it might take 15 minutes.

Remember that the data upgrade part of the upgrade process is generally only a small piece.  You still need to reconcile data in XE, then upgrade, then reconcile and then test again – this can take a lot of time.  You need to always make the technical piece more efficient to take the pressure off the other teams.

As you are aware there are 3 types of TC’s in the upgrade process (I’m talking specifically table conversions – data only update).

  • *MERGE
  • *ALTER
  • UBE

Each type of TC does what the name implies.  *MERGE is generally merging the contents of two tables into 1.  For example, F0911 and F0911T into F0911TCTEMP and then a rename back to F0911 and a heap of indexes along the way.

*ALTER is all about adding columns with default values.  ALTER TABLE X ADD COLUMN Y…  Sometimes when you are dealing with a large table (50000000 rows), this ALTER can take longer than a generation and a load (funny isn’t it!).

And UBE, my favourite.  UBE’s are generally the worst performers of all.  they are UBE, because generally they cannot do the table upgrade in any form os set based logic, so we go back to simple RAT (Row at a Time) processing.  RAT for 50 million rows!!!  No thanks.

So how can we make this process faster – or what sort of improvements can be gained with some tweaking?

I’ve been able to take the F4072 and F4074 upgrade times from 50 hours to 1.5.  That’s not all, F42119 from about 20 hours to 2.5.  This is using the same hardware and the same software (well, JD Edwards and a database), but re-engineering the scripts to use the power of the database, not just archaic slow RAT logic that is supported with the JD Edwards middleware.

I’ll start posting a little more detail about these tweaks.  So it might be that the upgrade that you thought impossible is very possible.

You must also be open to “pre-upgrading” your data.  Of course if you are a large rich multinational, you might consider goldenGate to be your silverBullet – which it could easily be.  But you could also kill this bird with a simple implementation of oracle streams or some simple triggers to pre-upgrade all of your tables and then only apply the logic to changes into the 9.1 database.  Of course for goldenGate or streams or triggers, you still need to reverse engineer all of the TC’s and apply that logic.

Friday, 9 May 2014

Convert World tables to OneWorld

Introduction:

This post shows a rapid method of converting world tables into OneWorld tables. This is not for the faint of heart. If you only like doing things that are in official oracle manuals, please close this browser window and forget you visited this page. Starting printing off the hundreds of table definitions and 750 custom DD items. Then sign into E1, create a project and being typing. See you in a month!  Oh and when you make a typo on 1 DD item or order the columns wrong, good luck finding it…

If you are more adventurous (in a very nerdy kinda way), keep reading (even if it is between your fingers as your hands cover your eyes).

Note that this procedure also allows for the copy to an oracle database, this is not mandatory for it to work.

This could be one of the largest cowboys you’ve ever read about.

Tables in OneWorld need the following to be valid:

· F9860 record

· F9861 record

· DD items / columns

· F98710 record – table header

· F98711 record(s) – table detail (row for each column)

· F98712 record – key header

· F98713 record – key details

So we are going to build all of the above with SQL from the AS/400 table definitions and catalogs.

 

Step 1 Create tables:

Copy data from AS/400 to oracle. http://shannonscncjdeblog.blogspot.com.au/2014/05/using-heterogeneous-database-services.html 

Use procedure outlined in “Creating ODBC connectivity from Oracle using heterogeneous database services”.

This is done with a simple set of “CREATE TABLE AS select * FROM” statements using the database link that you establish from the AS/400.

I was able to move about 30GB of data in 4 hours, this was the contents of 110 custom tables with about 165000000 rows.

 

F98710:

I have created my custom tables in CRPDTA.

--110 table header entries

insert into dv910.f98710 select 0, trim(table_name), 0,0,0,'E910','','','',''

from all_tables t1

where t1.owner = 'CRPDTA'

and (t1.table_name like 'F55%'

or t1.table_name like 'F56%'

or t1.table_name like 'F57%'

or t1.table_name like 'F58%'

or t1.table_name like 'F59%')

;

commit;

 

F98711:

insert into dv910.f98711

select 0,table_name,0,trim(substr(column_name,3,10)),column_id,trim(column_name),0,'E910',' ',' ',' ',' '

from all_tab_columns where owner = 'CRPDTA'

and (table_name like 'F55%'

or table_name like 'F56%'

or table_name like 'F57%'

or table_name like 'F58%'

or table_name like 'F59%');

commit;

F98712:

Note that I was creating a read only environment, so I did not care for the key or the format of the key. E1 just needs a PK. I chose the first column of the table.

insert into dv910.f98712 select 0, trim(table_name), 1,'Primary Index','1','1',0,'E910','','','',''

from all_tables t1

where t1.owner = 'CRPDTA'

and (t1.table_name like 'F55%'

or t1.table_name like 'F56%'

or t1.table_name like 'F57%'

or t1.table_name like 'F58%'

or t1.table_name like 'F59%');

commit

;

F98713:

insert into dv910.f98713

select 0,trim(table_name),1,'Primary Key', 0, trim(substr(column_name,3,10)),1,'A',0,'E910',' ', ' ', ' ', ' '

from all_tab_columns where owner = 'CRPDTA'

and column_id = 1

and (table_name like 'F55%'

or table_name like 'F56%'

or table_name like 'F57%'

or table_name like 'F58%'

or table_name like 'F59%');

commit;

 

F9200:

insert into dd910.f9200

select distinct trim(substr(column_name,3,10)), '55', '55', 'D', 'NDENT', 'SQL', 114100, 'MYPC', 6000

from all_tab_columns where owner = 'CRPDTA'

and (table_name like 'F55%'

or table_name like 'F56%'

or table_name like 'F57%'

or table_name like 'F58%'

or table_name like 'F59%')

and not exists

(select 1 FROM dd910.f9200

where trim(frdtai) = trim(substr(column_name,3,10))

)

;

commit;

 

F9203:

insert into dd910.f9203

select distinct trim(substr(column_name,3,10)), ' ', ' ', 'DEfault Description', 'DEFAULT DESCRIPTION', ' '

from all_tab_columns where owner = 'CRPDTA'

and (table_name like 'F55%'

or table_name like 'F56%'

or table_name like 'F57%'

or table_name like 'F58%'

or table_name like 'F59%')

and not exists

(select 1 FROM dd910.f9203

where trim(frdtai) = trim(substr(column_name,3,10))

)

;

F9210:

This was a little harder, I could have used decode statements in my SQL, but I was getting tired. I used the inserted spreadsheet (with formulas) to build the insert statements.

Note that the DD could have been done much better if the world -> E1 DD merge had been run, this would give proper descriptions. Once again, I can update the descriptions later if I care via the database link and knowing where to find them. My issue was that the CCSID conversion was not run over the world DD, therefore all of the columns in the DD files where junk when I tried to select them.

--741 custom DD items (not in 910 DD)

select distinct substr(column_name,3,10), data_type, data_length from all_tab_columns where owner = 'CRPDTA'

and (table_name like 'F55%'

or table_name like 'F56%'

or table_name like 'F57%'

or table_name like 'F58%'

or table_name like 'F59%')

and not exists

(select 1 FROM dd910.f9210

where trim(frdtai) = trim(substr(column_name,3,10))

)

;

Formula for spreadsheet:

=+CONCATENATE("INSERT INTO DD910.F9210 values('",TRIM(MID(C837,3,6)),"',' ','S',",G837,",0,' ',0,' ',' ','",H837,"',' ',' ', ' ', ' ', ' ', ' ', ' ',0,' ',' ','SMOIRPC',114250,' ','SMOIR',9000, 'Description",ROW(),"', ",IF(D837="NUMBER",9,IF(D837="CHAR",2,11)),",4,'N','N','N',' ',' ',' ',' ',' ',0",",' ',0,' ',0,' ',0,' ','N',0,'N' ",");")

Which generates:

INSERT INTO DD910.F9210 values('$A01',' ','S',3,0,' ',0,' ',' ','',' ',' ', ' ', ' ', ' ', ' ', ' ',0,' ',' ','SMOIRPC',114250,' ','SMOIR',9000, 'Description837', 2,4,'N','N','N',' ',' ',' ',' ',' ',0,' ',0,' ',0,' ',0,' ','N',0,'N' );

INSERT INTO DD910.F9210 values('$A02',' ','S',3,0,' ',0,' ',' ','',' ',' ', ' ', ' ', ' ', ' ', ' ',0,' ',' ','SMOIRPC',114250,' ','SMOIR',9000, 'Description838', 2,4,'N','N','N',' ',' ',' ',' ',' ',0,' ',0,' ',0,' ',0,' ','N',0,'N' );

INSERT INTO DD910.F9210 values('$A03',' ','S',3,0,' ',0,' ',' ','',' ',' ', ' ', ' ', ' ', ' ', ' ',0,' ',' ','SMOIRPC',114250,' ','SMOIR',9000, 'Description839', 2,4,'N','N','N',' ',' ',' ',' ',' ',0,' ',0,' ',0,' ',0,' ','N',0,'N' );

 

F9861:

insert into ol910.f9861

select distinct trim(t1.table_name), 'NZAKLEVFN740',' ', 'SMOIR', 114122, 'E910', '7894561','1',' ', ' ', ' ', ' ', 'DV910', ' ', 'COWBOY', 'NDENT',114200,8000

from all_tables t1

where t1.owner = 'CRPDTA'

and (t1.table_name like 'F55%'

or t1.table_name like 'F56%'

or t1.table_name like 'F57%'

or t1.table_name like 'F58%'

or t1.table_name like 'F59%')

;

commit ;

 

F9860:

insert into ol910.f9860

select distinct trim(t1.table_name), trim(t2.table_text) || ' ','55','55', 'TBLE', '2', max(substr(t3.column_name,1,2)),'C',' ', ' ',' ','N',' ','N',0,' ',' ',' ',' ', ' ', ' ', ' ', 0, ' ', ' ', 'JDBTRIG', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'COWBOY', 'SMOIR', 'SMOIR', 115000, 5000

from all_tables t1, all_tab_columns t3, qsys2.systables@a01proddta t2

where t1.owner = 'CRPDTA'

and (t1.table_name like 'F55%'

or t1.table_name like 'F56%'

or t1.table_name like 'F57%'

or t1.table_name like 'F58%'

or t1.table_name like 'F59%')

and t2.table_schema = 'A73A01DTA'

and trim(t1.table_name) = trim(t2.table_name)

and trim(t2.table_name) = trim(t3.table_name)

and t3.owner = 'CRPDTA'

and not exists

(select 1 from ol910.f9860 where trim(siobnm) = trim(t2.table_name))

group by trim(t1.table_name), trim(t2.table_text) || ' ','55','55', 'TBLE', '2','C',' ', ' ',' ','N',' ','N',0,' ',' ',' ',' ', ' ', ' ', ' ', 0, ' ', ' ', 'JDBTRIG', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'COWBOY', 'SMOIR', 'SMOIR', 115000, 5000 ;

commit ;

 

Extra for experts:

Damn that 400 for starting table columns with a $ sign, you cannot do this in oracle. Of course the FIRST table of 110 that I test in E1 has a $ as the first char of the columns. This is defined in F98711 and F9860 and the table itself. So I need to change all of these quickly.

F9860

update ol910.F9860 set sipfx = replace(sipfx,'$','Z') where sifuno = 'TBLE' and SIPFX like '$%';

F98711

update dv910.f98711 set tdsqlc = replace(tdsqlc,'$','Z') where tdsqlc like '$%' ;

DDL for tables

SELECT 'ALTER TABLE CRPDTA.' || table_name || ' RENAME COLUMN "' || column_name || '" TO ' || replace(column_name,'$','Z') ||';'

from all_tab_columns t1

where t1.owner = 'CRPDTA'

and (t1.table_name like 'F55%'

or t1.table_name like 'F56%'

or t1.table_name like 'F57%'

or t1.table_name like 'F58%'

or t1.table_name like 'F59%')

and column_name like '$%';

This creates a bunch of:

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OUB06" TO ZOUB06;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP2" TO ZOSRP2;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP3" TO ZOSRP3;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP4" TO ZOSRP4;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP5" TO ZOSRP5;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP6" TO ZOSRP6;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP7" TO ZOSRP7;

ALTER TABLE CRPDTA.F554108 RENAME COLUMN "$OSRP8" TO ZOSRP8;

 

Conclusion:

In summary I can now use UTB or databrowser on any one of the 110 custom world tables. Sure some of the custom DD items have rubbish descriptions, this is cosmetic and could be fixed. The data is there and queryable in E1. This table definitions can be promoted from DV and included in package builds as valid objects.

clip_image002

clip_image004