Thursday 28 May 2009

Package deployment

We were hesitating last night to deploy a package on a server that had active SBS jobs running.  We were doing a UA package and saw that there were PD SBS jobs running.

It seems that you can do this, supporting (and not so supporting) documentation is below about the 812 package deployment process.

SBS running

An active subsystem job that is constantly running with a processing status is seen by the deployment as a job that it can not put on hold.  This requires end users to manually stop any 'processing' subsystem jobs so that a package deployment can be successfully completed.  A full lock on the files being changed by the deployment is required and this requires a hold on all jobs in the queue and kernel processes on the logic server.

Recoverable failed deployments

Good news is that the package can be redeployed at anytime (they are on 8.96.1.5)

This issue is fixed in BUG:8356917 (Tools Release 8.96.1.3 or higher) in which user do not need to restart services in order to unlock the server. If the Package deployment hangs due to certain interuptions e.g. long running business functions,subsystem jobs, UBEs currently running, etc, user can resubmit the Package Deployment UBE R98825D by clicking the Deploy button in Package Deploy application after those interuptions has fininshed and the system will do a clean up to unlock the locks and re-deploy the package.

Package deploy notes.

The solution is based on the existing process of how the server deployment process works.  Once the R98825D is kicked off, the deployment process signals all kernels on the server to go into a wait status.  Any job or process currently running will continue to process to completion, however, any process or job that is in the queue will be put on hold.
The end effect on end users is that they will ultimately not be able to process anything that uses the logic server (i.e. BSFNS mapped to the server, UBE jobs, Scheduled Jobs) during the time it takes for the deployment to finish.  Update packages would be much faster and full packages can take more time depending on hardware speed and disk I/O.
With this effect occurring, it is not recommended that clients deploy packages during normal hours of operation in a production environment.  The effect on changes to the specs and business libraries on dependent processes still waiting in the queue for after the deploy finishes could cause unexpected results.  For instance, if a transaction requires 5 functions to run and they are dependent on each other in the process then it would be possible that a deployment submitted while the first function is running could change the outcome of the remaining functions based on changes to the objects being used.   For this reason, it is recommended that production deployments be scheduled during a down time or during a slow period where end users are limited or doing read only type processes.
A true 24/7 shop may require a unique or custom solution to hot swap the servers to provide minimal downtime.  This is not supported by GSC officially and should be referred to our field support or business partners for official support of these types of solutions.  Even with a hot swap type solution, ultimately end users will be impacted and transactional processes may be impacted.
Clients that are the least affected by deployments to a logic server are those running a true fat client setup which is supported through the 8.10 release.  A Citrix or Terminal Server setup has the same negative impact on end users like HTML users normally since OCMs would default business functions to run on the server.  Running business functions locally in a terminal server setup can make your end users less impacted by deployment but will also negatively impact performance and the total number of users that can run concurrently on those machines.  HTML users, which is the only option in 8.11 and above will always be negatively impacted by deployments and this is the reason why the subject ofdeployment recommendations has more recently become an important topic with the EnterpriseOne client base.
Other ideas to help minimize impact on a production system include using multifoundation or separate logic servers for DV and PY versus a PD system.  The deployment of a package to DV on the same machine as PD sharing the same port would have the same effect of locking down all kernels and negatively impacting productions users.  The use of separate ports on the same logic server can effectively separate a production pathcode from deployment events occurring in other pathcodes.
Some enhancements to the deployment process seen in 8.96_A1 and above is the ability for a deployment process that is hung due to long running business functions to be resubmitted and recover any held kernel processes.  This will help minimize the need to refresh services if a deployment process hangs or fails due to network failures.  Simply rerunning the R98825D will accomplish this in 8.96_A1 and above.
Update 2/7/08 -- Two new SARs have been entered to address issues with the new requirements around the package deployment.

No comments: