Upgrade to Weblogic 10.3.6

Upgrade to Weblogic 10.3.6

Upgrade to weblogic 10.3.6 is a Prereq for inplace upgrade weblogic 12c. Java compiler upgrade is also a mandatory requirement.
The patching procedure is as follows :

• Upgrade to weblogic 10.3.6 on OEL 6/7 is done by applying the patch 13529623 
• It is best to stage the patch on a local mountpoing on the node
• Patch takes a backup of the existing domain config. however, as fallback option a snapshot is advisable prior to beginning the patching effort.
• Patch is execut by the command  

java -jar wls1036_upgrade_generic.jar

• The patch screens are quite intuitive and prompts for backups wherever necessary. 
• The patch will need the administrator to supply the domain home and a few other details. It should all complete quite quickly. (depending on the options chosen)
• The Oracles used case note https://docs.oracle.com/cd/E24902_01/doc.91/e18840/upgrade_1036.htm#EOHWL194 explains the process in great detail. 

Java upgrade for weblogic

It is best to have the JAVA_HOME setup as a soft link directory before installing weblogic. The location of the JAVA_HOME is referenced in several files in the weblogic tech stack. If we have the JAVA_HOME setup as a symbolic link, there will be no need to update references to the JAVA_HOME in case there is future need to upgrade java to a higher release. It will be sufficient to drop and recreate the symbolic link to the new version. 

Using symbolic links 

oracle$:cd /mnt/oracle/products/

oracle$:/mnt/oracle/products$ cp -rp /csa/oracle_csa/infra/oracle/java/jdk/jdk-7/linux-x64/jdk_1.7.0_80 ./jdk_1.7.0_80

oracle$:/mnt/oracle/products$ ln -s ./jdk_1.7.0_80 java

oracle$:/mnt/oracle/products$ ls -rlt java

lrwxrwxrwx 1 oracle$: oinstall 14 Feb 15 14:04 java -> ./jdk_1.7.0_80

VERIFICATION STEPS

oracle$:/mnt/oracle/products$ cd java/bin

oracle$:/mnt/oracle/products/java/bin$ ./java -fullversion

java full version "1.7.0_80-b15" 

UPDATE CONFIG FILES TO USE THE LATEST JAVA

Identify the references to /usr/java and change the config by using the following script:

oracle$:/mnt/oracle/products/wls/10.3$ grep -il "/usr/java" *|grep -iv log |grep -iv out > list

execute the follosing script to change all references to old java compile to the new compiler

oracle$:/mnt/oracle/products/wls/10.3$ cat change_java.sh

####!/usr/bin/ksh

set -x

for i in `cat list`

do

   cp $i $i"_jdk_path_po"

   sed -i 's/\/usr\/java\/jdk1.6.0_31/\/mnt\/oracle\/products\/java/g' $i

done

oracle.security.jps.wls.listeners.JpsApplicationLifecycleListener Class not Found

Resolving the Exception

"oracle.security.jps.wls.listeners.JpsApplicationLifecycleListener Class not Found"  

  On re-start of a new install of OID11g on weblogic, the following error was seen:

'oracle.security.jps.wls.listeners.JpsApplicationLifecycleListener' Class not Found 

This strongly indicates an issue with the exception in Weblogic environment. Also the deployments show remain in 'Prepared' state.

The error can be fixed with setting the nodemanager property to use the weblogic startup script by changing the parameter "StartScriptEnabled=true"

This will include correct the environment during startup.

The deployments will also be activated effectively on startup.

Problems with reinstalling SOA [ERROR]:INST-07290

OIM OAM SOA implementation

I had to install OIM OAM and SOA a few times before I got it right. The box also had OID implemented on it with several thousand users. I could not afford to purge the OID datastore every time I had to reinstalled the OIM. Down the way, I had trouble with installing more than one installations of SOA in the same middleware home. 

The error stated as below :

[VALIDATION] [ERROR]:INST-07290: Specified Middleware home already has a Oracle SOA Suite 11g home. Only one Oracle SOA Suite 11g home location can reside under a Middleware home.

[VALIDATION] [SUGGESTION]:Provide a different Middleware home location

installation Failed. Exiting installation due to data validation failure.

 I found an easy workaround to get past the problem.

I had to backup the inventory.xml file in the global inventory location and comment the REFHOME LOC lines that point to the corrupted install as seen in the screenshot below :

 Then after purging the domain applications and after correcting the file $MW_HOME/domain-registry.xml, I would be all set to reinstall OIM OAM and SOA allover once again.

OID11g DIP migrating ldap-server to a new-host

Reconfiguring AD (Domain controller)

AD – OID synchronization is setup as explained in the document  : http://docs.oracle.com/cd/E14571_01/admin.1111/e10031/odip_adm_sync.htm#OIMIG1934

For any reconfiguration relating to AD such as demoting/renaming etc, the configuration has to be updated in the following procedure :  

• Identify the “HighestCommittedUSN” of the  domain controller in use by the enabled synchronization profile. (different replicating domain controllers )

 ldapsearch –h –p 389 –D –w “ –b “” –s base “objectclass=*” highestCommittedUSN

highestCommittedUSN=3537140

•     determine the “HighestCommittedUSN” of the  domain controller to which it should be reconfigured to

ldapsearch –h –p 389 –D –w “ –b “” –s base “objectclass=*” highestCommittedUSN

 highestCommittedUSN=31243468

• Disable the current synchronization profile

•    Edit the synchronization profile to connect to the appropriate domain controller ( in this example )

•     Reset the last applied change number

- Go to the Advanced Tab

- In the section "Additional Configuration Parameters" edit 'Skip Error To Sync Next Change' and set to true

- Check "Edit and Persist" box, then Change "Last Change Number" value to a value less than highestCommittedUSN of the new DC (in this case 31243468)

Save the change

•   Enable the synchronization profile
• Monitor synchronization completes without errors

NFS nolock option

Maximum availability architecture recommends clustered deployment of middleware components and for the admin server to be positioned on a NFS mount which can switch between the clustered nodes.

I implemented one such architecture and encountered the NFS Nolock problem recently. Here is the error and a safe resolution.

======
While attmpting to startup nodemanager using wlst, the node manager starts up fine. however, the folowing error is seen. if this error is ignored, we could landup wasting a lot of time troubleshoting further.

Jul 16, 2015 11:34:35 PM weblogic.nodemanager.server.NMServer initDomains
WARNING: Configuration error while reading domain directory: /mnt/oracle/****/****/**_Domain
java.io.IOException: No locks available
     at sun.nio.ch.FileChannelImpl.lock0(Native Method)
     at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:832)
     at weblogic.nodemanager.server.UserInfo.load(UserInfo.java:149)
     at weblogic.nodemanager.server.DomainManager.loadUserInfo(DomainManager.java:139)
     at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:90)
     at weblogic.nodemanager.server.DomainManager.(DomainManager.java:60)
     at weblogic.nodemanager.server.NMServer.initDomains(NMServer.java:225)
     at weblogic.nodemanager.server.NMServer.start(NMServer.java:197)
     at weblogic.nodemanager.server.NMServer.main(NMServer.java:382)
     at weblogic.NodeManager.main(NodeManager.java:31)

Jul 16, 2015 11:34:35 PM weblogic.nodemanager.server.Listener run
============

While starting Admin server, using wlst, the following error will be seen  :

There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /user_projects/domains/biadm10_bifoundation_domain/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
 
 =============
90% of the problem is resolved when you have determined that the problem is due to NFS and not due to the non availability or availability of the lck file :) and the remaining 10% is to make a small edit on  /etc/fstab

Note (Doc ID 1635553.1) has the solution explained. I had the problem resolved by enable the nolock mount option.

nolock,vers=3,tcp,timeo=300
nolock,vers=3,tcp,timeo=300


Cheers :)
Arun
 

FRM 93552

FRM 93552

I had to administer a new implementation n 11.1.2 version of forms and reports server.

I had a good chance to learn that there are a few fundamental differences in 10g and 11g version of forms.
It is best practice in 11g to make formweb.cfg and other config file changes by using the em console.  sometime it becomes an intermittent problem in case this thumb-rule is ignored.

Following forms error is a manifestation of a manual change made to formsweb.cfg on the forms server.

 

 In case the error is accompanied with the following errors, as is seen in the diagnostics log here :

 

There is a oracle support note that recommends a native library :

Getting FRM-93552 When Calling Forms (Doc ID 1345748.1)

here the recommendation is to include the actual path for ORACLE_HOME/jdk/jre/lib/amd64/native_threads LD_LIBRARY_PATH variable in the ******.env file. Again, it is best to update the *****.env file over the em console

 

This was a good lesson learned!!

Cheers, 

Arun

Volume check

In one project, We had a service provider supporting storage and infrastructure.

One fine morning we heard there would be a network outage impacting all services. This meant an overnight schedule involving :

• Prep work
• Prevention of autostart scripts/crons/batches and the enterprise scheduler
• Shutdown of the whole production stack prior to the outage
• Manual startup
• Validation of services
• Enable autostart, cron, batches and enterprise scheduler
• makeup lost schedule executions 
• Wrap up

This was a tedious schedule spanning several servers, several DBAs working through the night.

The service provider added that the storage may be drawn away from the servers during the outage and nfs volumes need to be validated before startup.

I put together a handy script to put to rest the possibility of manual errors in the wee hours of the morning
following is the script attached  :) --> enjoy

nfs volume check