Friday, August 16, 2013

WLST; obtaining parameters, recovering JDBC database user passwords and testing database connections

WLST (Weblogic Scripting Tool) is very powerful. Most of the things (and more) which can be done with the Weblogic console can also be done by means of WLST scripting.

I've already written a post describing two options for datasource monitoring; http://javaoraclesoa.blogspot.nl/2012/09/monitoring-datasources-on-weblogic.html. The methods described in that post have some drawbacks;
- by using a servlet, you are exposing server datasource status and you are using a custom developed servlet to achieve functionality. the servlet does not show connection errors, just OK or NOK. Also it does not take into account different managed servers and datasource targets.
- by using WLST as in the example in the post, you're not actually testing creating a connection but are just monitoring current status

What the script should do

At a customer I noticed database availability was an issue. This often caused data sources to go to 'Suspended' state. Also when the database was available again, we often encountered connection errors like 'ORA-12514: TNS:listener does not currently know of service requested in connect', 'ORA-011033: ORACLE initialization or shutdown in progress' and 'ORA-01017: invalid username/password; logon denied'. The customer used a multitude of datasources. We wanted a quick way to resume every one of them and determine connection exceptions in order to inform the DBA to fix it. Since the script would run on several machines, it should be able to determine the IP to connect to and the required paths (to for example SerializedSystemIni.dat) on it's own.

In the below image, the 'call chain' is illustrated. This post focusses on the WLST script. In a second post (http://javaoraclesoa.blogspot.nl/2013/09/inform-dba-if-weblogic-cant-connect-to.html) I'll describe how we automated the process of calling the script over SSH by using an Ant plugin in Maven on several environments so we could schedule this to run every morning in Jenkins and automatically mail the DBA's to go and fix their DB's.
How the script is implemented

Obtaining local information
Since the script needed to be as environment neutral as possible, I obtained several pieces of information from the machine the script runs on.

Obtaining the physical interface IP address
I obtained the IP address of the local machine by using the ip command. See; http://stackoverflow.com/questions/6243276/how-to-get-the-physical-interface-ip-address-from-an-interface

intf = 'eth0'
intf_ip = commands.getoutput("/sbin/ip address show dev " + intf).split()
intf_ip = intf_ip[intf_ip.index('inet') + 1].split('/')[0]
print 'Using IP: ',intf_ip

Obtaining the path to SerializedSystemIni.dat
I needed to obtain the path to SerializedSystemIni.dat for decrypting passwords (see later in this post). I obtained the path by (after connecting);

rootdir=cmo.getRootDirectory()
secdir=rootdir+'/security'

secdir is the directory where the SerializedSystemIni.dat file is usually located.

Obtaining parameters
I wanted my script to be flexible. I wanted to have the option to use a properties file (useful for the development environment), to use command-line arguments (useful when calling the script from Maven) and I wanted the script to ask for login details otherwise (useful for a DBA executing the script). This required 3 methods of obtaining parameters. The logic used was as followed;

- do we have a file? if so, use it else continue
- do we have command line arguments. if so, use them. if not, ask for them

Command line arguments
In Weblogic 10.3.6, Python/Jython 2.2.1 is used (https://forums.oracle.com/thread/2210448). This limits the use of several of the more recent Python libraries to make working with parameters more easy (such as optparse and argparse). We can however use getopt; http://davidmichaelkarr.blogspot.nl/2008/10/make-wlst-scripts-more-flexible-with.html

The above described library also covered asking for the parameters if they were not supplied. If this failed for whatever reason, raw_input can be used.

Properties file
Using a properties file was relatively easy. See for example; http://java-brew.blogspot.nl/2011/01/reading-properties-file-in-wlst-jython.html

Because we are using Jython with WLST, we can use the Properties class from java.util. No need to reinvent the wheel here.

Obtaining login details and testing connections
When a datasource is suspended and the connection pool is tested, you can the following exception;
Connection test failed with the following exception: weblogic.common.resourcepool.ResourceDisabledException: Pool testuserNonXa is Suspended, cannot allocate resources to applications.

When however a connection cannot be made, you can resume the datasource and test it and nothing will appear to be wrong. In the log file however you might see one of the previously mentioned exceptions; you won't be able to use the datasource to get to the database. So testing just the datasource from the Weblogic console is not enough to confirm it is working.

To determine if a connection could be made, I wanted to create a new connection to the database while not using the datasource but with the same connection details. For this I needed to obtain the connection information the datasource was using. Then I encountered the following challenge; the cleartext passwords could not be read due to Weblogic server policies; "Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean". Also see; http://serverfault.com/questions/386724/clear-text-credential-access-enabled-field

I did not want to change this policy as it would introduce a security vulnerability. I had to decode the passwords. Based on several blog posts such as; http://connectionserver.blogspot.nl/2009/06/recovering-weblogic-passwords.html I could recover the database user passwords (which is of course very useful...). The following which I found online is also interesting; http://recover-weblogic-password.appspot.com/. As you can see in the code below, you can uncomment a line to display the passwords in plain text on the console.

First I specified the path where the SerializedSystemIni.dat file could be found. Then I used that to decrypt the encrypted passwords I obtained from the MBeans. Then I used zxJDBC to connect to the database using the obtained credentials; http://www.informit.com/articles/article.aspx?p=26143

The script

Mind the indentation! It's Jython. You should execute this with the wlst.sh script in your Weblogic server installation.

import commands
import os
import weblogic.security.internal.SerializedSystemIni
import weblogic.security.internal.encryption.ClearOrEncryptedService
import traceback
import sys
import getopt

from com.ziclix.python.sql import zxJDBC
from java.io import FileInputStream

intf = 'eth0'
intf_ip = commands.getoutput("/sbin/ip address show dev " + intf).split()
intf_ip = intf_ip[intf_ip.index('inet') + 1].split('/')[0]
print 'Using IP: ',intf_ip
var_user=''
var_pass=''

try:
        fh = open("resume.properties", "r")
        fh.close()
        print 'Using resume.properties'
        propInputStream = FileInputStream("resume.properties")
        configProps = Properties()
        configProps.load(propInputStream)
        var_user=configProps.get("userName")
        var_pass=configProps.get("passWord")
except IOError:
        try:
                opts, args = getopt.getopt(sys.argv[1:], "", ["username=", "password="])
                for o, a in opts:
                        if o == "--username":
                                var_user=a
                                print 'User: ',var_user
                        elif o == "--password":
                                var_pass=a
                                print 'Pass: ',var_pass
                        else:
                                assert False, "unhandled option"
        except getopt.GetoptError, err:
                print 'No -u and -p commandline arguments and no resume.properties...'
                var_user = raw_input("Enter user: ")
                var_pass = raw_input("Enter pass: ")

connect(var_user,var_pass,intf_ip+':7001')
rootdir=cmo.getRootDirectory()
secdir=rootdir+'/security'
allServers=domainRuntimeService.getServerRuntimes();
if (len(allServers) > 0):
  for tempServer in allServers:
    print 'Processing: ',tempServer.getName()
    jdbcServiceRT = tempServer.getJDBCServiceRuntime();
    dataSources = jdbcServiceRT.getJDBCDataSourceRuntimeMBeans();
    if (len(dataSources) > 0):
                for dataSource in dataSources:
                        #print 'Resuming: ',dataSource.getName()
                        dataSource.resume()
                        dataSource.testPool()
                        cd('/JDBCSystemResources/' + dataSource.getName() + '/JDBCResource/' + dataSource.getName() + '/JDBCDriverParams/' + dataSource.getName() + '/Properties/' + dataSource.getName())
                        dbuser=cmo.lookupProperty('user').getValue()
                        #print 'User: ',dbuser
                        cd('/JDBCSystemResources/' + dataSource.getName() + '/JDBCResource/' + dataSource.getName() + '/JDBCDriverParams/' + dataSource.getName())
                        dburl=cmo.getUrl()
                        #print 'DbUrl: ',dburl
                        dbpassword=cmo.getPasswordEncrypted()
                        es=weblogic.security.internal.SerializedSystemIni.getEncryptionService(secdir)
                        ces=weblogic.security.internal.encryption.ClearOrEncryptedService(es)
                        dbpassword_decrypted=str(ces.decrypt("".join(map(chr, dbpassword))))
                        #print 'DbPassword: ',dbpassword_decrypted
                        dbdriver=cmo.getDriverName()
                        #print 'DbDriverName: ',dbdriver
                        try:
                                con=zxJDBC.connect(dburl,dbuser,dbpassword_decrypted,dbdriver)
                                cursor=con.cursor()
                                result=cursor.execute('select sysdate from dual')
                        except:
                                print 'ERROR: Url: ',dburl,' User: ',dbuser
                                traceback.print_exc()

Saturday, August 10, 2013

Book review; Oracle SOA Suite 11g Performance Tuning Cookbook

When implementing Oracle SOA 11g, performance is an often encountered topic. I have been involved at several customers in performance taskforces and I've read several books and a lot of Oracle documentation related to the topic. As such I was very interested in the book; Oracle SOA Suite 11g Performance Tuning Cookbook (http://bit.ly/12lrajU). I was curious as to whether this book would contribute much to the topic and if I would learn new and interesting things.

I can be a bit of a nit-picker at times. I personally have not written any books yet. For every chapter I've noted something positive and something which I probably would have done differently (or maybe when I know the reason why things are done a certain way I might agree on it). This is not meant as personal criticism to the writers as they clearly know what they are writing about.

Review per chapter

Chapter 1 Identifying problems
The first chapter is focused on how to identify problems. Problems can be high CPU usage (due to for example garbage collection) and for example slow database queries.

Positive
VisualVM was new to me and the analyses of stuck threads I might also use in the near future. JRockit Mission Control was already familiar (http://javaoraclesoa.blogspot.nl/2012/09/overview-monitoring-tools-oracle-jrockit.html) but it will be useful for others.

Negative
Because the book is in cookbook format and every recipe stands more or less on it's own, there is some repetition in for example how to use jps and jstat. I would have combined the recipes. What I was missing a bit here is concrete examples. How does the output of commands look. Also 'the survivor spaces and Eden' are introduced shortly which is a major topic in Chapter 5. The discussion on how to find problems with datasources could have been moved to Chapter 7.

Chapter 2 Monitoring Oracle SOA Suite
This chapter describes how VMWare's VFabric Hyperic HQ can be used to monitor Weblogic instances. Hyperic has an open source version; http://www.hyperic.com/hyperic-open-source-download.

Positive
The choice for Hyperic is well supported by arguments and the description on how to get it installed and configured is thorough. It looks like an interesting tool and works by installing an agent on the system to monitor. I'm currently using Nagios (http://www.nagios.org/) to monitor environments on a per (web)service level. I would have to try out Hyperic in order to determine if it can do that as well.

Negative
Monitoring is focused on OS/Weblogic health and not specific Oracle SOA Suite. The dms servlet is also discussed, however shortly. I would have left this out of the chapter and focus on Hyperic. If discussed, I would have expanded it a little such as for example add how it can be purged (for example at the end of http://javaoraclesoa.blogspot.nl/2012/08/issues-and-solutions-when-testing-and.html).

Chapter 3 Performance testing
JMeter is used to do performance testing on webservices.

Positive
Interesting topics were mainly distributed tests and the part on testing Amazon cloud services. Both were new to me.

Negative
I'm used to using SOAP UI for webservice tests and HermesJMS for JMS tests. Both are not discussed. It's most likely a matter of taste. The choice for JMeter is not supported by arguments. I missed performance testing of JMS queues here.

Chapter 4 JVM Memory
This chapter describes how the JVM uses memory and how this can be monitored/tuned.

Positive
This chapter contains a lot of explanation on the memory settings. This is something which I found often lacking in other books. 'A good rule of thumb is to stick to powers of 2', which is a good tip. Also; 'Note that by default the flag JAVA_USE_64BIT is set to false in %WL_HOME%\common\bin\commEnv.cmd.' is something I'll check out. The picture on JVM memory was clarifying. The division in JVM settings for JRockit and Hotspot is also useful.

Negative
This chapter also contains parts on JRockit Mission Control and VisualVM. Although these are very useful, it could have been combined with the part in Chapter 1.

Chapter 5 JVM Garbage Collection Tuning
The different garbage collection strategies for JRockit and Hotspot are explained here and how they can be tuned.

Positive
Disabling the RMI garbage collector was something new to me. Also disabling explicit garbage collection will be interesting to try and see if it makes a difference. Although applications shouldn't be using the System.gc() command, there are always programmers who sneak such code in.

Negative
Little to remark here. I liked reading the chapter. There was a bit more focus on Hotspot then on JRockit.

Chapter 6 Platform Tuning
This chapter describes options for tuning the platform. It is difficult to find a common denominator or focus point for the topics discussed in this chapter. The title fits.

Positive
JTA max transactions, HTTP accept backlog,  Stuck Thread Time, HugePages are settings I'll try out tuning at customers. Reducing OS swappiness is interesting as I've found that swap file usage can reduce performance dramatically. Also I got reminded that the OS also requires specific settings. I'll check if my customers adhere to the Oracle suggested settings. Of course there are also the more often mentioned settings such as tuning EJB timeouts present in this book.

Negative
First the stack is explained, SOA Suite runs on application server, application server runs on JVM, JVM runs on OS, OS runs on hardware. Then the chapter does not follow a top-down or bottom-up approach describing tuning options (or I've missed this). Suggesting to upgrade the JVM and use for example Java 1.7 is dangerous as I'm not sure Oracle supports that in combination with Oracle SOA Suite 11g (see; http://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html). Describing tuning options on OS level could have been expanded a bit. For example I'm curious as to specific tuning options for different Linux distributions.

Chapter 7 Data Sources and JMS
This chapter described tuning of JMS and datasources.

Positive
Good to know; the settings for datasource testing can reduce stale connection issues. Also when tuning XA datasources, the database has to be tuned for this also; distributed_lock_timeout. The part on JMS tuning was relatively new to me but I am certain to use the knowledge when the need arises. Also a lot on JDBC datasource tuning was already familiar to me but useful for others.

Negative
It would have helped if the error messages which can occur when certain settings are incorrectly/sub-optimally configured, would have been added. This might make finding the right recipe to use when a certain problem occurs easier.

Chapter 8 BPEL and BPMN Engine Tuning
This chapter describes tuning of the BPEL and BPMN engine.

Positive
Not much new here for me. The most important settings are covered in my opinion. The tuning of the Db settings is something I'll go check at customers as the SOAINFRA performance is often a bottleneck.

Negative
Purging is a hot topic and could have been more thoroughly explained. It would have been nice if the 11.1.1.7 enhancements were also discussed but since it is relatively new, one can't blaim the writers for that. SOAINFRA tablespace defragmentation is not described (the rebuild index, shrink space, enable row movement kind of statements. for example http://javaoraclesoa.blogspot.nl/2012/06/increasing-bpel-performance.html). There were no BPMN engine specific tuning suggestions.

Chapter 9 Mediator and BAM
This chapter describes tuning options for the Mediator and BAM.

Positive
BAM tuning is not described often. A lot of familiar settings for the Mediator (also from http://javaoraclesoa.blogspot.nl/2013/07/oracle-soa-blackbelt-training-june-2013.html) but useful for people who are new to it.

Negative
It is a relatively short chapter and could have been combined with Chapter 10.

Chapter 10 Rules and Human Workflow
Optimization options for business rules and human workflow are discussed here. I have little experience with the components.

Positive
Choosing the right client to access the workflow component is interesting. Also how the rule engine can be used efficiently is something I will take with me when the need arises to use the component.

Negative
Not much to remark here except the same as in Chapter 9.

Chapter 11 SOA Application Design
Application design has a great impact on performance. Several suggestions are given in this chapter which can help.

Positive
Although this chapter does not appear to have been the main focus of the book, it does touch several of the more important areas to look for performance improvements.

Negative
I could write an entire book on the topic if time would allow me. Also I have done several measures in order to check whether specific settings actually increase performance. See for example http://javaoraclesoa.blogspot.nl/2012/09/suggestions-to-increase-bpel-performance.html. The training I've recently followed also gave several best practices on the topic; http://javaoraclesoa.blogspot.nl/2013/07/oracle-soa-blackbelt-training-june-2013.html. What I found lacking was programming practices. Which patterns to use and which to avoid and for example how to process large batches efficiently. I missed how to code processes with keeping transaction bounderies in mind (http://javaoraclesoa.blogspot.nl/2013/06/oracle-soa-11g-bpel-transaction.html).

Chapter 12 High Performance Configuration
This chapter gives suggestions for how a configuration can help increase performance. It is focused on OHS, JMS, Virtualization and hardware.

Positive
The parts on virtualization and hardware required I will use as arguments to talk to the people responsible for that part at my current customer that some improvements might be a good idea.

Negative
The JMS part could have been put in the JMS chapter. Clustering issues are barely touched such as local optimization settings and issues with the EIS adapters. I would have expected to find them in this chapter.

Conclusion

I liked this book. I have learned some new things I can use at customers. This book also does not repeat the manuals and other books much which also makes it interesting. It is the first book I have seen with focus completely on performance and tuning of Oracle SOA Suite 11g. Not only does the book contain the usual performance tips and several new ones but it also provides some suggestions for tuning in a virtualized environment (including a suggestion on how to measure cloud performance) and hardware.

I am no fan of the recipe format and would have grouped some things differently to describe a more bottom up or top down approach to tuning a SOA Suite installation. Especially JVM tuning gets a lot of attention. It is clear the authors know what they write about. Also the writing style makes this book an easy and enjoyable read. I did not get bored.

I missed a bit tuning of the OSB. Maybe some BPEL programming practices or patterns which help improving performance. What I also missed was tuning from a high level down to a single BPEL process; how to bridge the gap from JVM measures to BPEL process instances. There is not much focus on dehydration store maintenance which can also be an important factor in improving performance. Clustering issues are barely touched.

On the whole this was an interesting read with a lot of useful suggestions.