DBCP Connection pooling – not so by default

Unlike in C#, one has to setup database connection pooling in Java. Not a big deal, as there are several ready-to-use libraries. We picked DBCP from apache commons. While unittests succeeded, stress tests were failing after exhausting all available TCP connections... To make long story short, the constructor we used to setup Object Pool did not pool connections: at first our code looked like this:

ObjectPool connPool = new GenericObjectPool(null, maxActive, GenericObjectPool.WHEN_EXHAUSTED_FAIL);

However, this only limits db connections but doesn’t reuse them. You can confirm this by either looking at the connection ids in my MySQL admin or simply running “netstat –a” and notice the ever-growing number of TCP connections – Windows holds on to TCP connection for a few minutes even after db connection is closed. The fix was rather simple: pass all required parameters:

ObjectPool connPool = new GenericObjectPool(null, maxActive,
GenericObjectPool.WHEN_EXHAUSTED_FAIL,
-1, //maxWait,
-1, //maxIdle,
false, //testOnBorrow,
false, // testOnReturn,
-1, //timeBetweenEvictionRunsMillis,
-1, //numTestsPerEvictionRun,
-1, //minEvictableIdleTimeMillis,
false); //testWhileIdle

C# wasn’t that bad after all, was it? ;)

Tracking requests in multithreaded environment

One day night you'll have to troubleshoot your application and figure out what requests are causing error logs to fill up.   You can expedite investigation if you are able to trace every log statement done during processing of the request that resulted in error.  Usually, you can use thread local storage to stick a a random number at the beginning of processing of every request and print it in the logs.  However, since you're using log4j (aren't you? :-) ), you can take advantage of log4j.MDC object - "an instrument for distinguishing interleaved log output from different sources."

Follow these 3 steps:

• create a log4j decorator object (MyLogDecorator)

import java.util.Random;
import org.apache.log4j.MDC;

public class MyLogDecorator { 
  private static Random myrandom = new Random(); 
  // use enum to ensure compile-time safety 
  public enum LoggerFields{
        RequestId,
        //plus other fields relevant to your app
    }

public static void put(LoggerFields key, Object value){
        MDC.put(key.toString(), value.toString());
}
public static void remove(LoggerFields key){
      MDC.remove(key.toString());
}
/* make sure that context is empty and add a

random request id */

public static void beginRequest(){
if (MDC.getContext()!=null &&

    MDC.getContext().size()>0){
endRequest();

}

put(LoggerFields.RequestId,

    myrandom.nextInt(10000000));
}

public static void endRequest()    {
for(LoggerFields key : LoggerFields.values())    {
            remove(key);
}}}

• call MyLogDecorator inside a Tomcat filter. Filter code is fairly standard.  Note that MyLogDecorator.beginRequest() call can be made by any application (e.g. multithreaded command-line up).

public void doFilter(...) {

    MyLogDecorator.beginRequest();
    chain.doFilter(request, response);
}

• update pattern layout in log4j.xml to automatically add request id to every log statement:

  <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} -%1X{RequestId} %c - %m%n"/>

The Taming of the JMeter: command-line arguments, tidy, bean shell

JMeter is powerful but poorly documented. Here’re few things that everyone could use:

• In order to change User Defined variables from the command line, convert them to properties:

Now you can store default values in user.properties file and override them by passing -J parameter on the command line: e.g. -Jthreadcount=1 -JdataFile=data1.xml

If user.properties file has changed, jmeter must be restared.

• JMeter allows for sophisticated XPath expressions that can be used to parse a random HTML page – just make sure to set “Use Tidy” check-box.

XPath expression can also include JMeter variables, e.g.

/html/body/table/tbody//tr[${__V(server${serverIndex}Section2Header)} + 2]//text()

This is pure goodness…

• What makes JMeter even more powerful is its ability to integrate with Bean Shell – you have to copy beanshell’s bsh-2.0b4.jar to jakarta-jmeter-2.3\lib\ext and rename this as bsh.jar – that’s it… Buyer beware: invoking bean shell in a loop is awfully slow. Debugging a bean shell script is nightmare consisting of adding print statements throughout your code. Passing –LDEBUG flag may help with troubleshooting.
  
  Sometimes I feel it’s just easier to dump data extracted with JMeter script and analyze it with a standalone java program.

If JMeter had tighter integration with BeanShell or, even better, allowed for calling user-specified java classes within the same

JVM (to speed things up), it’d rule!

Technorati Tags: JMeter

Deterministic unittests with ExecutorService

Java threadpool (ExecutorService) objects have some differences with .NET Threadpool:
- you can code to the ExecutorService interface
- you can call awaitTermination() to let all threads finish
- and you can use different thread pool objects, for example:

· Executors.newCachedThreadPool() is similar to .NET Threadpool where you let JVM manage the number of active threads. However, .NET Threadpool resists creating new threads (and rightfully so), while cached thread pool will create one on demand (according to Effective Java™, Second Edition.)
· Executors.newFixedThreadPool() let you control the number of thread.

While I usually default to a fixed thread pool, I think one should put code under stress to see which one is better and also allow for switching between two if underlying implementation change – code to the interface!

int numberOfThreads = //get from configs
if (numberOfThreads<=0){
executor = Executors.newCachedThreadPool();
}
else{
executor = Executors.newFixedThreadPool(numberOfThreads);
}

I use jmock a lot for unittests but quickly discovered that it is not threadsafe. I wanted deterministic execution of my unittests, but couldn’t give up on either Jmock or threadpool. I've read jmock faq on threads and got an idea to mock both threadpool and Future objects:

public class MockThreadPoolExecutorFixture implements ExecutorService {
private MockThreadPoolExecutorFixture(){}

public static ExecutorService getInstance(){
return new MockThreadPoolExecutorFixture();
}

@Override
public Future submit(Callable mytask){
MockFutureFixture myfuture = new MockFutureFixture(mytask);
return myfuture;
}
/*other API return default values of null or false – they were not used, feel free to override them as well if need be*/
}

public class MockFutureFixture implements Future {
private Callable mycall;
public MockFutureFixture(Callable arg0) {
mycall = arg0;
}

@Override
public V get() throws ExecutionException {
try{
return mycall.call();
}
catch (Exception e){
throw new ExecutionException(e);
}
}
/*other API return default values of null or false – they were not used, feel free to override them as well if need be*/
}
Of course, the next step is to stress the hell out of your code using real threadpool to see if you get thread synchronization problems.

Debugging under Tomcat

Ever wonder how to debug a webapp running under tomcat? This tomcat plug-in for Eclipse is just about perfect. Main documentation is in French… so here the trick: after following the installation steps simply configure it in Windows->Preferences->Tomcat (see attached screenshot).

Now you can start your app from within eclipse and debug away!

Perf Counters using JMX MBeans

Performance Counters are part of Windows and are easily available within any .NET app. A bit of kludgy install code and you’re pretty much done. Various monitoring system pick up data, monitor the site and alarm Ops.

JMX MBeans is a Java approach for monitoring and management of your application. MBeans not only allow you to instrument the code but also allow your application to notify listeners when something goes wrong. Moreover, they allow for a back-channel to your application to allow for changing behavior on a fly. While the latter feature if nice, I honestly don’t see anyone changing production environment in this manner. Still, it’s a rather slick solution for, say, evaluating various config settings during development or stress testing.

When tasked with instrumenting my application, start by implementing generic “perf counters”. Check .net documentation and implement similar objects in Java. For example, for Average perf counter specify two APIs: add and get

public void add(int delta)
{
operationscounter.incrementAndGet();
timecounter.addAndGet(delta);
}

public double get()
{
int currentValue = operationscounter.getAndSet(0);
int accumulatedTime = timecounter.getAndSet(0);

return accumulatedTime / (double) currentValue;
}

While both both operationscounter and timecounter are AtomicIntegers, the above APIs are not thread-safe. I purposely left out synchronized keyword to get better performance. We’re not counting $$ after all, so it’s ok if data is slightly off. The second - get - api also resets both counters to 0 (ok, check that denominator is not zero; check for overflow – small details).

Similarly, you can create a rate perf counter:
public void increment()
{
mycounter.incrementAndGet();
}

public double get()
{
long timeNow = System.currentTimeMillis();
final double delta = timeNow – lastPollTime;
lastPollTime = timeNow; //reset time stamp
double result = currentValue/delta * multiplier;
// multiplier=1000 when returning results in seconds
return result;
}

Once perf counters are written, instrument your application and expose data via an MBean. MBean is a regular java class implementing a specially constructed MBean interface: only few rules to follow: your class must have a default constructor; if you class name is MyService, than the MBean interface must be name MyServiceMBean, etc. Amazon has a few JMX-related books; they don’t cover MXBeans which are thoroughly described in this article.

e.g. MBean interface:
public interface MyServiceMBean
{
public int getRequestsCount();
public int getSystemFailuresCount();
public double getAverageProcessingTime();
public double getRequestRate();
}

public class MyService implements MyServiceMBean

{
...
}
As with Windows perf counters installation, there is a bit of initialization code (5 lines below) and then any JMX-aware monitoring system will be able to see MBeans and poll for data. You can also run jconsole during development phase to verify your JMX MBean implementation.

Initialization code:
MyService instance = new MyService();
String jmxlabel = “MyService:type=MoneyMakerService”;
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName(jmxlabel);
mbs.registerMBean(instance, name);

One caveat: you can’t pack various counter data (request count, rate, etc) into an object or return them as array: otherwise (as of today) data won’t be processed correctly by some management systems. MXBeans appeared to be an answer as the data can be indeed wrapped into a class and shows up nicely on the monitoring side… but, at least with Manage Engine, the values are treated as strings so you can’t alert on them. You can easily convert to MXBeans later on if need be – but the perf counters classes you build would not have to change!

Web Services

It’s all about web services nowadays. It was very simple to create one in .NET – just run dev studio wizard and off you go. Not so fast in Java. First you should pick a library that takes care of SOAP layer. There are a couple big ones, namely Axis2 and XFire/CXF, both look robust enough.

Axis2 is a complete re-write and in some cases is not compatible with original Axis clients. I still picked Axis2 as it has simple way of wrapping objects into web service calls and doesn’t require tagging them with attributes like CXF. Also, Axis2 allows your object be exposed via REST. Basically, you create a POJO (fancy word for a simple object with setters and getters), update service.xml file to reference this object and you’re done…
Well, there are some caveats:

• You must have an empty constructor.
• Do not overload constructors – it usually works but I’ve ran into scenario where objects were not de-serialized properly and wasted a day chasing down the problem: removing overloaded constructors fixed the issue.
• No helper functions other than setters/getters in your object – or they’ll show up in wsdl and mess things up on the client side.
  

e.g. my entire web service layer now looks like this:
public class MyWS
{
public MyWSResponse SomeRequest(String myarg)
{
MyWSResponse = MyLogicLayer.getInstance().processSomeRequest(myarg);
}
}

MyWSResponse has only an empty constructor and few setters/getters. Add the following to service.xml and you’re done.

<service name="MyWS" scope="application">
<description>
My SOAP Service
</description>
<messageReceivers>
<messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-out"
class="org.apache.axis2.rpc.receivers.RPCMessageReceiver"/>
</messageReceivers>
<parameter name="ServiceClass">com.mycompany.MyWS</parameter>
</service>

Entire logic is encapsulated in the MyLogicLayer – makes a very convenient setup for unittests.

You can run axis2 web service standalone or package it in a war file with other web services or servlets.

Axis2 comes with many jars – I recommend keeping them in a separate folder and package with your code in build time - this way upgrade to newer Axis2 release will be easy.

Finally, you can extend Axis’ AbstractHandler to create a module logging every incoming and outgoing message (which was rather painful to do in .NET). The skeleton code is below:

public InvocationResponse invoke(MessageContext msgContext) throws AxisFault
{
HttpServletRequest request = (HttpServletRequest)msgContext.getProperty("transport.http.servletRequest");
//call request.getHeaderNames() and log headers

//then log message body: msgContext.getEnvelope().toString()

return InvocationResponse.CONTINUE;
}

I also removed axis admin servlet from final deployment – I am a bit paranoid of having a GUI-based access to production environment (or I’ll be tempted to make ad-hoc changes to running system at 3am :-).