Perf Counters using JMX MBeans

Performance Counters are part of Windows and are easily available within any .NET app. A bit of kludgy install code and you’re pretty much done. Various monitoring system pick up data, monitor the site and alarm Ops.

JMX MBeans is a Java approach for monitoring and management of your application. MBeans not only allow you to instrument the code but also allow your application to notify listeners when something goes wrong. Moreover, they allow for a back-channel to your application to allow for changing behavior on a fly. While the latter feature if nice, I honestly don’t see anyone changing production environment in this manner. Still, it’s a rather slick solution for, say, evaluating various config settings during development or stress testing.

When tasked with instrumenting my application, start by implementing generic “perf counters”. Check .net documentation and implement similar objects in Java. For example, for Average perf counter specify two APIs: add and get

public void add(int delta)
{
operationscounter.incrementAndGet();
timecounter.addAndGet(delta);
}

public double get()
{
int currentValue = operationscounter.getAndSet(0);
int accumulatedTime = timecounter.getAndSet(0);

return accumulatedTime / (double) currentValue;
}

While both both operationscounter and timecounter are AtomicIntegers, the above APIs are not thread-safe. I purposely left out synchronized keyword to get better performance. We’re not counting $$ after all, so it’s ok if data is slightly off. The second - get - api also resets both counters to 0 (ok, check that denominator is not zero; check for overflow – small details).

Similarly, you can create a rate perf counter:
public void increment()
{
mycounter.incrementAndGet();
}

public double get()
{
long timeNow = System.currentTimeMillis();
final double delta = timeNow – lastPollTime;
lastPollTime = timeNow; //reset time stamp
double result = currentValue/delta * multiplier;
// multiplier=1000 when returning results in seconds
return result;
}

Once perf counters are written, instrument your application and expose data via an MBean. MBean is a regular java class implementing a specially constructed MBean interface: only few rules to follow: your class must have a default constructor; if you class name is MyService, than the MBean interface must be name MyServiceMBean, etc. Amazon has a few JMX-related books; they don’t cover MXBeans which are thoroughly described in this article.

e.g. MBean interface:
public interface MyServiceMBean
{
public int getRequestsCount();
public int getSystemFailuresCount();
public double getAverageProcessingTime();
public double getRequestRate();
}

public class MyService implements MyServiceMBean

{
...
}
As with Windows perf counters installation, there is a bit of initialization code (5 lines below) and then any JMX-aware monitoring system will be able to see MBeans and poll for data. You can also run jconsole during development phase to verify your JMX MBean implementation.

Initialization code:
MyService instance = new MyService();
String jmxlabel = “MyService:type=MoneyMakerService”;
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName(jmxlabel);
mbs.registerMBean(instance, name);

One caveat: you can’t pack various counter data (request count, rate, etc) into an object or return them as array: otherwise (as of today) data won’t be processed correctly by some management systems. MXBeans appeared to be an answer as the data can be indeed wrapped into a class and shows up nicely on the monitoring side… but, at least with Manage Engine, the values are treated as strings so you can’t alert on them. You can easily convert to MXBeans later on if need be – but the perf counters classes you build would not have to change!

Web Services

It’s all about web services nowadays. It was very simple to create one in .NET – just run dev studio wizard and off you go. Not so fast in Java. First you should pick a library that takes care of SOAP layer. There are a couple big ones, namely Axis2 and XFire/CXF, both look robust enough.

Axis2 is a complete re-write and in some cases is not compatible with original Axis clients. I still picked Axis2 as it has simple way of wrapping objects into web service calls and doesn’t require tagging them with attributes like CXF. Also, Axis2 allows your object be exposed via REST. Basically, you create a POJO (fancy word for a simple object with setters and getters), update service.xml file to reference this object and you’re done…
Well, there are some caveats:

• You must have an empty constructor.
• Do not overload constructors – it usually works but I’ve ran into scenario where objects were not de-serialized properly and wasted a day chasing down the problem: removing overloaded constructors fixed the issue.
• No helper functions other than setters/getters in your object – or they’ll show up in wsdl and mess things up on the client side.
  

e.g. my entire web service layer now looks like this:
public class MyWS
{
public MyWSResponse SomeRequest(String myarg)
{
MyWSResponse = MyLogicLayer.getInstance().processSomeRequest(myarg);
}
}

MyWSResponse has only an empty constructor and few setters/getters. Add the following to service.xml and you’re done.

<service name="MyWS" scope="application">
<description>
My SOAP Service
</description>
<messageReceivers>
<messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-out"
class="org.apache.axis2.rpc.receivers.RPCMessageReceiver"/>
</messageReceivers>
<parameter name="ServiceClass">com.mycompany.MyWS</parameter>
</service>

Entire logic is encapsulated in the MyLogicLayer – makes a very convenient setup for unittests.

You can run axis2 web service standalone or package it in a war file with other web services or servlets.

Axis2 comes with many jars – I recommend keeping them in a separate folder and package with your code in build time - this way upgrade to newer Axis2 release will be easy.

Finally, you can extend Axis’ AbstractHandler to create a module logging every incoming and outgoing message (which was rather painful to do in .NET). The skeleton code is below:

public InvocationResponse invoke(MessageContext msgContext) throws AxisFault
{
HttpServletRequest request = (HttpServletRequest)msgContext.getProperty("transport.http.servletRequest");
//call request.getHeaderNames() and log headers

//then log message body: msgContext.getEnvelope().toString()

return InvocationResponse.CONTINUE;
}

I also removed axis admin servlet from final deployment – I am a bit paranoid of having a GUI-based access to production environment (or I’ll be tempted to make ad-hoc changes to running system at 3am :-).

Tools

Development environment: use Eclipse. Although some people are raving about latest NetBeans release. Eclipse has more plugins though. Ctrl+Shift+L is your friend -- you'll see a popup listing all keyboard shortcuts.

.NET or Java, you have to have continuous builds, unittests, functional tests, code-analysis scans, sane source control practices in additional to design and code reviews, not to mention proper. I am not trying to sound pompous – it just makes sense after you try it.

Start with good code analysis tools and there are a few descent alternatives to FxCop in Java world. The sooner you set up code checks that easier it’d be to keep it clean.

• PMD is a static analysis tool that examines the source code. I am not crazy about all the rules and lowered “final” keyword priority to 4 but bumped up Cyclomatic Complexity to P1. CPD is part of PMD package and will not let you get away with copy-paste code that so often leads to errors. I don’t recall seeing a warning in FxCop.
  
• Findbugs is another good tool that examines compiled code (so don’t run it next to Code Coverage tool, like Clover). I set it up to run it on release builds only.

Both PMD and Findbugs have Eclipse plugins but don’t be tempted to suppress warnings or your code will end up being cluttered with comments that make no sense when a new tool comes around. Just fix what you need to fix and lower priority for warnings or remove checks altogether for things that you don’t care for (e.g. “variable names are too long” nags from PMD…).

• Code Coverage - Clover.

Continuous builds is the key (we used Bamboo to run our Ant scripts). Continuous builds should run unittests. Unittests without code coverage report don’t count, so setup Clover.

It’s cheap, effective, and will keep you honest especially when you start failing the builds when code coverage is below certain limit (we pick 90% in our team).

Clover plugin is very useful as well; just make sure to place instrumented class files in the project output folder (see screenshot).

Another useful hint for clover is to automatically clover off if (log.isDebugEnabled) and similar statements -- setup a statementContext filter in the Ant script:

<target name="with.clover"> <clover-setup>
<statementContext name="iflog" regexp="if.?\(log\.is.*"/>
<statementContext name="iflog_debug" regexp="if.?\(isDebugEnabled\).*"/>
<statementContext name="iflog_trace" regexp="if.?\(isTraceEnabled\).*"/>
…
</clover-setup>
</target>

<target name="clover.reports">
<clover-report>
<current outfile="${clover.report.html.dir}" title="MyProject"> <format type="html" filter="iflog,iflog_debug,iflog_trace"/>
</current> </clover-report>
<clover-check target="90%" filter="iflog,iflog_debug,iflog_trace" haltOnFailure="true"/> </target>

Notice clover-check with haltOnFailure option in the segment above -- it'll keep you honest!
Another caveat when using clover: while it recognizes code as test when it that contain JUnit attributes, it tends to treat any helper classes that you may need in unittests as real code (even if they’re in the tests folder and you marked tests folder as such in clover-setup). Not a bid deal – just use ///CLOVER:OFF directive and your code coverage will be fixed.

• Code Reviews - Crucible

We are starting to use a fantastic code review tool and one day I'll document the best practices; meanwhile, check out http://www.atlassian.com/software/crucible/screencast.jsp