(Part 3 of 3): Synopsis of articles & videos on Performance tuning, JVM, GC in Java, Mechanical Sympathy, et al

This is a continuation of the previous post titled (Part 2 of 3): Synopsis of articles & videos on Performance tuning, JVM, GC in Java, Mechanical Sympathy, et al.

In our first review, The Atlassian guide to GC tuning is an extensive post covering the methodology and things to keep in mind when tuning GC, practical examples are given and references to important resources are also made in the process. The next one How NOT to measure latency by Gil Tene, he discusses some common pitfalls encountered in measuring and characterizing latency, demonstrating and discussing some false assumptions and measurement techniques that lead to dramatically incorrect reporting results, and covers simple ways to sanity check and correct these situations.  Finally Kirk Pepperdine in his post Poorly chosen Java HotSpot Garbage Collection Flags and how to fix them! throws light on some JVM flags – he starts with some 700 flags and boils it down to merely 7 flags. Also cautions you to not just draw conclusions or to take action in a whim but consult and examine – i.e. measure don’t guess!

….read more (reblogged from the Java Advent Calendar)

(Part 2 of 3): Synopsis of articles & videos on Performance tuning, JVM, GC in Java, Mechanical Sympathy, et al

This is a continuation of the previous post titled (Part 1 of 3): Synopsis of articles & videos on Performance tuning, JVM, GC in Java, Mechanical Sympathy, et al.

Without any further ado, lets get started with our next set of blogs and videos, chop…chop…! This time its Martin Thompson’s blog posts and talks. Martin’s first post on Java Garbage collection distilled basically distils the GC process and the underlying components including throwing light on a number of interesting GC flags (-XX:…). In his next talk he does his myth busting shaabang about mechanical sympathy, what people correctly believe in and the misconceptions. In the talk on performance testing, Martin takes its further and fuses Java, OS and the hardware to show how understanding of all these aspects can help write better programs.


Java Garbage Collection Distilled by Martin Thompson

There are too many flags to allow tuning the GC to achieve the throughput and latency your application requires. There’s plenty of documentation on the specifics of the bells and whistles around them but none to guide you through them.

Using SonarQube ™ (formerly Sonar ™) installed on the Mac OS X Mountain Lion 10.8.4

Introduction (contd.)

Continuing from where we left in our previous blog post on Installing SonarQube™ (formerly Sonar™) on Mac OS X Mountain Lion 10.8.4 [01], we will cover how to use SonarQube given different situations.

This post might come across a bit more verbose than the previous one, namely with outputs of commands and screenshots illustrating how SonarQube responds to various user actions.

Running SonarQube to analyse projects

We will cover the two ways SonarQube can be used to analyse a project (written in one of the SonarQube supported programming languages [02]) either via maven or through sonar-runner (for non-maven projects) and also the different aspects of SonarQube which help as a static code analysis tool.
via maven
Go to the project folder containing the maven configuration file i.e. pom.xml and run one of the below commands depending on the end goal:

$ mvn clean install sonar:sonar
$ mvn install sonar:sonar
$ mvn sonar:sonar

$ mvn clean sonar:sonar -Dsonar.host.url=http://localhost:nnnn

(where nnnn is an alternative port number where SonarQube is listening)

outputs

Successful analysis of the project via the above commands would lead to the below output onto the console or log files:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29.923s
[INFO] Finished at: Fri Sep 13 18:07:01 BST 2013
[INFO] Final Memory: 62M/247M
[INFO] ------------------------------------------------------------------------
[INFO] [18:07:01.557] Execute org.apache.maven.plugins:maven-surefire-plugin:2.10:test done: 20372 ms
[INFO] [18:07:01.557] Execute maven plugin maven-surefire-plugin done: 20373 ms
.
.
.
[INFO] [18:07:09.526] ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard/index/com.webapplication:sub-webapp
[INFO] [18:07:09.528] Executing post-job class org.sonar.issuesreport.ReportJob
[INFO] [18:07:09.529] Executing post-job class org.sonar.plugins.core.issue.notification.SendIssueNotificationsPostJob
[INFO] [18:07:09.529] Executing post-job class org.sonar.plugins.core.batch.IndexProjectPostJob
[INFO] [18:07:09.580] Executing post-job class org.sonar.plugins.dbcleaner.ProjectPurgePostJob
[INFO] [18:07:09.590] -> Keep one snapshot per day between 2013-08-16 and 2013-09-12
[INFO] [18:07:09.591] -> Keep one snapshot per week between 2012-09-14 and 2013-08-16
[INFO] [18:07:09.591] [INFO] [18:07:09.614]  Keep one snapshot per month between 2008-09-19 and 2012-09-14
[INFO] [18:07:09.627] -> Delete data prior to: 2008-09-19
[INFO] [18:07:09.629] -> Clean webapp [id=1]
[INFO] [18:07:09.631] [INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 38.345s
[INFO] Finished at: Fri Sep 13 18:07:09 BST 2013
[INFO] Final Memory: 28M/255M
[INFO] ------------------------------------------------------------------------
Here are a couple of links to sample pom.xml files which should help with creating new or amend existing configurations to integrate maven projects with SonarQube (including additional maven CLI switches) i.e. Analyzing with Maven [03] and SonarQube examples on Github [04].

via sonar-runner

Go to the project folder containing the sonar-project.properties configuration file and run the below command:

</div>
<div></div>
<div>$ sonar-runner</div>
<div></div>
<div>

outputs

Successful analysis of the project via the above command would lead to the below output onto the console or log files:
SonarQube Runner 2.3
Java 1.7.0_25 Oracle Corporation (64-bit)
Mac OS X 10.8.5 x86_64
INFO: Runner configuration file: /opt/sonar-runner-2.3/conf/sonar-runner.properties
INFO: Project configuration file: /Users/manisarkar/bn_projects/TimelineJS/sonar-project.properties
INFO: Default locale: &quot;en_US&quot;, source code encoding: &quot;UTF-8&quot;
INFO: Work directory: /Users/manisarkar/bn_projects/TimelineJS/.sonar
INFO: SonarQube Server 3.7
14:11:20.927 INFO - Load batch settings
.
.
.
14:11:38.290 INFO - ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard/index/TimelineJS
14:11:38.292 INFO - Executing post-job class org.sonar.issuesreport.ReportJob
14:11:38.293 INFO - Executing post-job class org.sonar.plugins.core.issue.notification.SendIssueNotificationsPostJob
14:11:38.314 INFO - Executing post-job class org.sonar.plugins.core.batch.IndexProjectPostJob
14:11:38.356 INFO - Executing post-job class org.sonar.plugins.dbcleaner.ProjectPurgePostJob
14:11:38.365 INFO - -&gt; Keep one snapshot per day between 2013-08-19 and 2013-09-15
14:11:38.365 INFO - -&gt; Keep one snapshot per week between 2012-09-17 and 2013-08-19
14:11:38.365 INFO - -&gt; Keep one snapshot per month between 2008-09-22 and 2012-09-17
14:11:38.365 INFO - -&gt; Delete data prior to: 2008-09-22
14:11:38.368 INFO - -&gt; Clean TimelineJS [id=151]
14:11:38.372 INFO - INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
Total time: 19.099s
Final Memory: 14M/502M
INFO: ------------------------------------------------------------------------
Here are a couple of links to sample sonar-project.properties files to assisting in creating new ones i.e. Sonar setup for non-maven Java project [05] and Analyzing with SonarQube Runner [06].

Note: SonarQube Runner expects SonarQube to be running on the designated port, otherwise throws errors like i.e. ERROR: Sonar server http://localhost:9000 can not be reached. This of course can be changed via configuration files (see previous post [01]).

SonarQube Components

Once the build is complete and successful, the new or updated project can be found in the Dashboard. Drilling down into the project would bring up a screen like which is loaded with important metrics and analysis on various aspects of the project:

(above is a screen-shot of a sample application)

The prime important components that is of interest are Quality Index, Complexity Factor, Complexity (bottom left), Test coverage metrics (Unit Tests Coverage and Unit test success). Possibly Security violations. Package Tangle Index and Dependencies to cut, are definitely handy in order to keep clean packages and loosely coupled dependencies. On the same note, LCOM4 (Lack of Cohesion in Methods – lower the value the better)  and Complexity also throws light on how loosely coupled your classes, methods and functions are – it is also measured at the file level and an overall-level giving the full picture. All of these components are good indicators of Software Quality at the least if not Software Craftsmanship – how well is the underlying code written with care for quality in mind! Or it could be seen as-there-is-still-plenty-of-room-for-improvement-and-refactoring.

The Hotspot view now drills further into a few other important aspects of the analysis and highlights the areas that needs more attention or one ore more issue is near its culmination point – either have cross the maximum allowed limit or needs a bit more polishing to meet the requirements.

 

(above is a screenshot of a JDK7 as published on the nemo.sonarqube.org site)

I quite like the below Design component which gives a good break-down of the package dependencies across each other and highlights dependency-cycles. Its one of the more complex things to do on a medium to large project and can usually come in the way of modularisation.


(above is a screenshot of a sample application)

If you ever wanted to know the internal or external libraries a project uses, you might need to look at the contents of your project including the pom.xml file. No longer the case if you are using SonarQube, as Library is such a component that enlists components that your application depends on and is more reliable than searching for it manually.

(above is a screenshot of a sample application)

It is also possible to add any Widget on any Dashboard (Widgets are components that make up a Dashboard), like the one presented below.

(above is a screenshot of a sample application)

Issues Drilldown
Just being told that something is wrong and here’s the score on how much wrong or incorrect something is, does not help. A more constructive feedback is, here’s what’s wrong and this is what you can do to fix it.
The Issues Drilldown is one such dashboard where we can find such information or enough to know what’s wrong and where to go and how to fix it (sometimes). It also archives older and closed issues, and indicates how bad a problem it is by giving it various gradations of severity i.e. Blocker to Info.

(above is a screenshot of a sample application on nemo.sonarsource.org)

The Manage dashboards option at the top right corner of any of the Dashboard pages (as below) is used to create new dashboard pages into which widgets can be placed.
Similarly the Configure widgets link on every Dashboard page allows adding, removing or changing the position of the widgets anywhere on the Dashboard page.

(above is a screenshot of the Apache Commons Collection)

Tag or word Clouds is a very popular concept heavily used as a form of visualisation to convey metrics – as shown above, which is an illustration of the Apache Commons Collection library.

Commercial component – SQALE

SQALE is a proprietary component and not available in the community version, although a demo version is available via SonarQube’s Nemo project [07]. SQALE is a technical debt evaluation tool, more details of it can be found at [08].

(above is a screenshot of Apache Commons Collections)


Settings

Under the hood, this SonarQube instance relies on a number of default or customised configuration settings laid out as below.

(above is a screenshot of a sample application)

Configuration settings to individual components can be accessed and changed via this interface.



Update Centre

Many widgets that are populated in the various Dashboards seen so far can be enabled or disabled from the below page. It is also where updates and upgrades for all the widgets are available, including updates and upgrades for SonarQube itself.


(above is a screenshot of a sample application)

Upgrade process

Check out the upgrade process from [10], see also [11] to learn what should be done before and after the process.

Usually stopping and restarting SonarQube are common steps performed before and after applying an update or upgrade to one or more components or to SonarQube itself.


Conclusion

After assessing these features its clear that this product has a number of advantages over other solutions out there i.e. lots of free plugins, a plug-in based dashboard system, besides it being an open-source project, and a very good one to get started with. Having said that, there might be commercial products out there that have better quality assessment propositions but not necessarily useful unless yours is a big organisation.
Use SonarQube as a tool to create short-feedback loops, and apply improvements to your code base after assessing the rationale for the change suggested. In case the feedback is incorrect or is a false positive or false negative – one option would be to tweak the configuration settings behind the relevant component to see if the issue raised is applicable under a current conditions – basically either turning of the check or not taking the feedback literally.
Disclaimer: please do not take any of these metrics or the contents of the blog literally but rather as another point of view of  the quality of your code-base. Its important to know how to read the data and not read too much into the numbers and flags raised – there may still be false positives or false negatives. Some commercial products have taken this seriously and have invested to reduce the false positives and false negatives.

External resources

The below links have been used during the installation of SonarQube, and have been mentioned throughout the blog.

Notes

The terms Sonar and SonarQube have been used interchangeably in a number of places above. Some of it is due to the referred links not being updated, and others are due to the fact that scripts and program references have continued to be used with their original names to prevent issues with dependencies.

Do not take the settings, paths and file locations, url references, excetra mentioned in this blog literally, in some cases they would need to be adjusted to settings relevant to your environment.

Please note all the external links on this blog may or may not stay actual and is not feasible to be maintained as part of this blog post.

Please feel free to contribute to the above post in the form of constructive comments, useful links, additional information, excetra to improve the quality of the information provided. If something hasn’t worked for you and you have managed to make it work or have a work-around / alternative solution, please do share it with us!

(Part 1 of 3): Synopsis of articles & videos on Performance tuning, JVM, GC in Java, Mechanical Sympathy, et al

I have been contemplating for a number of months about reviewing a cache of articles and videos on topics like Performance tuning, JVM, GC in Java, Mechanical Sympathy, etc… and finally took the time to do it – may be this was the point in my intellectual progress when was I required to do such a thing!

Thanks to Attila-Mihaly for giving me the opportunity to write a post for his yearly newsletter Java Advent Calendar, hence a review on various Java related topics fits the bill! The selection of videos and articles are purely random, and based on the order in which they came to my knowledge. My hidden agenda is to mainly go through them to understand and broaden my own knowledge at the same time share any insight with others along the way….read more (reblogged from the Java Advent Calendar)