Tycho/JUnit/Jacoco for the Industrial SQL Connector for Mylyn

In the previous installments I created a repeatable Maven build from Hudson, and I set up static code analysis using SonarQube. It was revealed that there is considerable technical debt, and half of that is caused by 0% Coverage. Getting Test Coverage in place has to be done before tackling any of the other issues like duplication, complexity and violations.

Project Setup

The Project setup for the Industrial SQL Connector for Mylyn is slightly different from other examples found on the internet. The main abstract functionality is handled in the main two plugins:

• com.industrialtsi.mylyn.core for headless stuff and connection handling, and
• com.industrialtsi.mylyn.ui for the UI part.

Functionality for a specific kind of database is then "injected" using a fragment project. Three example projects are included:

• com.industrialtsi.mylyn.demo.memory is a very simple in memory task list for demonstration and testing purposes.
• com.industrialtsi.mylyn.demo.jpa accesses a simple Derby database but using EclipseLink JPA annotations
• com.industrialtsi.mylyn.demo.derby contains a link to a Derby database using the apache ibatis xml based SQL access language, that will allow quite complicated JOIN and UNION statements in its queries.
• org.apache.ibatis finally contains the ibatis stuff packaged in a separate plugin.

All the test code for all these projects finally is placed in a single plugin:

• com.industrialtsi.mylyn.tests contains al test code.

Getting the test code to compile under Tycho

The first step is to add the test project to the main POM in industrialtsi.mylyn.maven. When we run a simple maven build compilation fails:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:0.19.0:compile (default-compile) on project com.industrialtsi.mylyn.tests: Compilation failure: Compilation failure:
[ERROR] /Users/maarten/Workspaces/workspace-industrial-google/com.industrialtsi.mylyn.tests/src/com/industrialtsi/mylyn/demo/derby/test/DerbyIbatisPersistorTest.java:[18]
[ERROR] import com.industrialtsi.mylyn.core.persistence.DerbyIbatisPersistor;
[ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] The import com.industrialtsi.mylyn.core.persistence.DerbyIbatisPersistor cannot be resolved
[ERROR] /Users/maarten/Workspaces/workspace-industrial-google/com.industrialtsi.mylyn.tests/src/com/industrialtsi/mylyn/demo/derby/test/DerbyIbatisPersistorTest.java:[43]
[ERROR] return new DerbyIbatisPersistor();
[ERROR] ^^^^^^^^^^^^^^^^^^^^
[ERROR] DerbyIbatisPersistor cannot be resolved to a type
[ERROR] 2 problems (2 errors)

When compiling in the workspace, eclipse is very helpful in finding all the needed dependencies. As the missing import is located in a fragment of another plugin, normal dependency tricks via the MANIFEST.MF of com.industrialtsi.mylyn.tests or as dependency in the POM file don't work. The quick solution was to include the missing jar as Extra Classpath Entry in the build.properties of com.industrialtsi.mylyn.tests. If anybody knows a better way, please let me know!

Adding tycho-surefire to pom.xml

Next we need to configure the tycho surefire plugin to run the tests and process the results. I'm can rely on the tycho-surefire plugin to load my tests plugin and its direct dependencies. But I want to run a full workbench which I define with the <application> and <product> tags. I also want the com.industrialtsi.mylyn.demo.memory and com.industrialtsi.mylyn.demo.derby fragments to be loaded, which I do with the <dependencies> section. Then I disable the more complicated test till later.

<plugin>
    <groupId>org.eclipse.tycho</groupId>
    <artifactId>tycho-surefire-plugin</artifactId>
    <version>${tycho-version}</version>
    <configuration>
        <argLine>${ui.test.vmargs}</argLine>
        <useUIHarness>true</useUIHarness>
        <useUIThread>true</useUIThread>
        <product>org.eclipse.platform.ide</product>
        <application>org.eclipse.ui.ide.workbench</application>
        <dependencies>
            <dependency>
                <type>eclipse-plugin</type>
                <artifactId>com.industrialtsi.mylyn.demo.memory</artifactId>
                <version>0.9.10</version>
            </dependency>
            <dependency>
                <type>eclipse-plugin</type>
                <artifactId>com.industrialtsi.mylyn.demo.derby</artifactId>
                <version>0.9.10</version>
            </dependency>
        </dependencies>
        <includes>
            <include>**/*Test.java</include>
        </includes>
        <excludes>
            <exclude>**/IbatisPersistorTest.*</exclude>
            <exclude>**/PersistorsManagerTest.*</exclude>
            <exclude>**/DemoDerbyTest.*</exclude>
            <exclude>**/DerbyIbatisPersistorTest.*</exclude>
            <exclude>**/IbatisCorePluginTest.*</exclude>
            <exclude>**/TaskCreationTest.*</exclude>
        </excludes>
        <!-- Kill test JVM if tests take more than 10 minutes (600 seconds)
            to finish -->
        <forkedProcessTimeoutInSeconds>600</forkedProcessTimeoutInSeconds>
    </configuration>
</plugin>

Controlling memory and start Thread on Mac

Running on a Mac requires different startup parameters passed with the <argLine> tag. This is handeld best in the profiles section of the POM:

<profiles>
    <profile>
        <id>macosx</id>
        <activation>
            <os>
                <family>mac</family>
            </os>
        </activation>
        <properties>
            <ui.test.vmargs>-Xmx512m -XX:MaxPermSize=256m -XstartOnFirstThread</ui.test.vmargs>
        </properties>
    </profile>
    <profile>
        <id>other-os</id>
        <activation>
            <os>
                <family>!mac</family>
            </os>
        </activation>
        <properties>
            <ui.test.vmargs>-Xmx512m -XX:MaxPermSize=256m</ui.test.vmargs>
        </properties>
    </profile>
</profiles>

ready to run the tests in the workspace

Then we need to run a Tycho build using the m2e tools. It is important to realize that Plug-in Unit Tests are run in the integration test phase, so we need to specify that as a target.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.industrialtsi.mylyn.core.dto.IndustrialQueryParamsTest
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.039 sec
Running com.industrialtsi.mylyn.test.db.core.GenericQueryParamsTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec

Results :

Tests run: 23, Failures: 0, Errors: 0, Skipped: 0

[INFO] All tests passed!
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] com.industrialtsi.mylyn.maven ..................... SUCCESS [0.073s]
[INFO] org.apache.ibatis ................................. SUCCESS [1.477s]
[INFO] com.industrialtsi.mylyn.core ...................... SUCCESS [1.605s]
[INFO] com.industrialtsi.mylyn.demo.memory ............... SUCCESS [0.386s]
[INFO] com.industrialtsi.mylyn.ui ........................ SUCCESS [0.997s]
[INFO] com.industrialtsi.mylyn.feature ................... SUCCESS [2.616s]
[INFO] com.industrialtsi.mylyn.demo.derby ................ SUCCESS [0.685s]
[INFO] org.apache.ibatis.feature ......................... SUCCESS [2.321s]
[INFO] com.industrialtsi.mylyn.demo.derby.feature ........ SUCCESS [2.329s]
[INFO] com.industrialtsi.mylyn.demo.jpa .................. SUCCESS [1.254s]
[INFO] com.industrialtsi.mylyn.demo.jpa.feature .......... SUCCESS [2.399s]
[INFO] com.industrialtsi.mylyn.site ...................... SUCCESS [3.129s]
[INFO] com.industrialtsi.mylyn.tests ..................... SUCCESS [7.804s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 57.488s
[INFO] Finished at: Tue Jan 21 11:33:31 CET 2014
[INFO] Final Memory: 59M/554M
[INFO] ------------------------------------------------------------------------

Adding Jacoco coverage to the pom.xml

Lets start with adding the Jacoco plugin to the POM file.

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.6.4.201312101107</version>
    <executions>
        <execution>
            <id>prepare-integration-tests</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <include>com.industrialtsi.*</include>
                <include>org.junit.*</include>
                <!-- Where to put jacoco coverage report -->
                <destFile>${sonar.jacoco.reportPath}</destFile>
                <append>true</append>
            </configuration>
        </execution>
    </executions>
</plugin>

Configuring a multi-module plugin project for coverage with Jacoco has some special hurdles which easily lead to frustration. All coverage code must be placed in a single *.exec file to analyzed later, this is handled by the <destFile>${sonar.jacoco.reportPath}</destFile> line. This property is defined in the properties section:

<sonar.jacoco.reportPath>${project.build.directory}/../../com.industrialtsi.mylyn.tests/target/jacoco.exec</sonar.jacoco.reportPath>

Tycho has a special argline variable to pass to the tycho-surefire plugin named tycho.testArgLine that we must pass on in the tycho-surefire plugin:

    <artifactId>tycho-surefire-plugin</artifactId>
    <version>${tycho-version}</version>
    <configuration>
        <argLine>${ui.test.vmargs} ${tycho.testArgLine}</argLine>

Now we can run again and get to see that jacoco.exec is actually generated in the workspace. We can now run the unit tests from the test plugin and measure code coverage with jacoco from a maven build.

not to be forgotten: SonarQube setup

We are running SonarQube from Hudson using the Run Standalone Sonar Analysis build step. We want that analysis to reuse the jacoco.exec file generated during the maven integration-test step. The standalone step uses sonar-project.properties for configuration. So we add:

sonar.dynamicAnalysis=reuseReports
sonar.jacoco.reportPath=../com.industrialtsi.mylyn.tests/target/jacoco.exec

If we had run SonarQube from maven using the sonar:sonar or verify targets, we should configure sonar in the pom.xml.

<properties>
    ...
    <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
    <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
    ...
</properties>

I do both know that I'm working on it so I have flexibility later.

Now commit to SCM and start a Hudson build

Start a Hudson build manually or wait for SCM polling to kick in, but the result looks promising: 3.63% Coverage is not much, but easy to expand now that this works.

Finally: results in SonarQube

I managed to reduce Technical Debt with $ 390 and one man day compared to the start of this blog post. Seems like a low yield for a half day work, but I guess the calculations do not allow for the ramp up cost: running the first coverage is hardest. Progress should be easier now that the build and static analysis infrastructure is in place. Unit Test Coverage is up from 0% with only the two simplest tests executing. Next steps are activating the tests disabled earlier and adding more test. To be continued...

Analyzing the Industrial SQL Connector for Mylyn with SonarQube

Last year we set up Hudson to build the Industrial SQL Connector for Mylyn, a DIY connector project to connect Mylyn to a local SQL database for which I'm a committer. This blog post I will explain how I set up static code analysis on the same project using SonarQube. Installing and setting up SonarQube is better explained elsewhere, like http://www.sonarqube.org, but then setting up a set op eclipse plugin projects to be analyzed is more specific.

Preparation:

• Install SonarQube following instructions here.
• Then install the Sonar plugin into Hudson using Update Center and configure it following these instructions
• Lookup how to build the Industrial SQL Connector for Mylyn with Hudson here

Configuring the Industrial SQL Connector for Mylyn for analysis

Whether you want to analyze projects with the Maven sonar:sonar target or use the Hudson Invoke Standalone Sonar Analysis build step in both cases you need to create a sonar-project.properties file. As a matter of common sense I always put this file in the same project as the project with the maven master POM, in this case com.industrialtsi.mylyn.maven.

# required metadata
sonar.projectKey=com.industrialtsi.mylyn
sonar.projectName=Industrial SQL Connector for Mylyn
sonar.projectVersion=0.9.10-SNAPSHOT

# optional description
sonar.projectDescription=Industrial SQL Connector for Mylyn

# path to source directories (required)
#sonar.sources=src THIS IS SPECIFIED PER MODULE

# path to project binaries (optional), for example directory of Java bytecode
#sonar.binaries=target/classes THIS IS SPECIFIED PER MODULE

# optional comma-separated list of paths to libraries. Only path to JAR file is supported.
#sonar.libraries=lib/*.jar THIS IS SPECIFIED PER MODULE

# The value of the property must be the key of the language.
sonar.language=java

# modules one for each area of functionality, only plugin and fragment projects
sonar.modules=core,derby,jpa,memory,ui

# setup project base dir
core.sonar.projectBaseDir=com.industrialtsi.mylyn.core
derby.sonar.projectBaseDir=com.industrialtsi.mylyn.demo.derby
jpa.sonar.projectBaseDir=com.industrialtsi.mylyn.demo.jpa
memory.sonar.projectBaseDir=com.industrialtsi.mylyn.demo.memory
ui.sonar.projectBaseDir=com.industrialtsi.mylyn.ui

#set up source folders 
core.sonar.sources=src
derby.sonar.sources=src
jpa.sonar.sources=src
memory.sonar.sources=src
ui.sonar.sources=src

# set up binary folders
core.sonar.binaries=target/classes
derby.sonar.binaries=target/classes
jpa.sonar.binaries=target/classes
memory.sonar.binaries=target/classes
ui.sonar.binaries=target/classes

# set up libraries folders, where jars reside
core.sonar.libraries=
derby.sonar.libraries=lib/*.jar
jpa.sonar.libraries=lib/*.jar
memory.sonar.libraries=
ui.sonar.libraries=

Then we commit this file to the repository so Hudson can retrieve it.

Configuring Hudson to analyze the project

We go to the Hudson Job tab and press configure. Under Build we add a step Invoke Standalone Sonar Analysis and configure it as follows:

Building and examining the results

We make Hudson build the project and then go over to the SonarQube pages for the results, note that I have created a custom set of my favorite widgets for this: The Technical Debt widget tell us the technical debt in days, and also a percentage split of the main problems. Lack of coverage explains half the debt, with design, comments and complexity as other issues. There are very few violations and duplications, the happy result of running with Checkstyle, Findbugs and PMD inside Eclipse during the development. I always add a Most Violated Rules widget to the project dashboard. Exposing internal representation is the most common here as the very bad package cycles. The "Most Violated Resources" widget instantly tells me that the objects used to ferry query parameters around are the main problem area. I have a tagged sonar-project.properties file so I will be able to see progress in the future on lines of code, technical debt and documented API. Issues are mostly Critical and Major, so need to be fixed urgently. 1% duplications is not that serious, even though 0% is best of course. Test coverage is more serious as we've seen above that lack of Test Coverage accounts for half of technical debt. This maybe because the build is not yet configured to execute tests. The complexity stats show mainly whether design is good, a method should do one thing, a class should have one responsibility. The LCOM4 measure of 1.0/class is quite positive. The main problem in these is the startling 40.5% of package tangle index and more than 10 cycles! This needs to be looked at with highest priority.

Action Plan

So this short exercise (total time to setup 1,5 hours) revealed quite a lot of potential problems in the code base! How then to tackle and resolve these problems?

1. Ensure that the unit tests are executed and measured! Without unit tests we cannot begin to refactor safely.
2. Fix the Critical Issues in the code, but only when adequately covered by unit tests
3. Investigate and fix the Package Cycles problems, but again only when adequately covered by unit tests
4. Fix the Major Issues in the code, but only when adequately covered by unit tests
5. Fix the Code Duplications problems

I will report on my findings here. For unit test coverage I'm going to use Jacoco, which works well inside Eclipse using the Eclemma plugin, is preferred by SonarQube and can also be integrated with Tycho/Maven.