Building Wholly Graal with Truffle!

feature-image-building-graal-and-truffle

Citation: credits to the feature image go to David Luders and reused under a CC license, the original image can be found on this Flickr page.

Introduction

It has been some time, since the two posts [1][2] on Graal/GraalVM/Truffle, and a general request was when are you going to write something about “how to build” this awesome thing called Graal. Technically, we will be building HotSpot’s C2 compiler (look for C2 in the glossary list) replacement, called Graal. This binary is different from the  GraalVM suite you download from OTN via http://graalvm.org/downloads.

I wasn’t just going to stop at the first couple of posts on this technology. In fact, one of the best ways to learn and get an in-depth idea about any tech work, is to know how to build it.

Getting Started

Building JVMCI for JDK8, Graal and Truffle is fairly simple, and the instructions are available on the graal repo. We will be running them on both the local (Linux, MacOS) and container (Docker) environments. To capture the process as-code, they have been written in bash, see https://github.com/neomatrix369/awesome-graal/tree/master/build/x86_64/linux_macos.

During the process of writing the scripts and testing them on various environments, there were some issues, but these were soon resolved with the help members of the Graal team — thanks Doug.

Running scripts

Documentation on how to run the scripts are provided in the README.md on awesome-graal. For each of the build environments they are merely a single command:

Linux & MacOS

$ ./local-build.sh

$ RUN_TESTS=false           ./local-build.sh

$ OUTPUT_DIR=/another/path/ ./local-build.sh

Docker

$ ./docker-build.sh

$ DEBUG=true                ./docker-build.sh

$ RUN_TESTS=false           ./docker-build.sh

$ OUTPUT_DIR=/another/path/ ./docker-build.sh

Both the local and docker scripts pass in the environment variables i.e. RUN_TESTS and OUTPUT_DIR to the underlying commands. Debugging the docker container is also possible by setting the DEBUG environment variable.

For a better understanding of how they work, best to refer to the local and docker scripts in the repo.

Build logs

I have provided build logs for the respective environments in the build/x86_64/linux_macos  folder in https://github.com/neomatrix369/awesome-graal/

Once the build is completed successfully, the following messages are shown:

[snipped]

>> Creating /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal from /path/to/awesome-graal/build/x86_64/linux/graal-jvmci-8/jdk1.8.0_144/linux-amd64/product
Copying /path/to/awesome-graal/build/x86_64/linux/graal/compiler/mxbuild/dists/graal.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/jvmci
Copying /path/to/awesome-graal/build/x86_64/linux/graal/compiler/mxbuild/dists/graal-management.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/jvmci
Copying /path/to/awesome-graal/build/x86_64/linux/graal/sdk/mxbuild/dists/graal-sdk.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/boot
Copying /path/to/awesome-graal/build/x86_64/linux/graal/truffle/mxbuild/dists/truffle-api.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/truffle

>>> All good, now pick your JDK from /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal :-)

Creating Archive and SHA of the newly JDK8 with Graal & Truffle at /home/graal/jdk8-with-graal
Creating Archive jdk8-with-graal.tar.gz
Creating a sha5 hash from jdk8-with-graal.tar.gz
jdk8-with-graal.tar.gz and jdk8-with-graal.tar.gz.sha256sum.txt have been successfully created in the /home/graal/output folder.

Artifacts

All the Graal and Truffle artifacts are created in the graal/compiler/mxbuild/dists/ folder and copied to the newly built jdk8-with-graal folder, both of these will be present in the folder where the build.sh script resides:

jdk8-with-graal/jre/lib/jvmci/graal.jar
jdk8-with-graal/jre/lib/jvmci/graal-management.jar
jdk8-with-graal/jre/lib/boot/graal-sdk.jar
jdk8-with-graal/jre/lib/truffle/truffle-api.jar

In short, we started off with vanilla JDK8 (JAVA_HOME) and via the build script created an enhanced JDK8 with Graal and Truffle embedded in it. At the end of a successful build process, the script will create a .tar.gz archive file in the jdk8-with-graal-local folder, alongside this file you will also find the sha5 hash of the archive.

In case of a Docker build, the same folder is called jdk8-with-graal-docker and in addition to the above mentioned files, it will also contain the build logs.

Running unit tests

Running unit tests is a simple command:

mx --java-home /path/to/jdk8 unittest

This step should follow the moment we have a successfully built artifact in the jdk8-with-graal-local folder. The below messages indicate a successful run of the unit tests:

>>>> Running unit tests...
Warning: 1049 classes in /home/graal/mx/mxbuild/dists/mx-micro-benchmarks.jar skipped as their class file version is not supported by FindClassesByAnnotatedMethods
Warning: 401 classes in /home/graal/mx/mxbuild/dists/mx-jacoco-report.jar skipped as their class file version is not supported by FindClassesByAnnotatedMethods
WARNING: Unsupported class files listed in /home/graal/graal-jvmci-8/mxbuild/unittest/mx-micro-benchmarks.jar.jdk1.8.excludedclasses
WARNING: Unsupported class files listed in /home/graal/graal-jvmci-8/mxbuild/unittest/mx-jacoco-report.jar.jdk1.8.excludedclasses
MxJUnitCore
JUnit version 4.12
............................................................................................
Time: 5.334

OK (92 tests)

JDK differences

So what have we got that’s different from the JDK we started with. If we compare the boot JDK with the final JDK here are the differences:

Combination of diff between $JAVA_HOME and jdk8-with-graal and meld will give the above:

JDKversusGraalJDKDiff-02

JDKversusGraalJDKDiff-01

diff -y --suppress-common-lines $JAVA_HOME jdk8-with-graal | less
meld $JAVA_HOME ./jdk8-with-graal

Note: $JAVA_HOME points to your JDK8 boot JDK.

Build execution time

The build execution time was captured on both Linux and MacOS and there was a small difference between running tests and not running tests:

Running the build with or without tests on a quad-core, with hyper-threading:

 real 4m4.390s
 user 15m40.900s
 sys 1m20.386s
 ^^^^^^^^^^^^^^^
 user + sys = 17m1.286s (17 minutes 1.286 second)

Similar running the build with and without tests on a dual-core MacOS, with 4GB RAM, SSD drive, differs little:

 real 9m58.106s
 user 18m54.698s 
 sys 2m31.142s
 ^^^^^^^^^^^^^^^ 
 user + sys = 21m25.84s (21 minutes 25.84 seconds)

Disclaimer: these measurements can certainly vary across the different environments and configurations. If you have a more accurate way to benchmark such running processes, please do share back.

Summary

In this post, we saw how we can build Graal and Truffle for JDK8 on both local and container environments.

The next thing we will do is build them on a build farm provided by Adopt OpenJDK. We will be able to run them across multiple platforms and operating systems, including building inside docker containers. This binary is different from the GraalVM suite you download from OTN via http://graalvm.org/downloads, hopefully we will be able to cover GraalVM in a future post.

Thanks to Julien Ponge for making his build script available for re-use and the Graal team for supporting during the writing of this post.

Feel free to share your feedback at @theNeomatrix369. Pull requests with improvements and best-practices are welcome at https://github.com/neomatrix369/awesome-graal.

Advertisements

Truffle served in a Holy Graal: Graal and Truffle for polyglot language interpretation on the JVM

03 Hotspot versus GraalVM

Reblogging from ZeroTurnaround’s Rebellabs blog site

One of the most fascinating additions to Java 9 is the JVMCI: Java-Level JVM Compiler Interface, a Java based compiler interface which allows us to plug in a dynamic compiler into the JVM. One of the main inspirations for including it into Java 9 was due to project Graal — a dynamic state-of-the-art compiler written in Java.

In this post we look at the reasons Graal is such a fascinating project, its advantages, what are the general code optimization ideas, some performance comparisons, and why would you even bother with tinkering with a new compiler.

Like everyone else we were inspired by the vJUG session by Chris Seaton on Graal – it looks like a great tool and technology and so we decided to play with the technology and share it with the community.

…you can read the rest at ZeroTurnaround’s Rebellabs blogs


 

In case, you are wondering what some of the ASCII-art images in one of the paragraphs is about, here’s a bit of explanation, hopefully it will clear up any doubts.

How does it actually work?

A typical flow would look like this:

02-a Program to machine code diagram (excludes expansion)
AST → Abstract Syntax Tree  (explicit data structures in memory)

We all know that a JIT is embedded inside HotSpot or the JVM. It’s old, complicated, written in C++ and assembly and is fairly hard to understand. It is a black box and there is no way to hook or link into the JIT.  All the JVM languages have to go through the same route:  

02-b Program to machine code diagram (via byte-code)

(ASM = assembly)

The flow or route when dealing with traditional compilers and VM would be:

02-c Program to machine code diagram (via JIT)
But with Graal, we get the below route or flow:

02-d Program to machine code diagram (via AST)
(notice Graal skips the steps that create byte-code by directly generating platform specific machine code)

Graal basically helps moving the control-flow from Code to the JIT bypassing the JVM (HotSpot, in our case). It means we will be running faster and more performant applications, on the JVM. These applications will not be interpreted anymore but compiled to machine code on fly or even natively.


I hope you enjoyed the read, please feel free to share any constructive feedback, so we can improve the material for the community as a whole. We learnt a lot while drafting this post and hope the same for you.

Original post by @theNeomatrix369 and  @shelajev !

Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2

…continuing from Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2.

I ran into a number of issues and you can see from my commits, I pulled myself out of it, but to run this Dockerfile from the command-line I used this instruction:

$ docker build -t neomatrix369/openjdk9 .

you can also do it using the below if you have not set your permissions:

$ sudo docker build -t neomatrix369/openjdk9 .

and get the below (summarised) output:

Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon 
Step 0 : FROM phusion/baseimage:latest
 ---> 5a14c1498ff4
Step 1 : MAINTAINER Mani Sarkar (from @adoptopenjdk)
 ---> Using cache
 ---> 95e30b7f52b9
Step 2 : RUN apt-get update &&   apt-get install -y     libxt-dev zip pkg-config libX11-dev libxext-dev     libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev &&   rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 1ea3bbb15c2d
Step 3 : RUN apt-get update
 ---> Using cache
 ---> 6c3938f4d23d
Step 4 : RUN apt-get install -y mercurial ca-certificates-java build-essential
 ---> Using cache
 ---> e3f99b5a3bd3
Step 5 : RUN cd /tmp &&   hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 &&   cd openjdk9 &&   sh ./get_source.sh
 ---> Using cache
 ---> 26cfaf16b9fa
Step 6 : RUN apt-get install -y wget &&   wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
 ---> Using cache
 ---> 696889250fed
Step 7 : RUN tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt
 ---> Using cache
 ---> c25cc9201c1b
Step 8 : RUN cd /tmp/openjdk9 &&   bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45
 ---> Using cache
 ---> 4e425de379e6
Step 9 : RUN cd /tmp/openjdk9 &&   make clean images
 ---> Using cache
 ---> 2d9e17c870be
Step 10 : RUN cd /tmp/openjdk9 &&   cp -a build/linux-x86_64-normal-server-release/images/jdk     /opt/openjdk9
 ---> Using cache
 ---> 9250fac9b500
Step 11 : RUN cd /tmp/openjdk9 &&   find /opt/openjdk9 -type f -exec chmod a+r {} + &&   find /opt/openjdk9 -type d -exec chmod a+rx {} +
 ---> Using cache
 ---> d0c597d045d4
Step 12 : ENV PATH /opt/openjdk9/bin:$PATH
 ---> Using cache
 ---> 3965c3e47855
Step 13 : ENV JAVA_HOME /opt/openjdk9
 ---> Using cache
 ---> 5877e8efd939
Successfully built 5877e8efd939

The above action creates an image which is stored in your local repository (use docker images to enlist the images in the repo). If you want to load the image into a container, and access the files it has built or see anything else, do the below:

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 /bin/bash

this will take you to a bash prompt into the container and you can run any of your linux commands and access the file system.

Explaining docker run

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 java -version

will show you this

openjdk version "1.9.0-internal"
OpenJDK Runtime Environment (build 1.9.0-internal-_2015_06_04_06_46-b00)
OpenJDK 64-Bit Server VM (build 1.9.0-internal-_2015_06_04_06_46-b00, mixed mode)

Here’s a breakdown of the docker run command:

docker run The command to create and start a new Docker container.
-it To run in interactive mode, so you can see the after running the container.
neomatrix369/openjdk9 This is a reference to the image tag by name (which we created above).
java -version Runs the java command asking its version, inside the containing, which is assisted by the two environment variables PATH and JAVA_HOME which was set in the Dockerfile above.

Footnotes

You might have noticed I grouped very specific instructions with each step, especially the RUN commands, its because, each time I got one of these wrong, it would re-execute the step again, including the steps that ran fine and didn’t need re-executing. Not only is this unnecessary its not using our resources efficiently which is what Docker brings us. So any addition, edition or deletion to any step will only result in that step being executed, and not the other steps that are fine.

So one of the best practises is to keep the steps granular enough and pre-load files and data beforehand and give it to docker. It has amazing caching and archiving mechanisms built in.

Save our work

As we know if we do not save the container into the image, our changes are lost.

If I didn’t use the docker build command I used earlier I could have, after the build process was completed and image created, used the below command:

$ sudo docker commit [sha of the image] neomatrix369/openjdk9

Sharing your docker image on Docker hub

Once you are happy with your changes, and want to share it with community at large, do the below:

$ sudo docker push neomatrix369/openjdk9

and you will see these depending on which of your layers have been found in the repo and which ones are new (this one is an example snapshot of the process):

The push refers to a repository [neomatrix369/openjdk9] (len: 1)
5877e8efd939: Image already exists 
3965c3e47855: Image already exists 
d0c597d045d4: Image already exists 
9250fac9b500: Image already exists 
2d9e17c870be: Buffering to Disk
.
.
.

There is plenty of room for development and improvement of this Docker script. So happy hacking and would love to hear your feedback or contributions from you.

BIG Thanks

Big thanks to the below two who proof-read my post and added value to it, whilst enjoying the #Software #Craftsmanship developer community (organised and supported by @LSCC):
Oliver Nautsch – @ollispieps (JUG Switzerland)
Amir Bazazi (@Codurance) – @amirbazazi

Special thanks to Roberto Cortez (@radcortez) for your Docker posts, these inspired and helped me write my first Docker post.

Resources

[1] Docker
[2] Get into Docker – A Guide for Total Newbies
[3] Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images
[4] Docker posts on Voxxed
[5] OpenJDK
[6] Building OpenJDK
[7] Building OpenJDK on Linux, MacOs and Windows
[8] Virtual Machines (OpenJDK)
[9] Build your own OpenJDK
[10] Vagrant script (OpenJDK)
[11] YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it
[12] Dockerfile on github
[13] Adopt OpenJDK: Getting Started Kit
[14] London Java Community

Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2

Introduction

I think I have joined the Docker [1] party a bit late but that means by now everyone knows what Docker is and all the other basic fundamentals which I can very well skip, but if you are still interested, please check these posts Get into Docker – A Guide for Total Newbies [2] and Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images [3]. And if you still want to know more about this widely spoken topic, check out these Docker posts on Voxxed [4].

Why ?

Since everyone has been doing some sort of provisioning or spinning up of dev or pre-prod or test environments using Docker [1] I decided to do the same but with my favourite project i.e. OpenJDK [5].

So far you can natively build OpenJDK [6] across Linux, MacOs and Windows [7], or do the same things via virtual machines or vagrant instances, see more on then via these resources Virtual Machines, [8] Build your own OpenJDK [9] and this vagrant script [10]. All part of the Adopt OpenJDK initiative lead by London Java Community [14] and supported by JUGs all over the world.

Requirements

Most parts of post is for those using Linux distributions (this one was created on Ubuntu 14.04). Linux, MacOS and Windows users please refer to Docker‘s  Linux, MacOS and Windows instructions respectively.

Hints: MacOS and Windows users will need to install Boot2Docker and remember to run the below two commands (and check your Docker host environment variables):

$ boot2docker init
$ boot2docker up 
$ boot2docker shellinit 

For the MacOS, if the above throw FATA[…] error messages, please try the below:

$ sudo boot2docker init
$ sudo boot2docker up 
$ sudo boot2docker shellinit 

For rest of the details please refer to the links provided above. Once you have the above in place for the Windows or MacOS platform, by merely executing the Dockerfile using the docker build and docker run commands you can create / update a container and run it respectively.

*** Please refer to the above links and ensure Docker works for you for the above platforms – try out tutorials or steps proving that Docker run as expected before proceeding further. ***

Building OpenJDK 9 using Docker

Now I will show you how to do the same things as mentioned above using Docker.

So I read the first two resource I shared so far (and wrote the last ones). So lets get started, and I’m going to walk you through what the Dockerfile looks like, as I take you through each section of the Dockerfile code.

*** Please note the steps below are not meant to be executed on your command prompty, they form an integral part of the Dockerfile which you can download from here at the end of this post. ***

You have noticed unlike everyone else I have chosen a different OS image i.e. phusion/baseimage, why? Read YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it [11], to learn more about it.

FROM phusion/baseimage:latest

Each of the RUN steps below when executed becomes a Docker layer in isolation and gets assigned a SHA like this i.e. 95e30b7f52b9.

RUN \
  apt-get update && \
  apt-get install -y \
    libxt-dev zip pkg-config libX11-dev libxext-dev \
    libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev && \
  rm -rf /var/lib/apt/lists/*

The base image is updated and a number of dependencies are installed i.e. Mercurial (hg) and build-essential.

RUN \
  apt-get update && \
  apt-get install -y mercurial ca-certificates-java build-essential

Clone the OpenJDK 9 sources and download the latest sources from mercurial. You will notice that each of these steps are prefixed by this line cd /tmp &&, this is because each instruction is run in its own layer, as if it does not remember where it was when the previous instruction was run. Nothing to worry about, all your changes are still intact in the container.

RUN \
  cd /tmp && \
  hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 && \
  cd openjdk9 && \
  sh ./get_source.sh

Install only what you need when you need them, see below I downloaded wget and then the jdk binary. I also learnt how to use wget by passing the necessary params and headers to make the server give us the binary we request. Finally un-tar the file using the famous tar command.

RUN \
  apt-get install -y wget && \
  wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" \ 
http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

RUN \
  tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt

Run configure with the famous –with-boot-jdk=/opt/jdk1.8.0_45 to set the bootstrap jdk to point to jdk1.8.0_45.

RUN \
  cd /tmp/openjdk9 && \
  bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45

Now run the most important command:

RUN \  
  cd /tmp/openjdk9 && \
  make clean images

Once the build is successful, the artefacts i.e. jdk and jre images are created in the build folder.

RUN \  
  cd /tmp/openjdk9 && \
  cp -a build/linux-x86_64-normal-server-release/images/jdk \
    /opt/openjdk9

Below are some chmod ceremonies across the files and directories in the openjdk9 folder.

RUN \  
  cd /tmp/openjdk9 && \
  find /opt/openjdk9 -type f -exec chmod a+r {} + && \
  find /opt/openjdk9 -type d -exec chmod a+rx {} +

Two environment variable i.e. PATH and JAVA_HOME are created with the respective values assigned to them.

ENV PATH /opt/openjdk9/bin:$PATH
ENV JAVA_HOME /opt/openjdk9

You can find the entire source for the entire Dockerfile on github [12].

…more of this in the next post, Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2, we will use the docker build, docker run commands and some more docker stuff.

How is Java / JVM built ? Adopt OpenJDK is your answer!

Introduction & history
As some of you may already know, starting with Java 7, OpenJDK is the Reference Implementation (RI) to Java. The below time line gives you an idea about the history of OpenJDK:
OpenJDK history (2006 till date)
If you have wondered about the JDK or JRE binaries that you download from vendors like Oracle, Red Hat, etcetera, then the clue is that these all stem from OpenJDK. Each vendor then adds some extra artefacts that are not open source yet due to security, proprietary or other reasons.


What is OpenJDK made of ?
OpenJDK is made up of a number of repositories, namely corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn. Between OpenjJDK8 and OpenJDK9 there have been no new repositories introduced, but lots of new changes and restructuring, primarily due to Jigsaw – the modularisation of Java itself [2] [3] [4] [5].
repo composition, language breakdown (metrics are estimated)
Recent history
OpenJDK Build Benchmarks – build-infra (Nov 2011) by Fredrik Öhrström, ex-Oracle, OpenJDK hero!

Fredrik Öhrström visited the LJC [16] in November 2011 where he showed us how to build OpenJDK on the three major platforms, and also distributed a four page leaflet with the benchmarks of the various components and how long they took to build. The new build system and the new makefiles are a result of the build system being re-written (build-infra).


Below are screen-shots of the leaflets, a good reference to compare our journey:

How has Java the language and platform built over the years ?

Java is built by bootstrapping an older (previous) version of Java – i.e. Java is built using Java itself as its building block. Where older components are put together to create a new component which in the next phase becomes the building block. A good example of bootstrapping can be found at Scheme from Scratch [6] or even on Wikipedia [7].


OpenJDK8 [8] is compiled and built using JDK7, similarly OpenJDK9 [9] is compiled and built using JDK8. In theory OpenJDK8 can be compiled using the images created from OpenJDK8, similarly for OpenJDK9 using OpenJDK9. Using a process called bootcycle images – a JDK image of OpenJDK is created and then using the same image, OpenJDK is compiled again, which can be accomplished using a make command option:
$ make bootcycle-images       # Build images twice, second time with newly built JDK

make offers a number of options under OpenJDK8 and OpenJDK9, you can build individual components or modules by naming them, i.e.

$ make [component-name] | [module-name]
or even run multiple build processes in parallel, i.e.
$ make JOBS=<n>                 # Run <n> parallel make jobs
Finally install the built artefact using the install option, i.e.
$ make install


Some myths busted
OpenJDK or Hotspot to be more specific isn’t completely written in C/C++, a good part of the code-base is good ‘ole Java (see the composition figure above). So you don’t have to be a hard-core developer to contribute to OpenJDK. Even the underlying C/C++ code code-base isn’t scary or daunting to look at. For example here is an extract of a code snippet from vm/memory/universe.cpp in the HotSpot repo –
.
.
.
Universe::initialize_heap()

if (UseParallelGC) {
#ifndef SERIALGC
Universe::_collectedHeap = new ParallelScavengeHeap();
#else // SERIALGC
fatal("UseParallelGC not supported in this VM.");
#endif // SERIALGC

} else if (UseG1GC) {
#ifndef SERIALGC
G1CollectorPolicy* g1p = new G1CollectorPolicy();
G1CollectedHeap* g1h = new G1CollectedHeap(g1p);
Universe::_collectedHeap = g1h;
#else // SERIALGC
fatal("UseG1GC not supported in java kernel vm.");
#endif // SERIALGC

} else {
GenCollectorPolicy* gc_policy;

if (UseSerialGC) {
gc_policy = new MarkSweepPolicy();
} else if (UseConcMarkSweepGC) {
#ifndef SERIALGC
if (UseAdaptiveSizePolicy) {
gc_policy = new ASConcurrentMarkSweepPolicy();
} else {
gc_policy = new ConcurrentMarkSweepPolicy();
}
#else // SERIALGC
fatal("UseConcMarkSweepGC not supported in this VM.");
#endif // SERIALGC
} else { // default old generation
gc_policy = new MarkSweepPolicy();
}

Universe::_collectedHeap = new GenCollectedHeap(gc_policy);
}
. . .
(please note that the above code snippet might have changed since published here)
The things that appears clear from the above code-block are, we are looking at how pre-compiler notations are used to create Hotspot code that supports a certain type of GC i.e. Serial GC or Parallel GC. Also the type of GC policy is selected in the above code-block when one or more GC switches are toggled i.e. UseAdaptiveSizePolicy when enabled selects the Asynchronous Concurrent Mark and Sweep policy. In case of either Use Serial GC or Use Concurrent Mark Sweep GC are not selected, then the GC policy selected is Mark and Sweep policy. All of this and more is pretty clearly readable and verbose, and not just nicely formatted code that reads like English.


Further commentary can be found in the section called Deep dive Hotspot stuff in the Adopt OpenJDK Intermediate & Advance experiences [11] document.


Steps to build your own JDK or JRE
Earlier we mentioned about JDK and JRE images – these are no longer only available to the big players in the Java world, you and I can build such images very easily. The steps for the process have been simplified, and for a quick start see the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents. For detailed version of the same steps, please see the Adopt OpenJDK home page [13]. Basically building a JDK image from the OpenJDK code-base boils down to the below commands:


(setup steps have been made brief and some commands omitted, see links above for exact steps)
 $ hg clone http://hg.openjdk.java.net/jdk8/jdk8 jdk8  (a)...OpenJDK8
or
$ hg clone http://hg.openjdk.java.net/jdk9/jdk9 jdk9  (a)...OpenJDK9
$ ./get_source.sh                                     (b)
$ bash configure                                      (c)
$ make clean images                                   (d)
(setup steps have been made brief and some commands omitted, see links above for exact steps)


To explain what is happening at each of the steps above:
(a) We clone the openjdk mercurial repo just like we would using git clone ….
(b) Once we have step (a) completed, we change into the folder created, and run the get_source.sh command, which is equivalent to a git fetch or a git pull, since the step (a) only brings down base files and not all of the files and folders.
(c) Here we run a script that checks for and creates the configuration needed to do the compile and build process
(d) Once step (c) is success we perform a complete compile, build and create JDK and JRE images from the built artefacts


As you can see these are dead-easy steps to follow to build an artefact or JDK/JRE images [step (a) needs to be run only once].


Benefits
– contribute to the evolution and improvement of the Java the language & platform
– learn about the internals of the language and platform
– learn about the OS platform and other technologies whilst doing the above
– get involved in F/OSS projects
– stay on top the latest changes in the Java / JVM sphere
– knowledge and experience that helps professionally but also these are not readily available from other sources (i.e. books, training, work-experience, university courses, etcetera).
– advancement in career
– personal development (soft skills and networking)


Contribute
Join the Adopt OpenJDK [13] and Betterrev [15] projects and contribute by giving us feedback about everything Java including these projects. Join the Adoption Discuss mailing list [14] and other OpenJDK related mailing lists to start with, these will keep you updated with latest progress and changes to OpenJDK. Fork any of the projects you see and submit changes via pull-requests.


Thanks and support

Adopt OpenJDK [13] and umbrella projects have been supported and progressed with help of JCP [21], the Openjdk team [22], JUGs like London Java Community [16], SouJava [17] and other JUGs in Brazil, a number of JUGs in Europe i.e. BGJUG (Bulgarian JUG) [18], BeJUG (Belgium JUG) [19], Macedonian JUG [20], and a number of other small JUGs. We hope in the coming time more JUGs and individuals would get involved. If you or your JUG wish to participate please get in touch.

—-

Credits
Special thanks to +Martijn Verburg (incepted Adopt OpenJDK),+Richard Warburton, +Oleg Shelajev, +Mite Mitreski, +Kaushik Chaubal and +Julius G for helping improve the content and quality of this post, and sharing their OpenJDK experience with us.

—-

How to get started ?
Join the Adoption Discuss mailing list [14], go to the Adopt OpenJDK home page [13] to get started, followed by referring to the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents.


Please share your comments here or tweet at @theNeomatrix369.

Resources
[17] SouJava

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

Post #fosdem, #jfokus – lots more #java, #jsr & #openjdk news from all over!

In the last couple of weeks since FOSDEM 2013, Jfokus 2013 and, events and hackdays organised by LJC JUG members, we have had a lots of exciting news and resources to share with you. The source of the information have been mailing lists / forums, meetup events, twitter and other sources.

Plenty were spoken about #java, #jsr and #openjdk, the topics covered by the #adoptajar and #adoptopenjdk programs.

FOSDEM 2013

Speaker interviews:


Jfokus 2013

#Jfokus coverage of #java, a snapshot by Kevin Farnham:


@steveonjava – Nighthacking!

Watch all the recordings of Steve’s Nighthacking from FOSDEM 2013 through to Jfokus 2013!


LJC events & hackdays

Garbage collection – The useful parts


WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353)


Bring your Performance Problems Panel


@adoptajsr news feed

– Plenty of updates on polls, JSRs, presentations, github projects, etc… were discussed on twitter.
– WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353) – see above!
Suggestion to extend / improve the Java API – Thanks Hildeberto Mendonça for coming forward to contribute!
– Modernize Connector/MDB — Vote and comment – Thanks Richard Kolb for support such initiatives!
CDI 1.1 applications you can work with – Thanks Luigi for the contribution!


@adoptopenjdk news feed

– Updates on latest changes and developments in the OpenJDK world that were discussed amongs members on twitter.
– Potential plans to deprecate SPARC V8 support in HotSpot!
– StringBuffer to StringBuilder again – discussions rekindled!


Upcoming events and meetings

Adopt-a-JSR online meeting – February 27
Further hackdays, discussion panels and events to be organised by LJC JUG members.
26-27 March 2013, Devoxx UK

—-

Read about the Adopt OpenJDK program at the the Adopt OpenJDK java.net project website.
Join the Adopt OpenJDK mailing list at the google group. Find out how to join a Google group or send an email to adopt-openjdk@googlegroups.com to subscribe to the mailing list.

Read about the Adopt-a-JSR program at the java.net project website.
Join the Adopt-a-JSR mailing list at the java.net mailing list. Send an email to members@adoptajsr.java.net to subscribe to the mailing list.

How to build JTReg in Eclipse (for Ubuntu 12.04 LTS)

After taking inspiration from doing a whole lot of work with Eclipse and OpenJDK projects I took off on the journey to experiment with JTReg (http://openjdk.java.net/jtreg/) and attempt to build it in an IDE of my choice i.e. Eclipse. These instructions specify how you can build the JTReg using the Eclipse IDE running under Ubuntu (using OpenJDK’s Old build and Infrabuild systems).

I have kept the structure of this blog similar to the previous one so to maintain uniformity in my approach.

Firstly what is JTReg – it is a regression test harness used by the OpenJDK test framework. The different projects and packages in the OpenJDK system can be individually or wholely tested via JTReg.

Below is a list of systems and versions under which the installations and configurations were performed to come up with these instructions:

Synaptic Package Manager 0.75.9

Eclipse Indigo 3.7.2
—Eclipse JDT (Java) 3.7.2
—Eclipse CDT (C/C++)
——6.0.0.dist (C/C++ Remote debug launcher)
——1.0.2.dist (CDT GCC Cross Compiler)
——7.0.0.dist (GDB Common)

Ubuntu 12.04 LTS
Ant 1.8.4
JavaHelp 2.0
Java Test Harness 4.4
JUnit 4-4.8
OpenJDK 8
Java/Javac 1.7

You need to have an environment with OpenJDK installed and setup (as described in the How to build OpenJDK projects in Eclipse (for Ubuntu 12.04 LTS) – just follow the parts from the start upto the section ‘Preparing projects, folders and files’, as hereafter the ‘meaty OpenJDK shabhang’ starts and we don’t need it for the current work.

If you wish to just download the JTReg binary run tests from the command line, follow the instructions on the
Adopt OpenJDK’s Install JTReg page – configurations for both the Old build and the Infrabuild systems are available there.

At this point you should be able to manually run JTReg tests from the command-line interface (terminal).

Build and run JTReg in Eclipse

Downloading and setting up dependencies

Change the current folder location to the $HOME directory folder and download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:
Download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:

 $
 $ hg clone http://hg.openjdk.java.net/code-tools/jtreg
 $ 

this will create a folder by the name jtreg in the current location with the JTReg source files in it.

Alternative but less updated location: http://download.java.net/openjdk/jtreg/). If this route is used, then run the below command in the terminal mode, to download the zip file stored at the above location:

 cd ~
 wget http://www.java.net/download/openjdk/jtreg/promoted/4.1/b04/jtreg-4.1-src-b04_14_mar_2012.zip

Create a folder say under the Home directory ~/ called ‘jtreg’, unzip the zip file into this folder.

JavaHelp
Install Javahelp – required to build jtreg

 sudo apt-get install javahelp2

JavaTest Harness
Download the Javatest Harness 4.4 from http://download.java.net/jtharness/download.html

Run the below commands to download it off the web into ~

 cd ~
 wget http://download.java.net/jtharness/4.4.1/Rel/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Unzip the zip file into /opt/jtharness/4.4 by doing the below:

 cd /opt
 sudo mkdir jtharness
 sudo mkdir jtharness/4.4
 cd jtharness/4.4
 sudo unzip ~/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Apache Ant
Download Apache Ant 1.8.4 from http://archive.apache.org/dist/ant/binaries/ using the below commands:

 cd ~
 wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.4-bin.zip   

 cd /opt
 sudo unzip ~/apache-ant-1.8.4-bin.zip

JUnit
Download JUnit4 jar file from Maven Central (http://search.maven.org/#search and search for junit4) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/junit4
 sudo mv junit-4.nn.jar /opt/junit4

TestNG
Download TestNG jar file from Maven Central (http://search.maven.org/#search and search for testng) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/testng
 sudo mv testng-n.n.jar /opt/testng

Now make changes to the build.properties file to amend paths for JUnit4, TestNG JavaHelp and JavaTest Harness.

Amending the Ant build properties file

As the build.properties file is protected, you will need to invoke it using the below commands:

 cd ~/jtreg/make
 sudo gedit build.properties

Make the following changes to the build.properties file in the ~/jtreg/make/ folder

jtreg.build.resources=/opt
 jhhome = /usr/share/java
 
 # JTHarness or JavaTest (should be should be 4.3 or better)
 #javatest.home = ${jtreg.build.resources}/jtharness/4.3
 javatest.home = ${jtreg.build.resources}/jtharness/4.4
 
 # JavaHelp (should be version 2.0 or better)
 javahelp.home = ${jhhome}
 #jhall.jar = ${javahelp.home}/javahelp/lib/jhall.jar
 jhall.jar = ${javahelp.home}/jhall.jar
 #jh.jar = ${javahelp.home}/javahelp/lib/jh.jar
 jh.jar = ${javahelp.home}/jh.jar
 
 # JUnit (should be 4.5 or better)
 #junit.jar = ${jtreg.build.resources}/junit/4.5/junit-4.5.jar
 junit.jar = ${jtreg.build.resources}/junit4-n.n.n.jar
  
 # TestNG (should be 6.5 or better)
 #testng.jar = ${jtreg.build.resources}/testng/6.7/other-jars/testng-6.7-dist.jar
 testng.jar = ${jtreg.build.resources}/testng/testng-n.n.jar
 
 # Ant ((should be version 1.6.5 or better)
 #ant.home = ${jtreg.build.resources}/ant/1.7.1
 ant.home = ${jtreg.build.resources}/apache-ant-1.8.4
 ant.jar = ${ant.home}/lib/ant.jar

Ensure comments are placed against the old settings (just in case you need to revert). In order to find the correct location of the jh.jar and jhall.jar files use the ‘find’ command in terminal mode.

Setting up environment variables

Add the below two lines to the ~/.bashrc file (Infrabuild)

     export JT_HOME=$HOME/jtreg                         ............. [1]
     
     or 
     
     export JT_HOME=$HOME/jtreg/dist/jtreg              ..............[2]

[1] Applies if the jtreg binary has been downloaded directly from the JTReg website.
[2] Applies if the jtreg source has been downloaded directly from the JTReg repo and a binary is built locally.

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image

Note: the name of the sub-folder under $SOURCE_CODE/jdk8_tl/build/ folder can have other variants i.e. linux-x86_64-normal-server-release, so please check for this in your build environment before applying the above env settings.

Or the below two lines to the ~/.bashrc file (Old build)

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image

Now source the .bashrc file using the below command:

source ~/.bashrc

Link folder /jtreg/src to the existing project using the Configure Build Path option.

Add the /opt/apache-ant-1.8.4/lib/ant.jar to the project via the Build Path > Configure Build Path > Libraries option.

Add External JARS…, navigate to the /opt/apache-ant-1.8.4/lib/ folder, select ant.jar and click on Okay.

Importing project via Ant Build file

Run Eclipse.
Select File > New > Other > Java Project from Existing Ant build
Once done select the folder ~/jtreg/make/ and then select the build.xml.

Selecting this file populates all the ant tasks, select the option Link to the buildfile in the file system.

Now select the build.xml file, right mouse click and select the Run As > Ant Build… option, and do the below:
1) Select Main, and add -verbose to the Arguments section
2) Select Targets, and select clean from the list of targets and set the order as clean, build (if you want clean builds each time). Do skim through the list of Ant tasks, to learn the different actions that can be performed pre- and post- builds.
3) Select ClassPath and set the ClassPath for Ant Home to Apach Ant 1.8.4 by selecting Ant Home from the list, and then clicking on the Ant Home button on the bottom right hand corner and select /opt/apache-ant-1.8.4/ from the directory tree.
4) Select JRE, and select java-7-openjdk-amd64 from the list. If it does not exist on the list, click on Installed JREs… and install one by adding a JRE from /usr/lib/jvm/java-7-openjdk-amd64.

Now run the ant script by selecting Run As > Ant build.

See sign of success below (last 30 odd lines of echos from the Ant console)

dist-jtreg:
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/lib/junit.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg-no-junit.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
build:
BUILD SUCCESSFUL
Total time: 37 seconds

Run Configuration

TODO – able to run OpenJDK tests on one or multiple projects / packages from within Eclipse – anyone dares to go this path?

Newly created Eclipse projects

Now you have a number of ready created Eclipse projects in your Eclipse workspace.

Go to the workspace folder where Eclipse creates the project files and settings i.e. into the /home/openjdk/workspace folder.

There is a folder (called jtreg) representing the project you are currently working in, there are atleast two to four hidden files and folders in this location containing the below:

 .project
 .classpath
 .settings

Importing project via Make file

TODO – importing the JTReg Make file and being able to do the same thing as above – anyone wishes to contribute?

Build and run JTReg from CLI (terminal)

The steps mentioned in sections Downloading and setting up dependencies, Amending the Ant build properties file and Setting up environment variables must be performed before going further.

Once the above settings are in place, build JTReg from the CLI via the following commands

 cd ~/sources/jtreg/make
 ant

This should give the same output as illustrated above under Eclipse section:


 jar-jtreg:
      [jar] Building jar: /home/saiubuntu/sources/jtreg/dist/jtreg/lib/jtreg.jar
 
 -dist-jtreg.doc:
     [copy] Copying 1 file to /home/saiubuntu/sources/jtreg/dist/jtreg/doc/jtreg
 
 -dist-jtreg.bin:
 
 -dist-jtreg.legal:
 
 dist-jtreg:
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/jtreg.zip
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/open-jtreg.zip
 
 build:
 
 BUILD SUCCESSFUL
 Total time: 9 seconds

Once a build is successful, a number of distribution files are created in the following folders under the jtreg folder structure:

 dist
 ├── jtreg
 │   ├── lib
 │   │   ├── javatest.jar
 │   │   ├── jh.jar
 │   │   ├── jtreg.jar
 │   │   └── junit.jar
 │   ├── linux
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   ├── solaris
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   └── win32
 │       └── bin
 │           ├── jtdiff
 │           └── jtreg
 ├── jtreg.zip
 └── open-jtreg.zip

The files under each of the OS names i.e. linux, solaris, windows are bash script files that execute the jtreg.jar file, for e.g.

 ./dist/jtreg/linux/bin/jtreg

or

 java -jar dist/jtreg/lib/jtreg.jar 

the above two will do the exact same thing and expects parameters, see usage screen (only a snapshot of the entire text):

 Documentation Options
                Options for additional documentation
    -h [words...] | -help [words...] | -usage [words...]
                    Command line help. Give words to see help info containing
                    those or use "-help all" to see all available help.
    -n | -relnote   Release notes
    -onlineHelp [words...]
                    Show the online help. You can also show the online help from
                    the desktop Help menu.
    -t | -tagspec   Tag specification supported by this implementation
    -version        Give information about the version of jtreg in use.
 ... [multi-page usage text]

The most simplest syntax to run jtreg is

  [absolute/relative path]/jjtreg [options] [folder]

for example

  [absolute/relative path]/jtreg -verbose:fail [absolute/relative path]/jdk/test/java/lang/invoke/

[absolute/relative path] – applies depending on where you position in the OpenJDK folder structure.

It is also possible to build jtreg via the make command but that would require additional installation of dependent components and configuration. Also have a look at Adopt OpenJDK’s Install JTReg page to see how tests are run via the CLI.

Credit and kudos

My sincere thanks to Jonathan Gibbon’s from OpenJDK’s JTReg team who has been kindly responding to my queries and has walked me through the last important steps in the process!

Thanks to the Adopt OpenJDK initiative led by Martijn Verburg and others – its a pleasure to support our community!

Big thanks to Rabea Gransberger for spending the time and meticulously going through the steps and reporting issues and suggestions.

Come and join us and take on a project (like I did) and make something out of the project and yourself!

Please provide any feedback on areas that work differently for you or does not work for you at all. If you have fixed any issue please let us know so that we can update the information here.

I will continue to refine the instructions above and also include your useful feedback as and when I receive them.