Truffle served in a Holy Graal: Graal and Truffle for polyglot language interpretation on the JVM

03 Hotspot versus GraalVM

Reblogging from ZeroTurnaround’s Rebellab blog site

One of the most fascinating additions to Java 9 is the JVMCI: Java-Level JVM Compiler Interface, a Java based compiler interface which allows us to plug in a dynamic compiler into the JVM. One of the main inspirations for including it into Java 9 was due to project Graal — a dynamic state-of-the-art compiler written in Java.

In this post we look at the reasons Graal is such a fascinating project, its advantages, what are the general code optimization ideas, some performance comparisons, and why would you even bother with tinkering with a new compiler.

Like everyone else we were inspired by the vJUG session by Chris Seaton on Graal – it looks like a great tool and technology and so we decided to play with the technology and share it with the community.

…you can read the rest at ZeroTurnaround’s Rebellab blogs


 

In case, you are wondering what some of the ASCII-art images in one of the paragraphs is about, here’s a bit of explanation, hopefully it will clear up any doubts.

How does it actually work?

A typical flow would look like this:

02-a Program to machine code diagram (excludes expansion)
AST → Abstract Syntax Tree  (explicit data structures in memory)

We all know that a JIT is embedded inside HotSpot or the JVM. It’s old, complicated, written in C++ and assembly and is fairly hard to understand. It is a black box and there is no way to hook or link into the JIT.  All the JVM languages have to go through the same route:  

02-b Program to machine code diagram (via byte-code)

(ASM = assembly)

The flow or route when dealing with traditional compilers and VM would be:

02-c Program to machine code diagram (via JIT)
But with Graal, we get the below route or flow:

02-d Program to machine code diagram (via AST)
(notice Graal skips the steps that create byte-code by directly generating platform specific machine code)

Graal basically helps moving the control-flow from Code to the JIT bypassing the JVM (HotSpot, in our case).


I hope you enjoyed the read, please feel free to share any constructive feedback, so we can improve the material for the community as a whole. We learnt a lot while drafting this post and hope the same for you.

Original post by @theNeomatrix369 and  @shelajev !

Advertisements

Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2

…continuing from Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2.

I ran into a number of issues and you can see from my commits, I pulled myself out of it, but to run this Dockerfile from the command-line I used this instruction:

$ docker build -t neomatrix369/openjdk9 .

you can also do it using the below if you have not set your permissions:

$ sudo docker build -t neomatrix369/openjdk9 .

and get the below (summarised) output:

Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon 
Step 0 : FROM phusion/baseimage:latest
 ---> 5a14c1498ff4
Step 1 : MAINTAINER Mani Sarkar (from @adoptopenjdk)
 ---> Using cache
 ---> 95e30b7f52b9
Step 2 : RUN apt-get update &&   apt-get install -y     libxt-dev zip pkg-config libX11-dev libxext-dev     libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev &&   rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 1ea3bbb15c2d
Step 3 : RUN apt-get update
 ---> Using cache
 ---> 6c3938f4d23d
Step 4 : RUN apt-get install -y mercurial ca-certificates-java build-essential
 ---> Using cache
 ---> e3f99b5a3bd3
Step 5 : RUN cd /tmp &&   hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 &&   cd openjdk9 &&   sh ./get_source.sh
 ---> Using cache
 ---> 26cfaf16b9fa
Step 6 : RUN apt-get install -y wget &&   wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
 ---> Using cache
 ---> 696889250fed
Step 7 : RUN tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt
 ---> Using cache
 ---> c25cc9201c1b
Step 8 : RUN cd /tmp/openjdk9 &&   bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45
 ---> Using cache
 ---> 4e425de379e6
Step 9 : RUN cd /tmp/openjdk9 &&   make clean images
 ---> Using cache
 ---> 2d9e17c870be
Step 10 : RUN cd /tmp/openjdk9 &&   cp -a build/linux-x86_64-normal-server-release/images/jdk     /opt/openjdk9
 ---> Using cache
 ---> 9250fac9b500
Step 11 : RUN cd /tmp/openjdk9 &&   find /opt/openjdk9 -type f -exec chmod a+r {} + &&   find /opt/openjdk9 -type d -exec chmod a+rx {} +
 ---> Using cache
 ---> d0c597d045d4
Step 12 : ENV PATH /opt/openjdk9/bin:$PATH
 ---> Using cache
 ---> 3965c3e47855
Step 13 : ENV JAVA_HOME /opt/openjdk9
 ---> Using cache
 ---> 5877e8efd939
Successfully built 5877e8efd939

The above action creates an image which is stored in your local repository (use docker images to enlist the images in the repo). If you want to load the image into a container, and access the files it has built or see anything else, do the below:

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 /bin/bash

this will take you to a bash prompt into the container and you can run any of your linux commands and access the file system.

Explaining docker run

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 java -version

will show you this

openjdk version "1.9.0-internal"
OpenJDK Runtime Environment (build 1.9.0-internal-_2015_06_04_06_46-b00)
OpenJDK 64-Bit Server VM (build 1.9.0-internal-_2015_06_04_06_46-b00, mixed mode)

Here’s a breakdown of the docker run command:

docker run The command to create and start a new Docker container.
-it To run in interactive mode, so you can see the after running the container.
neomatrix369/openjdk9 This is a reference to the image tag by name (which we created above).
java -version Runs the java command asking its version, inside the containing, which is assisted by the two environment variables PATH and JAVA_HOME which was set in the Dockerfile above.

Footnotes

You might have noticed I grouped very specific instructions with each step, especially the RUN commands, its because, each time I got one of these wrong, it would re-execute the step again, including the steps that ran fine and didn’t need re-executing. Not only is this unnecessary its not using our resources efficiently which is what Docker brings us. So any addition, edition or deletion to any step will only result in that step being executed, and not the other steps that are fine.

So one of the best practises is to keep the steps granular enough and pre-load files and data beforehand and give it to docker. It has amazing caching and archiving mechanisms built in.

Save our work

As we know if we do not save the container into the image, our changes are lost.

If I didn’t use the docker build command I used earlier I could have, after the build process was completed and image created, used the below command:

$ sudo docker commit [sha of the image] neomatrix369/openjdk9

Sharing your docker image on Docker hub

Once you are happy with your changes, and want to share it with community at large, do the below:

$ sudo docker push neomatrix369/openjdk9

and you will see these depending on which of your layers have been found in the repo and which ones are new (this one is an example snapshot of the process):

The push refers to a repository [neomatrix369/openjdk9] (len: 1)
5877e8efd939: Image already exists 
3965c3e47855: Image already exists 
d0c597d045d4: Image already exists 
9250fac9b500: Image already exists 
2d9e17c870be: Buffering to Disk
.
.
.

There is plenty of room for development and improvement of this Docker script. So happy hacking and would love to hear your feedback or contributions from you.

BIG Thanks

Big thanks to the below two who proof-read my post and added value to it, whilst enjoying the #Software #Craftsmanship developer community (organised and supported by @LSCC):
Oliver Nautsch – @ollispieps (JUG Switzerland)
Amir Bazazi (@Codurance) – @amirbazazi

Special thanks to Roberto Cortez (@radcortez) for your Docker posts, these inspired and helped me write my first Docker post.

Resources

[1] Docker
[2] Get into Docker – A Guide for Total Newbies
[3] Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images
[4] Docker posts on Voxxed
[5] OpenJDK
[6] Building OpenJDK
[7] Building OpenJDK on Linux, MacOs and Windows
[8] Virtual Machines (OpenJDK)
[9] Build your own OpenJDK
[10] Vagrant script (OpenJDK)
[11] YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it
[12] Dockerfile on github
[13] Adopt OpenJDK: Getting Started Kit
[14] London Java Community

Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2

Introduction

I think I have joined the Docker [1] party a bit late but that means by now everyone knows what Docker is and all the other basic fundamentals which I can very well skip, but if you are still interested, please check these posts Get into Docker – A Guide for Total Newbies [2] and Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images [3]. And if you still want to know more about this widely spoken topic, check out these Docker posts on Voxxed [4].

Why ?

Since everyone has been doing some sort of provisioning or spinning up of dev or pre-prod or test environments using Docker [1] I decided to do the same but with my favourite project i.e. OpenJDK [5].

So far you can natively build OpenJDK [6] across Linux, MacOs and Windows [7], or do the same things via virtual machines or vagrant instances, see more on then via these resources Virtual Machines, [8] Build your own OpenJDK [9] and this vagrant script [10]. All part of the Adopt OpenJDK initiative lead by London Java Community [14] and supported by JUGs all over the world.

Requirements

Most parts of post is for those using Linux distributions (this one was created on Ubuntu 14.04). Linux, MacOS and Windows users please refer to Docker‘s  Linux, MacOS and Windows instructions respectively.

Hints: MacOS and Windows users will need to install Boot2Docker and remember to run the below two commands (and check your Docker host environment variables):

$ boot2docker init
$ boot2docker up 
$ boot2docker shellinit 

For the MacOS, if the above throw FATA[…] error messages, please try the below:

$ sudo boot2docker init
$ sudo boot2docker up 
$ sudo boot2docker shellinit 

For rest of the details please refer to the links provided above. Once you have the above in place for the Windows or MacOS platform, by merely executing the Dockerfile using the docker build and docker run commands you can create / update a container and run it respectively.

*** Please refer to the above links and ensure Docker works for you for the above platforms – try out tutorials or steps proving that Docker run as expected before proceeding further. ***

Building OpenJDK 9 using Docker

Now I will show you how to do the same things as mentioned above using Docker.

So I read the first two resource I shared so far (and wrote the last ones). So lets get started, and I’m going to walk you through what the Dockerfile looks like, as I take you through each section of the Dockerfile code.

*** Please note the steps below are not meant to be executed on your command prompty, they form an integral part of the Dockerfile which you can download from here at the end of this post. ***

You have noticed unlike everyone else I have chosen a different OS image i.e. phusion/baseimage, why? Read YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it [11], to learn more about it.

FROM phusion/baseimage:latest

Each of the RUN steps below when executed becomes a Docker layer in isolation and gets assigned a SHA like this i.e. 95e30b7f52b9.

RUN \
  apt-get update && \
  apt-get install -y \
    libxt-dev zip pkg-config libX11-dev libxext-dev \
    libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev && \
  rm -rf /var/lib/apt/lists/*

The base image is updated and a number of dependencies are installed i.e. Mercurial (hg) and build-essential.

RUN \
  apt-get update && \
  apt-get install -y mercurial ca-certificates-java build-essential

Clone the OpenJDK 9 sources and download the latest sources from mercurial. You will notice that each of these steps are prefixed by this line cd /tmp &&, this is because each instruction is run in its own layer, as if it does not remember where it was when the previous instruction was run. Nothing to worry about, all your changes are still intact in the container.

RUN \
  cd /tmp && \
  hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 && \
  cd openjdk9 && \
  sh ./get_source.sh

Install only what you need when you need them, see below I downloaded wget and then the jdk binary. I also learnt how to use wget by passing the necessary params and headers to make the server give us the binary we request. Finally un-tar the file using the famous tar command.

RUN \
  apt-get install -y wget && \
  wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" \ 
http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

RUN \
  tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt

Run configure with the famous –with-boot-jdk=/opt/jdk1.8.0_45 to set the bootstrap jdk to point to jdk1.8.0_45.

RUN \
  cd /tmp/openjdk9 && \
  bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45

Now run the most important command:

RUN \  
  cd /tmp/openjdk9 && \
  make clean images

Once the build is successful, the artefacts i.e. jdk and jre images are created in the build folder.

RUN \  
  cd /tmp/openjdk9 && \
  cp -a build/linux-x86_64-normal-server-release/images/jdk \
    /opt/openjdk9

Below are some chmod ceremonies across the files and directories in the openjdk9 folder.

RUN \  
  cd /tmp/openjdk9 && \
  find /opt/openjdk9 -type f -exec chmod a+r {} + && \
  find /opt/openjdk9 -type d -exec chmod a+rx {} +

Two environment variable i.e. PATH and JAVA_HOME are created with the respective values assigned to them.

ENV PATH /opt/openjdk9/bin:$PATH
ENV JAVA_HOME /opt/openjdk9

You can find the entire source for the entire Dockerfile on github [12].

…more of this in the next post, Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2, we will use the docker build, docker run commands and some more docker stuff.

How is Java / JVM built ? Adopt OpenJDK is your answer!

Introduction & history
As some of you may already know, starting with Java 7, OpenJDK is the Reference Implementation (RI) to Java. The below time line gives you an idea about the history of OpenJDK:
OpenJDK history (2006 till date)
If you have wondered about the JDK or JRE binaries that you download from vendors like Oracle, Red Hat, etcetera, then the clue is that these all stem from OpenJDK. Each vendor then adds some extra artefacts that are not open source yet due to security, proprietary or other reasons.


What is OpenJDK made of ?
OpenJDK is made up of a number of repositories, namely corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn. Between OpenjJDK8 and OpenJDK9 there have been no new repositories introduced, but lots of new changes and restructuring, primarily due to Jigsaw – the modularisation of Java itself [2] [3] [4] [5].
repo composition, language breakdown (metrics are estimated)
Recent history
OpenJDK Build Benchmarks – build-infra (Nov 2011) by Fredrik Öhrström, ex-Oracle, OpenJDK hero!

Fredrik Öhrström visited the LJC [16] in November 2011 where he showed us how to build OpenJDK on the three major platforms, and also distributed a four page leaflet with the benchmarks of the various components and how long they took to build. The new build system and the new makefiles are a result of the build system being re-written (build-infra).


Below are screen-shots of the leaflets, a good reference to compare our journey:

How has Java the language and platform built over the years ?

Java is built by bootstrapping an older (previous) version of Java – i.e. Java is built using Java itself as its building block. Where older components are put together to create a new component which in the next phase becomes the building block. A good example of bootstrapping can be found at Scheme from Scratch [6] or even on Wikipedia [7].


OpenJDK8 [8] is compiled and built using JDK7, similarly OpenJDK9 [9] is compiled and built using JDK8. In theory OpenJDK8 can be compiled using the images created from OpenJDK8, similarly for OpenJDK9 using OpenJDK9. Using a process called bootcycle images – a JDK image of OpenJDK is created and then using the same image, OpenJDK is compiled again, which can be accomplished using a make command option:
$ make bootcycle-images       # Build images twice, second time with newly built JDK

make offers a number of options under OpenJDK8 and OpenJDK9, you can build individual components or modules by naming them, i.e.

$ make [component-name] | [module-name]
or even run multiple build processes in parallel, i.e.
$ make JOBS=<n>                 # Run <n> parallel make jobs
Finally install the built artefact using the install option, i.e.
$ make install


Some myths busted
OpenJDK or Hotspot to be more specific isn’t completely written in C/C++, a good part of the code-base is good ‘ole Java (see the composition figure above). So you don’t have to be a hard-core developer to contribute to OpenJDK. Even the underlying C/C++ code code-base isn’t scary or daunting to look at. For example here is an extract of a code snippet from vm/memory/universe.cpp in the HotSpot repo –
.
.
.
Universe::initialize_heap()

if (UseParallelGC) {
#ifndef SERIALGC
Universe::_collectedHeap = new ParallelScavengeHeap();
#else // SERIALGC
fatal("UseParallelGC not supported in this VM.");
#endif // SERIALGC

} else if (UseG1GC) {
#ifndef SERIALGC
G1CollectorPolicy* g1p = new G1CollectorPolicy();
G1CollectedHeap* g1h = new G1CollectedHeap(g1p);
Universe::_collectedHeap = g1h;
#else // SERIALGC
fatal("UseG1GC not supported in java kernel vm.");
#endif // SERIALGC

} else {
GenCollectorPolicy* gc_policy;

if (UseSerialGC) {
gc_policy = new MarkSweepPolicy();
} else if (UseConcMarkSweepGC) {
#ifndef SERIALGC
if (UseAdaptiveSizePolicy) {
gc_policy = new ASConcurrentMarkSweepPolicy();
} else {
gc_policy = new ConcurrentMarkSweepPolicy();
}
#else // SERIALGC
fatal("UseConcMarkSweepGC not supported in this VM.");
#endif // SERIALGC
} else { // default old generation
gc_policy = new MarkSweepPolicy();
}

Universe::_collectedHeap = new GenCollectedHeap(gc_policy);
}
. . .
(please note that the above code snippet might have changed since published here)
The things that appears clear from the above code-block are, we are looking at how pre-compiler notations are used to create Hotspot code that supports a certain type of GC i.e. Serial GC or Parallel GC. Also the type of GC policy is selected in the above code-block when one or more GC switches are toggled i.e. UseAdaptiveSizePolicy when enabled selects the Asynchronous Concurrent Mark and Sweep policy. In case of either Use Serial GC or Use Concurrent Mark Sweep GC are not selected, then the GC policy selected is Mark and Sweep policy. All of this and more is pretty clearly readable and verbose, and not just nicely formatted code that reads like English.


Further commentary can be found in the section called Deep dive Hotspot stuff in the Adopt OpenJDK Intermediate & Advance experiences [11] document.


Steps to build your own JDK or JRE
Earlier we mentioned about JDK and JRE images – these are no longer only available to the big players in the Java world, you and I can build such images very easily. The steps for the process have been simplified, and for a quick start see the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents. For detailed version of the same steps, please see the Adopt OpenJDK home page [13]. Basically building a JDK image from the OpenJDK code-base boils down to the below commands:


(setup steps have been made brief and some commands omitted, see links above for exact steps)
 $ hg clone http://hg.openjdk.java.net/jdk8/jdk8 jdk8  (a)...OpenJDK8
or
$ hg clone http://hg.openjdk.java.net/jdk9/jdk9 jdk9  (a)...OpenJDK9
$ ./get_source.sh                                     (b)
$ bash configure                                      (c)
$ make clean images                                   (d)
(setup steps have been made brief and some commands omitted, see links above for exact steps)


To explain what is happening at each of the steps above:
(a) We clone the openjdk mercurial repo just like we would using git clone ….
(b) Once we have step (a) completed, we change into the folder created, and run the get_source.sh command, which is equivalent to a git fetch or a git pull, since the step (a) only brings down base files and not all of the files and folders.
(c) Here we run a script that checks for and creates the configuration needed to do the compile and build process
(d) Once step (c) is success we perform a complete compile, build and create JDK and JRE images from the built artefacts


As you can see these are dead-easy steps to follow to build an artefact or JDK/JRE images [step (a) needs to be run only once].


Benefits
– contribute to the evolution and improvement of the Java the language & platform
– learn about the internals of the language and platform
– learn about the OS platform and other technologies whilst doing the above
– get involved in F/OSS projects
– stay on top the latest changes in the Java / JVM sphere
– knowledge and experience that helps professionally but also these are not readily available from other sources (i.e. books, training, work-experience, university courses, etcetera).
– advancement in career
– personal development (soft skills and networking)


Contribute
Join the Adopt OpenJDK [13] and Betterrev [15] projects and contribute by giving us feedback about everything Java including these projects. Join the Adoption Discuss mailing list [14] and other OpenJDK related mailing lists to start with, these will keep you updated with latest progress and changes to OpenJDK. Fork any of the projects you see and submit changes via pull-requests.


Thanks and support

Adopt OpenJDK [13] and umbrella projects have been supported and progressed with help of JCP [21], the Openjdk team [22], JUGs like London Java Community [16], SouJava [17] and other JUGs in Brazil, a number of JUGs in Europe i.e. BGJUG (Bulgarian JUG) [18], BeJUG (Belgium JUG) [19], Macedonian JUG [20], and a number of other small JUGs. We hope in the coming time more JUGs and individuals would get involved. If you or your JUG wish to participate please get in touch.

—-

Credits
Special thanks to +Martijn Verburg (incepted Adopt OpenJDK),+Richard Warburton, +Oleg Shelajev, +Mite Mitreski, +Kaushik Chaubal and +Julius G for helping improve the content and quality of this post, and sharing their OpenJDK experience with us.

—-

How to get started ?
Join the Adoption Discuss mailing list [14], go to the Adopt OpenJDK home page [13] to get started, followed by referring to the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents.


Please share your comments here or tweet at @theNeomatrix369.

Resources
[17] SouJava

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

Post #fosdem, #jfokus – lots more #java, #jsr & #openjdk news from all over!

In the last couple of weeks since FOSDEM 2013, Jfokus 2013 and, events and hackdays organised by LJC JUG members, we have had a lots of exciting news and resources to share with you. The source of the information have been mailing lists / forums, meetup events, twitter and other sources.

Plenty were spoken about #java, #jsr and #openjdk, the topics covered by the #adoptajar and #adoptopenjdk programs.

FOSDEM 2013

Speaker interviews:


Jfokus 2013

#Jfokus coverage of #java, a snapshot by Kevin Farnham:


@steveonjava – Nighthacking!

Watch all the recordings of Steve’s Nighthacking from FOSDEM 2013 through to Jfokus 2013!


LJC events & hackdays

Garbage collection – The useful parts


WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353)


Bring your Performance Problems Panel


@adoptajsr news feed

– Plenty of updates on polls, JSRs, presentations, github projects, etc… were discussed on twitter.
– WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353) – see above!
Suggestion to extend / improve the Java API – Thanks Hildeberto Mendonça for coming forward to contribute!
– Modernize Connector/MDB — Vote and comment – Thanks Richard Kolb for support such initiatives!
CDI 1.1 applications you can work with – Thanks Luigi for the contribution!


@adoptopenjdk news feed

– Updates on latest changes and developments in the OpenJDK world that were discussed amongs members on twitter.
– Potential plans to deprecate SPARC V8 support in HotSpot!
– StringBuffer to StringBuilder again – discussions rekindled!


Upcoming events and meetings

Adopt-a-JSR online meeting – February 27
Further hackdays, discussion panels and events to be organised by LJC JUG members.
26-27 March 2013, Devoxx UK

—-

Read about the Adopt OpenJDK program at the the Adopt OpenJDK java.net project website.
Join the Adopt OpenJDK mailing list at the google group. Find out how to join a Google group or send an email to adopt-openjdk@googlegroups.com to subscribe to the mailing list.

Read about the Adopt-a-JSR program at the java.net project website.
Join the Adopt-a-JSR mailing list at the java.net mailing list. Send an email to members@adoptajsr.java.net to subscribe to the mailing list.

How to build JTReg in Eclipse (for Ubuntu 12.04 LTS)

After taking inspiration from doing a whole lot of work with Eclipse and OpenJDK projects I took off on the journey to experiment with JTReg (http://openjdk.java.net/jtreg/) and attempt to build it in an IDE of my choice i.e. Eclipse. These instructions specify how you can build the JTReg using the Eclipse IDE running under Ubuntu (using OpenJDK’s Old build and Infrabuild systems).

I have kept the structure of this blog similar to the previous one so to maintain uniformity in my approach.

Firstly what is JTReg – it is a regression test harness used by the OpenJDK test framework. The different projects and packages in the OpenJDK system can be individually or wholely tested via JTReg.

Below is a list of systems and versions under which the installations and configurations were performed to come up with these instructions:

Synaptic Package Manager 0.75.9

Eclipse Indigo 3.7.2
—Eclipse JDT (Java) 3.7.2
—Eclipse CDT (C/C++)
——6.0.0.dist (C/C++ Remote debug launcher)
——1.0.2.dist (CDT GCC Cross Compiler)
——7.0.0.dist (GDB Common)

Ubuntu 12.04 LTS
Ant 1.8.4
JavaHelp 2.0
Java Test Harness 4.4
JUnit 4-4.8
OpenJDK 8
Java/Javac 1.7

You need to have an environment with OpenJDK installed and setup (as described in the How to build OpenJDK projects in Eclipse (for Ubuntu 12.04 LTS) – just follow the parts from the start upto the section ‘Preparing projects, folders and files’, as hereafter the ‘meaty OpenJDK shabhang’ starts and we don’t need it for the current work.

If you wish to just download the JTReg binary run tests from the command line, follow the instructions on the
Adopt OpenJDK’s Install JTReg page – configurations for both the Old build and the Infrabuild systems are available there.

At this point you should be able to manually run JTReg tests from the command-line interface (terminal).

Build and run JTReg in Eclipse

Downloading and setting up dependencies

Change the current folder location to the $HOME directory folder and download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:
Download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:

 $
 $ hg clone http://hg.openjdk.java.net/code-tools/jtreg
 $ 

this will create a folder by the name jtreg in the current location with the JTReg source files in it.

Alternative but less updated location: http://download.java.net/openjdk/jtreg/). If this route is used, then run the below command in the terminal mode, to download the zip file stored at the above location:

 cd ~
 wget http://www.java.net/download/openjdk/jtreg/promoted/4.1/b04/jtreg-4.1-src-b04_14_mar_2012.zip

Create a folder say under the Home directory ~/ called ‘jtreg’, unzip the zip file into this folder.

JavaHelp
Install Javahelp – required to build jtreg

 sudo apt-get install javahelp2

JavaTest Harness
Download the Javatest Harness 4.4 from http://download.java.net/jtharness/download.html

Run the below commands to download it off the web into ~

 cd ~
 wget http://download.java.net/jtharness/4.4.1/Rel/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Unzip the zip file into /opt/jtharness/4.4 by doing the below:

 cd /opt
 sudo mkdir jtharness
 sudo mkdir jtharness/4.4
 cd jtharness/4.4
 sudo unzip ~/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Apache Ant
Download Apache Ant 1.8.4 from http://archive.apache.org/dist/ant/binaries/ using the below commands:

 cd ~
 wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.4-bin.zip   

 cd /opt
 sudo unzip ~/apache-ant-1.8.4-bin.zip

JUnit
Download JUnit4 jar file from Maven Central (http://search.maven.org/#search and search for junit4) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/junit4
 sudo mv junit-4.nn.jar /opt/junit4

TestNG
Download TestNG jar file from Maven Central (http://search.maven.org/#search and search for testng) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/testng
 sudo mv testng-n.n.jar /opt/testng

Now make changes to the build.properties file to amend paths for JUnit4, TestNG JavaHelp and JavaTest Harness.

Amending the Ant build properties file

As the build.properties file is protected, you will need to invoke it using the below commands:

 cd ~/jtreg/make
 sudo gedit build.properties

Make the following changes to the build.properties file in the ~/jtreg/make/ folder

jtreg.build.resources=/opt
 jhhome = /usr/share/java
 
 # JTHarness or JavaTest (should be should be 4.3 or better)
 #javatest.home = ${jtreg.build.resources}/jtharness/4.3
 javatest.home = ${jtreg.build.resources}/jtharness/4.4
 
 # JavaHelp (should be version 2.0 or better)
 javahelp.home = ${jhhome}
 #jhall.jar = ${javahelp.home}/javahelp/lib/jhall.jar
 jhall.jar = ${javahelp.home}/jhall.jar
 #jh.jar = ${javahelp.home}/javahelp/lib/jh.jar
 jh.jar = ${javahelp.home}/jh.jar
 
 # JUnit (should be 4.5 or better)
 #junit.jar = ${jtreg.build.resources}/junit/4.5/junit-4.5.jar
 junit.jar = ${jtreg.build.resources}/junit4-n.n.n.jar
  
 # TestNG (should be 6.5 or better)
 #testng.jar = ${jtreg.build.resources}/testng/6.7/other-jars/testng-6.7-dist.jar
 testng.jar = ${jtreg.build.resources}/testng/testng-n.n.jar
 
 # Ant ((should be version 1.6.5 or better)
 #ant.home = ${jtreg.build.resources}/ant/1.7.1
 ant.home = ${jtreg.build.resources}/apache-ant-1.8.4
 ant.jar = ${ant.home}/lib/ant.jar

Ensure comments are placed against the old settings (just in case you need to revert). In order to find the correct location of the jh.jar and jhall.jar files use the ‘find’ command in terminal mode.

Setting up environment variables

Add the below two lines to the ~/.bashrc file (Infrabuild)

     export JT_HOME=$HOME/jtreg                         ............. [1]
     
     or 
     
     export JT_HOME=$HOME/jtreg/dist/jtreg              ..............[2]

[1] Applies if the jtreg binary has been downloaded directly from the JTReg website.
[2] Applies if the jtreg source has been downloaded directly from the JTReg repo and a binary is built locally.

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image

Note: the name of the sub-folder under $SOURCE_CODE/jdk8_tl/build/ folder can have other variants i.e. linux-x86_64-normal-server-release, so please check for this in your build environment before applying the above env settings.

Or the below two lines to the ~/.bashrc file (Old build)

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image

Now source the .bashrc file using the below command:

source ~/.bashrc

Link folder /jtreg/src to the existing project using the Configure Build Path option.

Add the /opt/apache-ant-1.8.4/lib/ant.jar to the project via the Build Path > Configure Build Path > Libraries option.

Add External JARS…, navigate to the /opt/apache-ant-1.8.4/lib/ folder, select ant.jar and click on Okay.

Importing project via Ant Build file

Run Eclipse.
Select File > New > Other > Java Project from Existing Ant build
Once done select the folder ~/jtreg/make/ and then select the build.xml.

Selecting this file populates all the ant tasks, select the option Link to the buildfile in the file system.

Now select the build.xml file, right mouse click and select the Run As > Ant Build… option, and do the below:
1) Select Main, and add -verbose to the Arguments section
2) Select Targets, and select clean from the list of targets and set the order as clean, build (if you want clean builds each time). Do skim through the list of Ant tasks, to learn the different actions that can be performed pre- and post- builds.
3) Select ClassPath and set the ClassPath for Ant Home to Apach Ant 1.8.4 by selecting Ant Home from the list, and then clicking on the Ant Home button on the bottom right hand corner and select /opt/apache-ant-1.8.4/ from the directory tree.
4) Select JRE, and select java-7-openjdk-amd64 from the list. If it does not exist on the list, click on Installed JREs… and install one by adding a JRE from /usr/lib/jvm/java-7-openjdk-amd64.

Now run the ant script by selecting Run As > Ant build.

See sign of success below (last 30 odd lines of echos from the Ant console)

dist-jtreg:
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/lib/junit.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg-no-junit.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
build:
BUILD SUCCESSFUL
Total time: 37 seconds

Run Configuration

TODO – able to run OpenJDK tests on one or multiple projects / packages from within Eclipse – anyone dares to go this path?

Newly created Eclipse projects

Now you have a number of ready created Eclipse projects in your Eclipse workspace.

Go to the workspace folder where Eclipse creates the project files and settings i.e. into the /home/openjdk/workspace folder.

There is a folder (called jtreg) representing the project you are currently working in, there are atleast two to four hidden files and folders in this location containing the below:

 .project
 .classpath
 .settings

Importing project via Make file

TODO – importing the JTReg Make file and being able to do the same thing as above – anyone wishes to contribute?

Build and run JTReg from CLI (terminal)

The steps mentioned in sections Downloading and setting up dependencies, Amending the Ant build properties file and Setting up environment variables must be performed before going further.

Once the above settings are in place, build JTReg from the CLI via the following commands

 cd ~/sources/jtreg/make
 ant

This should give the same output as illustrated above under Eclipse section:


 jar-jtreg:
      [jar] Building jar: /home/saiubuntu/sources/jtreg/dist/jtreg/lib/jtreg.jar
 
 -dist-jtreg.doc:
     [copy] Copying 1 file to /home/saiubuntu/sources/jtreg/dist/jtreg/doc/jtreg
 
 -dist-jtreg.bin:
 
 -dist-jtreg.legal:
 
 dist-jtreg:
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/jtreg.zip
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/open-jtreg.zip
 
 build:
 
 BUILD SUCCESSFUL
 Total time: 9 seconds

Once a build is successful, a number of distribution files are created in the following folders under the jtreg folder structure:

 dist
 ├── jtreg
 │   ├── lib
 │   │   ├── javatest.jar
 │   │   ├── jh.jar
 │   │   ├── jtreg.jar
 │   │   └── junit.jar
 │   ├── linux
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   ├── solaris
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   └── win32
 │       └── bin
 │           ├── jtdiff
 │           └── jtreg
 ├── jtreg.zip
 └── open-jtreg.zip

The files under each of the OS names i.e. linux, solaris, windows are bash script files that execute the jtreg.jar file, for e.g.

 ./dist/jtreg/linux/bin/jtreg

or

 java -jar dist/jtreg/lib/jtreg.jar 

the above two will do the exact same thing and expects parameters, see usage screen (only a snapshot of the entire text):

 Documentation Options
                Options for additional documentation
    -h [words...] | -help [words...] | -usage [words...]
                    Command line help. Give words to see help info containing
                    those or use "-help all" to see all available help.
    -n | -relnote   Release notes
    -onlineHelp [words...]
                    Show the online help. You can also show the online help from
                    the desktop Help menu.
    -t | -tagspec   Tag specification supported by this implementation
    -version        Give information about the version of jtreg in use.
 ... [multi-page usage text]

The most simplest syntax to run jtreg is

  [absolute/relative path]/jjtreg [options] [folder]

for example

  [absolute/relative path]/jtreg -verbose:fail [absolute/relative path]/jdk/test/java/lang/invoke/

[absolute/relative path] – applies depending on where you position in the OpenJDK folder structure.

It is also possible to build jtreg via the make command but that would require additional installation of dependent components and configuration. Also have a look at Adopt OpenJDK’s Install JTReg page to see how tests are run via the CLI.

Credit and kudos

My sincere thanks to Jonathan Gibbon’s from OpenJDK’s JTReg team who has been kindly responding to my queries and has walked me through the last important steps in the process!

Thanks to the Adopt OpenJDK initiative led by Martijn Verburg and others – its a pleasure to support our community!

Big thanks to Rabea Gransberger for spending the time and meticulously going through the steps and reporting issues and suggestions.

Come and join us and take on a project (like I did) and make something out of the project and yourself!

Please provide any feedback on areas that work differently for you or does not work for you at all. If you have fixed any issue please let us know so that we can update the information here.

I will continue to refine the instructions above and also include your useful feedback as and when I receive them.

How to build OpenJDK projects in Eclipse (for Ubuntu 12.04 LTS)

The instructions below specify how you can build the following OpenJDK projects using the Eclipse IDE for both the new InfraBuild and the Old Build systems:

Swing
JMX
JConsole
Langtools
JAXP and JAXWS
Hotspot
CORBA
JDK — sub-projects – Old Build
and
JDK — sub-projects – Infrabuild

Below is a list of systems and versions under which the installations and configurations were performed to come up with these instructions:

Synaptic Package Manager 0.75.9

Eclipse Indigo 3.7.2
—Eclipse JDT (Java) 3.7.2
—Eclipse CDT (C/C++)
——6.0.0.dist (C/C++ Remote debug launcher)
——1.0.2.dist (CDT GCC Cross Compiler)
——7.0.0.dist (GDB Common)

Ubuntu 12.04 LTS
Ant 1.8.2
OpenJDK 8
Java/Javac 1.7

In order to start, you have to create a VM (or install in your native environment), get the OpenJDK source code, install a number of packages, and build the OpenJDK from command line, as it described in Adopt OpenJDK VM Build (Old Build and InfraBuild).


Installing Eclipse for Java via the Synaptic Package Manager

Install Synaptic Package Manager using Ubuntu Software Centre

Install Eclipse packages via the Synaptic Package Manager by searching for “Eclipse-jdt” (without quotes).
Select eclipse-jdt from the packages list and the rest of the dependencies will also get selected.

eclipse-jdt

Eclipse Java Development Tool (eclipse-jdt)

Once that is done, click on Apply.
A Summary window will appear, click on the Apply button again.
Then wait till the Downloading Package Files is finished.
Run Eclipse via the Dash Home

DO NOT USE THE “Check for Updates” from the Help menu in Eclipse, this could lead to problems with the Java perspective or render the Eclipse platform unusable (error message). Although you can use the “Install New Software” option from the Help menu – its best not to.

PLEASE USE THE “Synaptic Package Manager” to do any “updates / upgrades” to Eclipse packages and plugins.

You can use the below commands (optional) to update and upgrade packages:

sudo apt-get update
(password requested)

sudo apt-get upgrade

Also accept any available updates from the Ubuntu Software Centre. Once this is done, a reboot of the Ubuntu system will be required.

Run Eclipse

Check if the Java perspective is available (see under File > New > New Project menu options).

JRE Runtime Environment
To add the run-time environment into Eclipse do the following:
a) Go to Windows > Preferences > Java
b) Select Installed JRE
c) Add a JRE by clicking on the Add button
d) Browse to the /usr/lib/jvm/java-7-openjdk-amd64 folder and select it.

Select Installed JRE

Select Installed JRE

e) Click on Finish to add the new JRE to Eclipse’s list of JREs.

InstalledJRE

InstalledJRE for all build.xml scripts

Shutdown Eclipse once done.


Installing Eclipse C/C++ plug-in (Eclipse CDT) via the Synaptic Package Manager

Run the Synaptic Package Manager
Search for “eclipse-cdt” (without quotes) and click on Search

eclipse-cdt

Eclipse C/C++ Development Tool (eclipse-cdt)

A list of packages appear, select ‘eclipse-cdt’ from the list. The a list of dependencies appear, select them all. Select Mark for Installation.
Click on the Apply button on the ribbon panel followed by a click on the Apply button at the bottom right hand corner.

If the Synaptic Package Manager detects updates/upgrades and wishes to perform them, accept and allow the changes to be applied.

Click on Next and finally Finished.

Post Eclipse installations, Eclipse would need to be restarted. Post installation of updates/upgrades Ubuntu would need to be restarted.

Before building any packages through command-line or Eclipse run the below command to install the ‘g++4.6-multlib’ library:

sudo apt-get install g++-4.6-multilib

It ensures that command-line and Eclipse build actions go smoothly wherever the g++ library is needed.

Preparing projects, folders and files

Go to into the ~/sources/jdk8_tl/jdk/make/ folder and copy the netbeans folder and call it eclipse. Similarly go into the ~/sources/jdk8_tl/langtools/make/ folder and copy the netbeans folder and call it eclipse.

Then go to the individual folders and rename the nbproject to ecproject wherever applicable. Remove the contents of this folder.

Open ~/sources/jdk8_tl/jdk/make/eclipse/common/shared.xml in a text editor, go to lines 130 and 132 and replace 1.5 and 1.6 to ${javac.version.source} and ${javac.version.target}. This will eventually be replaced in each of the individual build.properties files for each of the packages as shown below.

Iterate through the below folders to look for the build.xml and build.properties files, if either of them do not exist, create the file:

~/sources/jdk8_tl/jaxp/
~/sources/jdk8_tl/jaxws
~/sources/jdk8_tl/jdk/make/eclipse/j2se/
~/sources/jdk8_tl/jdk/make/eclipse/jconsole/
~/sources/jdk8_tl/jdk/make/eclipse/jmx/
~/sources/jdk8_tl/jdk/make/eclipse/swing/
~/sources/jdk8_tl/langtools/make/eclipse/langtools/

Ensure the build.xml scripts have Ant tasks in each of them.

Otherwise create them taking the build script at ~/sources/jdk8_tl/jdk/make/eclipse/swing/build.xml as an example. For one of the packages the Ant Task does exist but it is surrounded by comments, and would require uncommenting.

Look for the ‘javac’ tag in this file, a typical Ant task looks like the below:

    <target name="run" depends="-init">
        <property name="jvm.args" value=""/>
        <javac srcdir="${swing.demo.src}" destdir="${swing.demo.classes}"
            fork="true" failonerror="true"
            classpath="${dist.dir}/lib/swing.jar:${classes.dir}"
            debug="${javac.debug}" debuglevel="${javac.debuglevel}">
            <compilerarg line="${javac.options}"/>
        </javac>
        <java classname="SampleTree" classpath="${swing.demo.classes}"
            fork="true" failonerror="true"
            jvm="${bootstrap.jdk}/bin/java">
            <jvmarg line="${demo.bootclasspath}"/>
            <arg line="${jvm.args}"/>
        </java>
    </target>

Search and replace all references to the texts “netbeans” and “nbproject” in the above files, to “eclipse” and “ecproject” respectively (without the quotes).

Amend the build.xml scripts to include source files, output and distribution folders (optional).

Also insert the following line at the top of the build.xml file just under the tag for all the above build.xml files:

  <property name="build.properties" location="./build.properties"/>

After importing the above packages, change the JRE version of the build.xml for each of the packages from 1.6 to 1.7 by using the right-mouse-click,

Ant-EditConfiguration (1.6 to 1.7)

Change JRE from 1.6 to 1.7 for all build scripts.

select the menu option Run As > Ant Build…, select the JRE tab, select java-7-openjdk-amd64 from the list next to Separate JRE.

Ant-SelectingAntBuild

Starting the build process by running the Ant Build script.

From within all the build.xml files remove references to the -demo-init, shared-clean, etc… defined with the depends attribute in the tags. Alternatively create dummy ones in each of the script where required in order to validate them sucessfully.

Amend the Ant Task attached to the build.xml by opening the Ant properties page and checking the ‘clean’ task and reording the build and clean taks in the following order, by using the Order button:

clean or shared-clean
build

Now individually import these projects into Eclipse (after ensuring both Java and C/C++ projects can be loaded into Eclipse) after making amendments to the build.properties file for the relevant projects.

NewProject (Import project from Ant Build script)

NewProject (Import project from Ant Build script)

NewJavaProject (next screen)

NewJavaProject (next screen)

SelectBuildXML

Select the build.xml file to import the project.

a) Swing
Before importing any project make the following amendments

Insert the below lines to the ~/sources/jdk8_tl/jdk/make/eclipse/swing/build.properties file:
Old_Build

#PATH to JDK8
bootstrap.jdk=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image
basedir=~/sources/jdk8_tl/jdk/make/eclipse/swing/
javac.version.source=1.5
javac.version.target=1.7

InfraBuild

#PATH to JDK8
bootstrap.jdk=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image
basedir=~/sources/jdk8_tl/jdk/make/eclipse/swing/
javac.version.source=1.5
javac.version.target=1.7

If file already exists check if the contents have the above lines, if not add these lines to the bottom of the build.properties file.

Run Eclipse and do the below:

Import a project using the Import a project from existing Ant BuildFile option from New > Project > Other, click on Browse, go to the ~/sources/jdk8_tl/jdk/make/eclipse/swing folder and select build.xml.

Once the build file is loaded, select the ‘Link to buildfile in the file system’ option. The project gets loaded as expected and gets the name defined in the script.

Open the build.xml properties by selecting the file, and selecting the Run As > Build Ant… option from the menu. Set the Target to “shared-clean, build” by selecting shared-clean from the list of targets and changing the order using the Order button.

Run the Build of Swing project by selecting the build.xml file and right mouse click on the Run as > Ant build option.

Observe the messages in the Console tab in Eclipse and wait for sometime till the build is complete (depends on performance of your computer).

See How to build the Swing project for a more detailed approach.

b) JMX
Perform the same step as for Swing using the respective files in the ~/sources/jdk8_tl/jdk/make/eclipse/jmx/ folder. Also amend the build.properties file in the same manner.

Run the Build of JMX project by selecting the build.xml file and right mouse click on the Run as > Ant build option.

Observe the messages in the Console tab in Eclipse and wait for sometime till the build is complete (depends on performance of your computer).

See How to build the JMX project for a more detailed approach.

c) JConsole
Perform the same step as for Swing using the respective files in the ~/sources/jdk8_tl/jdk/make/eclipse/jconsole/ folder.

Amend the build.properties file a bit diferent, insert the below lines instead:

Old Build

#PATH to JDK8
bootstrap.jdk=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image
basedir=~/sources/jdk8_tl/jdk/make/eclipse/jconsole/
javac.version.source=1.7
javac.version.target=1.7

InfraBuild

#PATH to JDK8
bootstrap.jdk=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image
basedir=~/sources/jdk8_tl/jdk/make/eclipse/jconsole/
javac.version.source=1.7
javac.version.target=1.7

Run the Build of JConsole project by selecting the build.xml file and right mouse click on the Run as > Ant build option.

Observe the messages in the Console tab in Eclipse and wait for sometime till the build is complete (depends on performance of your computer).

See How to build the JConsole project for a more detailed approach.

d) Langtools

Edit the build.properties for langtools located under ~/sources/jdk8_tl/langtools,and amend the following bits in the boot.java.home section:

boot.java.home = /usr/lib/jvm/java-7-openjdk-amd64
boot.javac.source = 7
boot.javac.target = 7

Copy the build.properties file from the ~/sources/jdk8_tl/langtools/make/ folder to the ~/sources/jdk8_tl/langtools/make/eclipse/langtools folder.

In addition for langtools, change the following two tags at the top of the file from

<project name="langtools-netbeans" default="build" basedir="../../..">
       <property name="langtools.properties" location="make/netbeans/langtools/nbproject/private/langtools.properties"/>

to

<project name="langtools-eclipse" default="build" basedir="../../..">
       <property name="langtools.properties" location="make/eclipse/langtools/ecproject/private/langtools.properties"/>

Ensure the build.properties in the /home/openjdk/sources/jdk8_tl/langtools/make/ folder has the below line in it otherwise add/update it:

boot.java.home = usr/lib/jvm/java-7-openjdk-amd64

Build the project via the right-mouse click Run As > Ant Build option and wait for it to finish.

See How to build the Langtools project for a more detailed approach.

e) JAXP and JAXWS

For both these packages, edit the build.properties for each of them located under ~/sources/jdk8_tl/, add the below line to it:

bootstrap.dir=/usr/lib/jvm/java-7-openjdk-amd64

Perform the same tasks as for the Swing package with these two packages and watch for it to build.

Build the project via the right-mouse click Run As > Ant Build option and wait for it to finish.

See How to build the JAXP and JAXWS projects for a more detailed approach.

f) Hotspot

Building Hotspot via the command-line interface (Old build)

Perform the below instructions to build Hotspot using the old build system (ensure environment variables are set – see the sub-heading Old Build. And the ~/.bashrc file source-ed before proceeding):

source ~/.bashrc

cd ~/sources/jdk8_tl/hotspot/make
make clean
make all
-or-
make clean all

(the first make just does a clean build, the next one does a full or incremental build, whilst the last one always does a clean full build)

A successful build will result in the following build messages that can be found in the log file (https://github.com/neomatrix369/BuildHelpers/blob/master/EclipseProjectsForOpenJDK/Logs/hotspot/hotspotBuild.log). An incremental (another attempt) build results in the following log messages (https://github.com/neomatrix369/BuildHelpers/blob/master/EclipseProjectsForOpenJDK/Logs/hotspot/hotspotBuild2ndPass.log).

Building Hotspot via the command-line interface (InfraBuild)

Configuration
We do not need to set the ALT_BOOTDIR or ALT_JDK_IMPORT_PATH environment variables, instead use the –with-jdk=/usr/lib/jvm/java-7-openjdk-amd64 parameter when calling the configure bashscript, for e.g.

bash ../autoconf/configure –with-boot-jdk=/usr/lib/jvm/java-7-openjdk-amd64/

run the above command from within the ~/sources/jdk8_tl/common/makefiles folder. The jdk or jre path specified with the –with-jdk parameter may vary and is specific to your linux environment.

A successful run results in the output of the following log messages (https://github.com/neomatrix369/BuildHelpers/blob/master/EclipseProjectsForOpenJDK/Logs/configureInfraBuild.log).

Building
Perform the below instructions to build Hotspot using the Infrabuild system (ensure the necessary enviroment variables are set and the ALT_… environment variables are unset – see the sub-heading InfraBuild. And the ~/.bashrc file source-ed before proceeding)

source ~/.bashrc
make clean hotspot NEW_BUILD=true &> hotspotInfraBuild.log

A successful run results in the output of the following log messages (https://github.com/neomatrix369/BuildHelpers/blob/master/EclipseProjectsForOpenJDK/Logs/hotspot/hotspotInfraBuild.log). An incremental (another attempt) build results in the following log messages (https://github.com/neomatrix369/BuildHelpers/blob/master/EclipseProjectsForOpenJDK/Logs/hotspot/hotspotInfraBuild2ndPass.log).

Perform the configuration settings in Eclipse for Hotspot using the instructions at Eclipse C/C++ (CDT) – Hacking Hotspot in Eclipse and the Adding include paths to your C/C++ project instructions sets.

Ensure the below environment variables are added to the project:
Old Build

LANG=C
ALT_BOOTDIR=/usr/lib/jvm/java-7-openjdk-amd64
ALT_JDK_IMPORT_PATH=/usr/lib/jvm/java-7-openjdk-amd64
PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image
ZIP_DEBUGINFO_FILES=0

InfraBuild

LANG=C
PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image
ZIP_DEBUGINFO_FILES=0
ARCH_DATA_MODEL=64

Special steps for creating the C/C++ projects
Add the ‘make’ folder to the Source folder list by going to Project Properties > C/C++ General / Paths and Symbols > Source Location.
Click on Add Folder and select the /hotspot/make folder.

Then add an external builder by going to Project Properties > Builders
Click on New, choose Program as configuration type, click on OK and
call the builder ‘make_hotspot’.
Link it to /usr/bin/make by setting as the Location
Make the Working directory point to the make or makefiles folder – use the Browse Filesystem option to locate it.

For you’d like to do incremental builds, add “all” to the arguments list and for a full build add “clean” and “all” in order to determine the build action each time the project is built:
clean
all

Finally add the above enlisted environment variables for Hotspot to the External Builder configuration under the Environment tab, as shown in the screen-shot (enlists two of the remaining expected environment variables):

External Builder Environment Settings

External Builder Environment Settings

Now under Project Properties > C/C++ build
Under the Builder Settings tab
Check Generate Makefiles Automatically
Select External Builder as the Builder type (see at the top of the tab)
Keep the Generate Makefiles automatically checked.
Click on Apply.

Now Go to the Behaviour tab and uncheck the options [Build on resource save…] and [Build(incremental build)] options to ensure that the External builder configuration is used when the build action is invoked on a project.

Click on OK.

Build the project via the Project > Build Project option and wait for it to finish. At this point the project should be in a state to be build by either the Internal or the External Builder.

See How to build the Hotspot project for a more detailed approach.

g) Corba

Do the same actions as above for Corba as for the Hotspot project and translate names and locations that relate to the Corba project.

Build the project via the Project > Build Project option and wait for it to finish. At this point the project should be in a state to be build by either the Internal or the External Builder.

See How to build the Corba project for a more detailed approach.

h) JDK — sub-projects – Old Build

This build depends on successful builds of the Hotspot, JAX, JAXWS, Langtools and Corba packages.

Add the below lines to the build.properties file in case it does not already exist:

make.options=\
ALT_BOOTDIR=/usr/lib/jvm/java-7-openjdk-amd64 \
ALT_LANGTOOLS_DIST=~/sources/jdk8_tl/langtools/dist \
ALT_CORBA_DIST=~/sources/jdk8_tl/corba/dist \
ALT_JAXP_DIST=~/sources/jdk8_tl/jaxp/dist \
ALT_JAXWS_DIST=~/sources/jdk8_tl/jaxws/dist \
HOTSPOT_IMPORT_PATH=~/sources/jdk8_tl/hotspot/build/linux-amd64_backup/hotspot/import \
ALT_HOTSPOT_IMPORT_PATH=~/sources/jdk8_tl/hotspot/build/linux-amd64_backup/hotspot/import \
HOTSPOT_SERVER_PATH=~/sources/jdk8_tl/build/linux-amd64_backup/hotspot/import/jre/lib/amd64/server \
ALT_HOTSPOT_SERVER_PATH=~/sources/jdk8_tl/build/linux-amd64_backup/hotspot/import/jre/lib/amd64/server \
OPENJDK=true

Import a project using the Import a project from existing Ant BuildFile option via the New > Project > Other menu option and select ~/sources/jdk8_tl/jdk/make/eclipse/j2se/build.xml file for the j2se project.

Alternatively the whole project can be loaded as a C/C++ project following the same instructions as for the Hotspot project above. The above environment variables settings will be required to be added to the project.

Build the project via the Project > Build Project option and wait for it to finish.

Once all of these packages are build successfully, close down Eclipse.

For sanity check purposes please ensure that the individual packages / projects build from the command prompt and the OpenJDK project as a whole builds from the command prompt i.e.

cd ~/sources/jdk8_tl/[...project..]/make/    or   cd ~/sources/jdk8_tl/jdk/make/[...project..]/
make all

This is to ensure one process hasn’t negatively impacted the other. For a list of project names refer to the section at the top of page.

See How to build the JDK sub-projects – Old build for a more detailed approach.

i) JDK — sub-projects – Infrabuild

Do the same actions as above for the Hotspot project and translate names and locations that relate to the OpenJDK project. Import the project from /home/openjdk/sources/jdk8_tl/common/makefiles folder, rename the project to something like ‘OpenJDK’ instead of the ‘makefiles’ name given by default. Depending on the build system used, apply the below environment variable and argument list settings (does not require the traditional ALT_ ones anymore – instead they should all be unset).

LANG=C
PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image
ZIP_DEBUGINFO_FILES=0

When setting up the External Builder setting for the Infrabuild system, you can apply one of the below entries in the arguments list:

images
clean images
images VERBOSE=”-d -p”
clean images VERBOSE=”-d -p”

Note: Please refer to http://openjdk.java.net/projects/build-infra/guide.html for further details on what the arguments mean in the Infrabuild system.

The rest of the instructions to setup the Hotspot project apply. Build the project via the Project > Build Project option and wait for it to finish. The above results in creation of the j2re and j2sdk images in the /jdk8_tl/build/linux-x64-normal-server-release/images folder (in folders j2re-image and j2sdk-image respectively).

Note: you can jump start the process by using the ready-made projects made available in the Github repo mentioned below. A clean build using the InfraBuild system would take an hour on a VM and under 15 minutes on a native machine – all this subject to various performance factors of a system. The Old build system is slower in this respect.

See How to build the JDK sub-projects – Infrabuild for a more detailed approach.


Full OpenJDK build Logs

See Full OpenJDK build Logs to view the logs produced from a full build action.


Full OpenJDK build output

See Full build output: artefacts for details on build outputs (artefacts) produced from a full build action.


Newly created Eclipse projects

Now you have a number of ready created Eclipse projects in your Eclipse workspace.

Go to the workspace folder where Eclipse creates the project files and settings i.e. into the /home/openjdk/workspace folder.

There are a number of folders each representing the project you have been working on, in each folder there are atleast two to four hidden files and folders such as:

.project
.cproject
.classpath
.settings

Copy these over to the folders for each of the respective packages under the ecproject folder in the respective folders as shown below.

~/sources/jdk8_tl/jaxp/ecproject/
~/sources/jdk8_tl/jaxws/ecproject/
~/sources/jdk8_tl/jdk/make/eclipse/j2se/ecproject/
~/sources/jdk8_tl/jdk/make/eclipse/jconsole/ecproject/
~/sources/jdk8_tl/jdk/make/eclipse/jmx/ecproject/
~/sources/jdk8_tl/jdk/make/eclipse/swing/ecproject/
~/sources/jdk8_tl/langtools/make/eclipse/langtools/ecproject/

An archive of the list of files that were modified across the openJDK system can be found at (include .cproject, .project, build.xml, build.properties, etc… files):

Github repo with directory structure of files and folders (last updated 18/09/2012) for both the Old Build and the InfraBuild system

In the Infrabuild folder in the repo, two new projects namely openJDK (images) and openJDK (images VERBOSE) have been included to build the whole OpenJDK system via the InfraBuild system.

Here is a list of packages which were not built during the process, they all have the necessary make and build scripts in them:

awt2d
jarzip
jdwpgen
world

Please provide any feedback on areas that work differently for you. If you have fixed an issue also let us know so that we can update the information here.

This blog has been inspired by a couple of JUG members working on the OpenJDK project, below are links leading upto them. Their instructions and systemic approach have helped in the process:

Netbeans projects for OpenJDK – Martijn Verburg
Netbeans projects for OpenJDK – Andrii Rodionov
Hacking Hotspot in Eclipse – Roman Kennke
Adding include paths to your C/C++ project – Roland
Sachin Handiekar – Screen-shots and other improvements.
Girish Balakrishnan – for reviewing and feedback on the Hotspot instructions.
Jonathan Gibbons (from the OpenJDK team) – Thanks for reviewing the Langtools section.

A number of other online resources and videos have also helped in the process.