How to build Graal-enabled JDK8 on CircleCI?

Citation: feature image on the blog can be found on flickr and created by Luca Galli. The image in one of the below sections can be also found on flickr and created by fklv (Obsolete hipster).

---

The GraalVM compiler is a replacement to HotSpot’s server-side JIT compiler widely known as the C2 compiler. It is written in Java with the goal of better performance (among other goals) as compared to the C2 compiler. New changes starting with Java 9 mean that we can now plug in our own hand-written C2 compiler into the JVM, thanks to JVMCI. The researchers and engineers at Oracle Labs) have created a variant of JDK8 with JVMCI enabled which can be used to build the GraalVM compiler. The GraalVM compiler is open source and is available on GitHub (along with the HotSpot JVMCI sources) needed to build the GraalVM compiler). This gives us the ability to fork/clone it and build our own version of the GraalVM compiler.

In this post, we are going to build the GraalVM compiler with JDK8 on CircleCI. The resulting artifacts are going to be:

– JDK8 embedded with the GraalVM compiler, and
– a zip archive containing Graal & Truffle modules/components.

Note: we are not covering how to build the whole of the GraalVM suite in this post, that can be done via another post. Although these scripts can be used to that, and there exists a branch which contains the rest of the steps.

Why use a CI tool to build the GraalVM compiler?

Continuous integration (CI) and continuous deployment (CD) tools have many benefits. One of the greatest is the ability to check the health of the code-base. Seeing why your builds are failing provides you with an opportunity to make a fix faster. For this project, it is important that we are able to verify and validate the scripts required to build the GraalVM compiler for Linux and macOS, both locally and in a Docker container.

A CI/CD tool lets us add automated tests to ensure that we get the desired outcome from our scripts when every PR is merged. In addition to ensuring that our new code does not introduce a breaking change, another great feature of CI/CD tools is that we can automate the creation of binaries and the automatic deployment of those binaries, making them available for open source distribution.

Let’s get started

During the process of researching CircleCI as a CI/CD solution to build the GraalVM compiler, I learned that we could run builds via two different approaches, namely:

– A CircleCI build with a standard Docker container (longer build time, longer config script)
– A CircleCI build with a pre-built, optimised Docker container (shorter build time, shorter config script)

We will now go through the two approaches mentioned above and see the pros and cons of both of them.

Approach 1: using a standard Docker container

For this approach, CircleCI requires a docker image that is available in Docker Hub or another public/private registry it has access to. We will have to install the necessary dependencies in this available environment in order for a successful build. We expect the build to run longer the first time and, depending on the levels of caching, it will speed up.

To understand how this is done, we will be going through the CircleCI configuration file section-by-section (stored in .circleci/circle.yml), see config.yml in .circleci for the full listing, see commit df28ee7 for the source changes.

Explaining sections of the config file

The below lines in the configuration file will ensure that our installed applications are cached (referring to the two specific directories) so that we don’t have to reinstall the dependencies each time a build occurs:

    dependencies:
      cache_directories:
        - "vendor/apt"
        - "vendor/apt/archives"

We will be referring to the docker image by its full name (as available on http://hub.docker.com under the account name used – adoptopenjdk). In this case, it is a standard docker image containing JDK8 made available by the good folks behind the Adopt OpenJDK build farm. In theory, we can use any image as long as it supports the build process. It will act as the base layer on which we will install the necessary dependencies:

        docker:
          - image: adoptopenjdk/openjdk8:jdk8u152-b16

Next, in the pre-Install Os dependencies step, we will restore the cache, if it already exists, this may look a bit odd, but for unique key labels, the below implementation is recommended by the docs):

          - restore_cache:
              keys:
                - os-deps-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
                - os-deps-{{ arch }}-{{ .Branch }}

Then, in the Install Os dependencies step we run the respective shell script to install the dependencies needed. We have set this step to timeout if the operation takes longer than 2 minutes to complete (see docs for timeout):

          - run:
              name: Install Os dependencies
              command: ./build/x86_64/linux_macos/osDependencies.sh
              timeout: 2m

Then, in then post-Install Os dependencies step, we save the results of the previous step – the layer from the above run step (the key name is formatted to ensure uniqueness, and the specific paths to save are included):

          - save_cache:
              key: os-deps-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
              paths:
                - vendor/apt
                - vendor/apt/archives

Then, in the pre-Build and install make via script step, we restore the cache, if one already exists:

          - restore_cache:
              keys:
                - make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
                - make-382-{{ arch }}-{{ .Branch }}

Then, in the Build and install make via script step we run the shell script to install a specific version of make and it is set to timeout if step takes longer than 1 minute to finish:

          - run:
              name: Build and install make via script
              command: ./build/x86_64/linux_macos/installMake.sh
              timeout: 1m

Then, in the post Build and install make via script step, we save the results of the above action to the cache:

          - save_cache:
              key: make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
              paths:
                - /make-3.82/
                - /usr/bin/make
                - /usr/local/bin/make
                - /usr/share/man/man1/make.1.gz
                - /lib/

Then, we define environment variables to update JAVA_HOME and PATH at runtime. Here the environment variables are sourced so that we remember them for the next subsequent steps till the end of the build process (please keep this in mind):

          - run:
              name: Define Environment Variables and update JAVA_HOME and PATH at Runtime
              command: |
                echo '....'     <== a number of echo-es displaying env variable values
                source ${BASH_ENV}

Then, in the step to Display Hardware, Software, Runtime environment and dependency versions, as best practice we display environment-specific information and record it into the logs for posterity (also useful during debugging when things go wrong):

          - run:
              name: Display HW, SW, Runtime env. info and versions of dependencies
              command: ./build/x86_64/linux_macos/lib/displayDependencyVersion.sh

Then, we run the step to setup MX – this is important from the point of view of the GraalVM compiler (mx) is a specialised build system created to facilitate compiling and building Graal/GraalVM and  components):

          - run:
              name: Setup MX
              command: ./build/x86_64/linux_macos/lib/setupMX.sh ${BASEDIR}

Then, we run the important step to Build JDK JVMCI (we build the JDK with JVMCI enabled here) and timeout, if the process takes longer than 15 minutes without any output or if the process takes longer than 20 minutes in total to finish:

          - run:
              name: Build JDK JVMCI
              command: ./build/x86_64/linux_macos/lib/build_JDK_JVMCI.sh ${BASEDIR} ${MX}
              timeout: 20m
              no_output_timeout: 15m

Then, we run the step Run JDK JVMCI Tests, which runs tests as part of the sanity check after building the JDK JVMCI:

          - run:
              name: Run JDK JVMCI Tests
              command: ./build/x86_64/linux_macos/lib/run_JDK_JVMCI_Tests.sh ${BASEDIR} ${MX}

Then, we run the step Setting up environment and Build GraalVM Compiler, to set up the build environment with the necessary environment variables which will be used by the steps to follow:

          - run:
              name: Setting up environment and Build GraalVM Compiler
              command: |
                echo ">>>> Currently JAVA_HOME=${JAVA_HOME}"
                JDK8_JVMCI_HOME="$(cd ${BASEDIR}/graal-jvmci-8/ && ${MX} --java-home ${JAVA_HOME} jdkhome)"
                echo "export JVMCI_VERSION_CHECK='ignore'" >> ${BASH_ENV}
                echo "export JAVA_HOME=${JDK8_JVMCI_HOME}" >> ${BASH_ENV}
                source ${BASH_ENV}

Then, we run the step Build the GraalVM Compiler and embed it into the JDK (JDK8 with JVMCI enabled) which timeouts if the process takes longer than 7 minutes without any output or longer than 10 minutes in total to finish:

          - run:
              name: Build the GraalVM Compiler and embed it into the JDK (JDK8 with JVMCI enabled)
              command: |
                echo ">>>> Using JDK8_JVMCI_HOME as JAVA_HOME (${JAVA_HOME})"
                ./build/x86_64/linux_macos/lib/buildGraalCompiler.sh ${BASEDIR} ${MX} ${BUILD_ARTIFACTS_DIR}
              timeout: 10m
              no_output_timeout: 7m

Then, we run the simple sanity checks to verify the validity of the artifacts created once a build has been completed, just before archiving the artifacts:

          - run:
              name: Sanity check artifacts
              command: |
                ./build/x86_64/linux_macos/lib/sanityCheckArtifacts.sh ${BASEDIR} ${JDK_GRAAL_FOLDER_NAME}
              timeout: 3m
              no_output_timeout: 2m

Then, we run the step Archiving artifacts (means compressing and copying final artifacts into a separate folder) which timeouts if the process takes longer than 2 minutes without any output or longer than 3 minutes in total to finish:

          - run:
              name: Archiving artifacts
              command: |
                ./build/x86_64/linux_macos/lib/archivingArtifacts.sh ${BASEDIR} ${MX} ${JDK_GRAAL_FOLDER_NAME} ${BUILD_ARTIFACTS_DIR}
              timeout: 3m
              no_output_timeout: 2m

For posterity and debugging purposes, we capture the generated logs from the various folders and archive them:

          - run:
              name: Collecting and archiving logs (debug and error logs)
              command: |
                ./build/x86_64/linux_macos/lib/archivingLogs.sh ${BASEDIR}
              timeout: 3m
              no_output_timeout: 2m
              when: always
          - store_artifacts:
              name: Uploading logs
              path: logs/

Finally, we store the generated artifacts at a specified location – the below lines will make the location available on the CircleCI interface (we can download the artifacts from here):

          - store_artifacts:
              name: Uploading artifacts in jdk8-with-graal-local
              path: jdk8-with-graal-local/

Approach 2: using a pre-built optimised Docker container

For approach 2, we will be using a pre-built docker container, that has been created and built locally with all necessary dependencies, the docker image saved and then pushed to a remote registry for e.g. Docker Hub. And then we will be referencing this docker image in the CircleCI environment, via the configuration file. This saves us time and effort for running all the commands to install the necessary dependencies to create the necessary environment for this approach (see the details steps in Approach 1 section).

We expect the build to run for a shorter time as compared to the previous build and this speedup is a result of the pre-built docker image (we will see in the Steps to build the pre-built docker image section), to see how this is done). The additional speed benefit comes from the fact that CircleCI caches the docker image layers which in turn results in a quicker startup of the build environment.

We will be going through the CircleCI configuration file section-by-section (stored in .circleci/circle.yml) for this approach, see config.yml in .circleci for the full listing, see commit e5916f1 for the source changes.

Explaining sections of the config file

Here again, we will be referring to the docker image by it’s full name. It is a pre-built docker image neomatrix369/graalvm-suite-jdk8 made available by neomatrix369. It was built and uploaded to Docker Hub in advance before the CircleCI build was started. It contains the necessary dependencies for the GraalVM compiler to be built:

        docker:
          - image: neomatrix369/graal-jdk8:${IMAGE_VERSION:-python-2.7}
        steps:
          - checkout

All the sections below do the exact same tasks (and for the same purpose) as in Approach 1, see Explaining sections of the config file section.

Except, we have removed the below sections as they are no longer required for Approach 2:

    - restore_cache:
              keys:
                - os-deps-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
                - os-deps-{{ arch }}-{{ .Branch }}
          - run:
              name: Install Os dependencies
              command: ./build/x86_64/linux_macos/osDependencies.sh
              timeout: 2m
          - save_cache:
              key: os-deps-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
              paths:
                - vendor/apt
                - vendor/apt/archives
          - restore_cache:
              keys:
                - make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
                - make-382-{{ arch }}-{{ .Branch }}
          - run:
              name: Build and install make via script
              command: ./build/x86_64/linux_macos/installMake.sh
              timeout: 1m
          - save_cache:
              key: make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
              paths:
                - /make-3.82/
                - /usr/bin/make
                - /usr/local/bin/make
                - /usr/share/man/man1/make.1.gz

In the following section, I will go through the steps show how to build the pre-built docker image. It will involve running the bash scripts – ./build/x86_64/linux_macos/osDependencies.sh and ./build/x86_64/linux_macos/installMake.sh to install the necessary dependencies as part of building a docker image. And, finally pushing the image to Docker Hub (can be pushed to any other remote registry of your choice).

Steps to build the pre-built docker image

– Run build-docker-image.sh (see bash script source) which depends on the presence of Dockerfile (see docker script source). The Dockerfile does all the necessary tasks of running the dependencies inside the container i.e. runs the bash scripts ./build/x86_64/linux_macos/osDependencies.sh and ./build/x86_64/linux_macos/installMake.sh:

    $ ./build-docker-image.sh

– Once the image has been built successfully, run push-graal-docker-image-to-hub.sh after setting the USER_NAME and IMAGE_NAME (see source code) otherwise it will use the default values as set in the bash script:

    $ USER_NAME="[your docker hub username]" IMAGE_NAME="[any image name]" \
        ./push-graal-docker-image-to-hub.sh

CircleCI config file statistics: Approach 1 versus Approach 2

Areas of interest	Approach 1	Approach 2
Config file (full source list)	build-on-circleci	build-using-prebuilt-docker-image
Commit point (sha)	df28ee7	e5916f1
Lines of code (loc)	110 lines	85 lines
Source lines (sloc)	110 sloc	85 sloc
Steps (steps: section)	19	15
Performance (see Performance section)	Some speedup due to caching, but slower than Approach 2	Speed-up due to pre-built docker image, and also due to caching at different steps. Faster than Approach 1 Ensure DLC layering is enabled (its a paid feature)

What not to do?

Approach 1 issues

I came across things that wouldn’t work initially, but were later fixed with changes to the configuration file or the scripts:

• please make sure the .circleci/config.yml is always in the root directory of the folder
• when using the store_artifacts directive in the .circleci/config.yml file setting, set the value to a fixed folder name i.e. jdk8-with-graal-local/ – in our case, setting the path to ${BASEDIR}/project/jdk8-with-graal didn’t create the resulting artifact once the build was finished hence the fixed path name suggestion.
• environment variables: when working with environment variables, keep in mind that each command runs in its own shell hence the values set to environment variables inside the shell execution environment isn’t visible outside, follow the method used in the context of this post. Set the environment variables such that all the commands can see its required value to avoid misbehaviours or unexpected results at the end of each step.
• caching: use the caching functionality after reading about it, for more details on CircleCI caching refer to the caching docs. See how it has been implemented in the context of this post. This will help avoid confusions and also help make better use of the functionality provided by CircleCI.

Approach 2 issues

• Caching: check the docs when trying to use the Docker Layer Caching (DLC) option as it is a paid feature, once this is known the doubts about “why CircleCI keeps downloading all the layers during each build” will be clarified, for Docker Layer Caching details refer to docs. It can also clarify why in non-paid mode my build is still not as fast as I would like it to be.

General note:

• Light-weight instances: to avoid the pitfall of thinking we can run heavy-duty builds, check the documentation on the technical specifications of the instances. If we run the standard Linux commands to probe the technical specifications of the instance we may be misled by thinking that they are high specification machines. See the step that enlists the Hardware and Software details of the instance (see Display HW, SW, Runtime env. info and versions of dependencies section). The instances are actually Virtual Machines or Container like environments with resources like 2CPU/4096MB. This means we can’t run long-running or heavy-duty builds like building the GraalVM suite. Maybe there is another way to handle these kinds of builds, or maybe such builds need to be decomposed into smaller parts.
• Global environment variables: as each run line in the config.yml, runs in its own shell context, from within that context environment variables set by other executing contexts do not have access to these values. Hence in order to overcome this, we have adopted two methods:

• pass as variables as parameters to calling bash/shell scripts to ensure scripts are able to access the values in the environment variables
• use the source command as a run step to make environment variables accessible globally

End result and summary

We see the below screen (the last step i.e. Updating artifacts enlists where the artifacts have been copied), after a build has been successfully finished:

The artifacts are now placed in the right folder for download. We are mainly concerned about the jdk8-with-graal.tar.gz artifact.

Performance

Before writing this post, I ran multiple passes of both the approaches and jotted down the time taken to finish the builds, which can be seen below:

– Approach 1: standard CircleCI build (caching enabled)
– 13 mins 28 secs
– 13 mins 59 secs
– 14 mins 52 secs
– 10 mins 38 secs
– 10 mins 26 secs
– 10 mins 23 secs
– Approach 2: using pre-built docker image (caching enabled, DLC) feature unavailable)
– 13 mins 15 secs
– 15 mins 16 secs
– 15 mins 29 secs
– 15 mins 58 secs
– 10 mins 20 secs
– 9 mins 49 secs

Note: Approach 2 should show better performance when using a paid tier, as Docker Layer Caching) is available as part of this plan.

Sanity check

In order to be sure that by using both the above approaches we have actually built a valid JDK embedded with the GraalVM compiler, we perform the following steps with the created artifact:

– Firstly, download the jdk8-with-graal.tar.gz artifact from under the Artifacts tab on the CircleCI dashboard (needs sign-in):

– Then, unzip the .tar.gz file and do the following:

    tar xvf jdk8-with-graal.tar.gz

– Thereafter, run the below command to check the JDK binary is valid:

    cd jdk8-with-graal
    ./bin/java -version

– And finally check if we get the below output:

    openjdk version "1.8.0-internal"
    OpenJDK Runtime Environment (build 1.8.0-internal-jenkins_2017_07_27_20_16-b00)
    OpenJDK 64-Bit Graal:compiler_ab426fd70e30026d6988d512d5afcd3cc29cd565:compiler_ab426fd70e30026d6988d512d5afcd3cc29cd565 (build 25.71-b01-internal-jvmci-0.46, mixed mode)

– Similarly, to confirm if the JRE is valid and has the GraalVM compiler built in, we do this:

    ./bin/jre/java -version

– And check if we get a similar output as above:

    openjdk version "1.8.0-internal"
    OpenJDK Runtime Environment (build 1.8.0-internal-jenkins_2017_07_27_20_16-b00)
    OpenJDK 64-Bit Graal:compiler_ab426fd70e30026d6988d512d5afcd3cc29cd565:compiler_ab426fd70e30026d6988d512d5afcd3cc29cd565 (build 25.71-b01-internal-jvmci-0.46, mixed mode)

With this, we have successfully built JDK8 with the GraalVM compiler embedded in it and also bundled the Graal and Truffle components in an archive file, both of which are available for download via the CircleCI interface.

Note: you will notice that we do perform sanity checks of the binaries built just before we pack them into compressed archives, as part of the build steps (see bottom section of CircleCI the configuration files section).

Nice badges!

We all like to show-off and also like to know the current status of our build jobs. A green-colour, build status icon is a nice indication of success, which looks like the below on a markdown README page:

We can very easily embed both of these status badges displaying the build status of our project (branch-specific i.e. master or another branch you have created) built on CircleCI (see docs) on how to do that).

Conclusions

We explored two approaches to build the GraalVM compiler using the CircleCI environment. They were good experiments to compare performance between the two approaches and also how we can do them with ease. We also saw a number of things to avoid or not to do and also saw how useful some of the CircleCI features are. The documentation and forums do good justice when trying to make a build work or if you get stuck with something.

Once we know the CircleCI environment, it’s pretty easy to use and always gives us the exact same response (consistent behaviour) every time we run it. Its ephemeral nature means we are guaranteed a clean environment before each run and a clean up after it finishes. We can also set up checks on build time for every step of the build, and abort a build if the time taken to finish a step surpasses the threshold time-period.

The ability to use pre-built docker images coupled with Docker Layer Caching on CircleCI can be a major performance boost (saves us build time needed to reinstall any necessary dependencies at every build). Additional performance speedups are available on CircleCI, with caching of the build steps – this again saves build time by not having to re-run the same steps if they haven’t changed.

There are a lot of useful features available on CircleCI with plenty of documentation and everyone on the community forum are helpful and questions are answered pretty much instantly.

Next, let’s build the same and more on another build environment/build farm – hint, hint, are you think the same as me? Adopt OpenJDK build farm)? We can give it a try!

Thanks and credits to Ron Powell from CircleCI and Oleg Šelajev from Oracle Labs for proof-reading and giving constructive feedback. 

Please do let me know if this is helpful by dropping a line in the comments below or by tweeting at @theNeomatrix369, and I would also welcome feedback, see how you can reach me, above all please check out the links mentioned above.

Useful resources

– Links to useful CircleCI docs
– About Getting started | Videos
– About Docker
– Docker Layer Caching
– About Caching
– About Debugging via SSH
– CircleCI cheatsheet
– CircleCI Community (Discussions)
– Latest community topics
– CircleCI configuration and supporting files
– Approach 1: https://github.com/neomatrix369/awesome-graal/tree/build-on-circleci (config file and other supporting files i.e. scripts, directory layout, etc…)
– Approach 2: https://github.com/neomatrix369/awesome-graal/tree/build-on-circleci-using-pre-built-docker-container (config file and other supporting files i.e. scripts, directory layout, etc…)
– Scripts to build Graal on Linux, macOS and inside the Docker container
– Truffle served in a Holy Graal: Graal and Truffle for polyglot language interpretation on the JVM
– Learning to use Wholly GraalVM!
– Building Wholly Graal with Truffle!

Building Wholly Graal with Truffle!

Citation: credits to the feature image go to David Luders and reused under a CC license, the original image can be found on this Flickr page.

Introduction

It has been some time, since the two posts [1][2] on Graal/GraalVM/Truffle, and a general request was when are you going to write something about “how to build” this awesome thing called Graal. Technically, we will be building HotSpot’s C2 compiler (look for C2 in the glossary list) replacement, called Graal. This binary is different from the  GraalVM suite you download from OTN via http://graalvm.org/downloads.

I wasn’t just going to stop at the first couple of posts on this technology. In fact, one of the best ways to learn and get an in-depth idea about any tech work, is to know how to build it.

Getting Started

Building JVMCI for JDK8, Graal and Truffle is fairly simple, and the instructions are available on the graal repo. We will be running them on both the local (Linux, MacOS) and container (Docker) environments. To capture the process as-code, they have been written in bash, see https://github.com/neomatrix369/awesome-graal/tree/master/build/x86_64/linux_macos.

During the process of writing the scripts and testing them on various environments, there were some issues, but these were soon resolved with the help members of the Graal team — thanks Doug.

Running scripts

Documentation on how to run the scripts are provided in the README.md on awesome-graal. For each of the build environments they are merely a single command:

Linux & MacOS

$ ./local-build.sh

$ RUN_TESTS=false           ./local-build.sh

$ OUTPUT_DIR=/another/path/ ./local-build.sh

Docker

$ ./docker-build.sh

$ DEBUG=true                ./docker-build.sh

$ RUN_TESTS=false           ./docker-build.sh

$ OUTPUT_DIR=/another/path/ ./docker-build.sh

Both the local and docker scripts pass in the environment variables i.e. RUN_TESTS and OUTPUT_DIR to the underlying commands. Debugging the docker container is also possible by setting the DEBUG environment variable.

For a better understanding of how they work, best to refer to the local and docker scripts in the repo.

Build logs

I have provided build logs for the respective environments in the build/x86_64/linux_macos  folder in https://github.com/neomatrix369/awesome-graal/

Once the build is completed successfully, the following messages are shown:

[snipped]

>> Creating /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal from /path/to/awesome-graal/build/x86_64/linux/graal-jvmci-8/jdk1.8.0_144/linux-amd64/product
Copying /path/to/awesome-graal/build/x86_64/linux/graal/compiler/mxbuild/dists/graal.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/jvmci
Copying /path/to/awesome-graal/build/x86_64/linux/graal/compiler/mxbuild/dists/graal-management.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/jvmci
Copying /path/to/awesome-graal/build/x86_64/linux/graal/sdk/mxbuild/dists/graal-sdk.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/boot
Copying /path/to/awesome-graal/build/x86_64/linux/graal/truffle/mxbuild/dists/truffle-api.jar to /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal/jre/lib/truffle

>>> All good, now pick your JDK from /path/to/awesome-graal/build/x86_64/linux/jdk8-with-graal :-)

Creating Archive and SHA of the newly JDK8 with Graal & Truffle at /home/graal/jdk8-with-graal
Creating Archive jdk8-with-graal.tar.gz
Creating a sha5 hash from jdk8-with-graal.tar.gz
jdk8-with-graal.tar.gz and jdk8-with-graal.tar.gz.sha256sum.txt have been successfully created in the /home/graal/output folder.

Artifacts

All the Graal and Truffle artifacts are created in the graal/compiler/mxbuild/dists/ folder and copied to the newly built jdk8-with-graal folder, both of these will be present in the folder where the build.sh script resides:

jdk8-with-graal/jre/lib/jvmci/graal.jar
jdk8-with-graal/jre/lib/jvmci/graal-management.jar
jdk8-with-graal/jre/lib/boot/graal-sdk.jar
jdk8-with-graal/jre/lib/truffle/truffle-api.jar

In short, we started off with vanilla JDK8 (JAVA_HOME) and via the build script created an enhanced JDK8 with Graal and Truffle embedded in it. At the end of a successful build process, the script will create a .tar.gz archive file in the jdk8-with-graal-local folder, alongside this file you will also find the sha5 hash of the archive.

In case of a Docker build, the same folder is called jdk8-with-graal-docker and in addition to the above mentioned files, it will also contain the build logs.

Running unit tests

Running unit tests is a simple command:

mx --java-home /path/to/jdk8 unittest

This step should follow the moment we have a successfully built artifact in the jdk8-with-graal-local folder. The below messages indicate a successful run of the unit tests:

>>>> Running unit tests...
Warning: 1049 classes in /home/graal/mx/mxbuild/dists/mx-micro-benchmarks.jar skipped as their class file version is not supported by FindClassesByAnnotatedMethods
Warning: 401 classes in /home/graal/mx/mxbuild/dists/mx-jacoco-report.jar skipped as their class file version is not supported by FindClassesByAnnotatedMethods
WARNING: Unsupported class files listed in /home/graal/graal-jvmci-8/mxbuild/unittest/mx-micro-benchmarks.jar.jdk1.8.excludedclasses
WARNING: Unsupported class files listed in /home/graal/graal-jvmci-8/mxbuild/unittest/mx-jacoco-report.jar.jdk1.8.excludedclasses
MxJUnitCore
JUnit version 4.12
............................................................................................
Time: 5.334

OK (92 tests)

JDK differences

So what have we got that’s different from the JDK we started with. If we compare the boot JDK with the final JDK here are the differences:

Combination of diff between $JAVA_HOME and jdk8-with-graal and meld will give the above:

diff -y --suppress-common-lines $JAVA_HOME jdk8-with-graal | less

meld $JAVA_HOME ./jdk8-with-graal

Note: $JAVA_HOME points to your JDK8 boot JDK.

Build execution time

The build execution time was captured on both Linux and MacOS and there was a small difference between running tests and not running tests:

Running the build with or without tests on a quad-core, with hyper-threading:

 real 4m4.390s
 user 15m40.900s
 sys 1m20.386s
 ^^^^^^^^^^^^^^^
 user + sys = 17m1.286s (17 minutes 1.286 second)

Similar running the build with and without tests on a dual-core MacOS, with 4GB RAM, SSD drive, differs little:

 real 9m58.106s
 user 18m54.698s 
 sys 2m31.142s
 ^^^^^^^^^^^^^^^ 
 user + sys = 21m25.84s (21 minutes 25.84 seconds)

Disclaimer: these measurements can certainly vary across the different environments and configurations. If you have a more accurate way to benchmark such running processes, please do share back.

Summary

In this post, we saw how we can build Graal and Truffle for JDK8 on both local and container environments.

The next thing we will do is build them on a build farm provided by Adopt OpenJDK. We will be able to run them across multiple platforms and operating systems, including building inside docker containers. This binary is different from the GraalVM suite you download from OTN via http://graalvm.org/downloads, hopefully we will be able to cover GraalVM in a future post.

Thanks to Julien Ponge for making his build script available for re-use and the Graal team for supporting during the writing of this post.

Feel free to share your feedback at @theNeomatrix369. Pull requests with improvements and best-practices are welcome at https://github.com/neomatrix369/awesome-graal.

My first couple of months at Codurance

Some background

Some of you may know me from the various meetups in the city, especially my attendance at a number of LJC and LSCC meetup events. Attending these events I learnt about various conferences like Devoxx, SoCraTes, JAX London,  Java2Days, OpenFest, and I ended up attending and later presenting on various topic including Adopt OpenJDK.

During this time I met a lot of people with various levels of experience and my interest and urge to learn more about the Java/JVM platform, Code Quality, Software Design, XP Practices, Software Craftsmanship, etc…, were on the rise and saw no end. And whilst attending these events I came across Sandro and Mash, who were in those days hosting LSCC events. I went to many of LSCC events, especially liked the hands-on sessions (which are still my favourite).

I also noticed that many things I learnt at such events and conferences wouldn’t always be immediately recognised or accepted at the workplace. And moving to another work environment didn’t always solve this problem fully. I found that I wasn’t learning what I wanted from my peers and the things I learnt from the community I couldn’t apply at work. Besides very few were really in tuned with what the community was about. So one fine day I decided to take charge of my career and make a serious decision and take up the Apprenticeship program offered by Codurance and go through the process.

I was urged to go this way after being inspired by Sandro’s book: The Software Craftsman, attending all the SoCraTes UK conferences, and meeting with developers who valued and took pride of their work namely their craft.

I was urged to go this way after being inspired by Sandro’s book: The Software Craftsman, attending all the SoCraTes UK conferences, and meeting with developers who valued and took pride of their work namely their craft.

Where we are just now

It’s now been nearly two months since I have been working for Codurance, a formidable force. And so it’s also about time that I share my experiences with my fellow mates and the community around me.

During my first few weeks at Codurance, I have been busy learning various things that have been chalked out for becoming a craftsman.

When working on a kata or learning a concept, we paired or did what is known as ‘mob programming’ along with other apprentices and craftsmen. And most of the time used the pomodoro technique. Time boxing our work in intervals is something done both in groups and working individually. We would have a lot of discussions and retrospectives after working on a problem or writing some code from scratch.

Structure of my program

We used an internal tool based on the concept of Impact Mapping. I soon got interested in it when I saw my colleague Franzi (who is now a craftswoman) had used it to plan out her Apprenticeship route. Such a tool helps map out our goals and the tasks we need to perform to achieve it. And this can differ from person-to-person, depending on what they want to work on (driven by the Apprentice).

My mentor and other craftsmen reviewed them to get an idea of what I wanted to achieve for myself. And then its up to me to apply my own drive and perseverance to achieve the individual stories. My mentor and I meet and talk informally on a regular basis, many times pairing on a kata or a project or on the white board trying to get my head around a concept.

Days in the life of an Apprentice

I found the working hours quite flexible, remote working is also an option (when you are on the bench or if the client allows, if you are in a project). Our co-founders are understanding and compassionate about our individual situations.

Meetings are at their minimum, except for a weekly Apprentices meeting (run by an Apprentice and guided by at least one Craftsperson) and a bi-monthly company-wide catchup.

The Apprentices meetings are full of fun — we are accompanied by at least one Craftsperson, who disperses their knowledge and experience from a wide variety of topics designed to help us in the journey and fill the gaps in our knowledge and experience.

A bi-monthly catchup involves sharing of knowledge via lightning talks, discussions and pairing sessions on pet projects over pizzas and beer (and of course veggies and non-alcoholic beverages for the teetotalers).

Katas, code reviews, mob programming and projects make up a learning week – all of these done individually or when pairing with another.

Katas

On a daily basis I have worked on different katas or try to solve the same kata in various different ways (using different testing and refactoring approaches). This in turn gave me better insights into designing and refactoring techniques. Trying to solve the same problem in different ways has a positive impact on our problem solving skills especially when writing code. In my case I also learnt how to use the different libraries and methods to write tests. I would like to cite Samir, thanks to you, for the suggesting this approach during the first week of my Apprenticeship.

Code reviews

Just last week we did a group code review and time-boxed ourselves, performed a retrospective at the end of each interval and ensured we delivered a good chunk of the feedback before close of play. Such regular code review exercises are helping all of us learn about how to code better as we are not only learning from feedback from the tools we used, but also through exchange of feedback from our peers who were involved in the group code review session.

Software Design, Specification Gathering & Communication

Recently we had an interesting mob-programming session where we were trying to model and write a game. At the end of the session, we had a retrospective, discussing the things we did well and didn’t do well. Each of the apprentices and craftsmen were performing a specific role i.e. Developer, Domain Expert, etc… We learnt in retrospective, about areas where we could have done better and should focus on. That any test written gives immediate feedback about how well we have understood the domain and if we were taking the right approach. Why a certain approach when starting a project is more advantageous than another approach. What questions to ask and why it is important to ask the right questions to the domain expert or to give the right level of information to another developer and vice-versa. Sandro has described this process in detail in his blog post recently.

Fun, socialising and sharing

I found our office environment to be conducive to learning, sharing and collaboration. We even have a pairing rota that we use from time-to-time to record or suggest pairing sessions during the week.

We share links to events, conferences, tweets, interesting articles, videos, blog posts, etc… via slack, document discussions and brain dumps via Google doc, huddles during lunch- and tea- breaks to talk about anything we are working on. Thanks to the library of printed and digital books to our disposal, the huge collection of blog posts and videos on our site.

The apprentices and some craftsmen have collectively started a social event which of course happens every Friday, sometimes it’s dinner at a nearby restaurant, while at other times an indoor movie over snacks and drinks at our office premises.

It is worthwhile and that’s why we are here

It is a privilege to be able to work alongside very experienced craftsmen from our industry. We are very lucky and thankful to have the opportunity to be guided and mentored by talented and like minded developers.

This is my first job where the company has a completely flat hierarchy and where we share similar values.

Closing note

Work is fun and learning is enjoyable when we love what we do and are amongst friends with similar goals and aspirations.

Thank you for taking the time to read this post and I hope it was interesting. Looking forward to write more and share such experiences in future posts.

Many thanks to Sandro, Tomaz, Alex, Franzi and David for all the feedback provided for this blog post.

 

Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2

…continuing from Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2.

I ran into a number of issues and you can see from my commits, I pulled myself out of it, but to run this Dockerfile from the command-line I used this instruction:

$ docker build -t neomatrix369/openjdk9 .

you can also do it using the below if you have not set your permissions:

$ sudo docker build -t neomatrix369/openjdk9 .

and get the below (summarised) output:

Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon 
Step 0 : FROM phusion/baseimage:latest
 ---> 5a14c1498ff4
Step 1 : MAINTAINER Mani Sarkar (from @adoptopenjdk)
 ---> Using cache
 ---> 95e30b7f52b9
Step 2 : RUN apt-get update &&   apt-get install -y     libxt-dev zip pkg-config libX11-dev libxext-dev     libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev &&   rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 1ea3bbb15c2d
Step 3 : RUN apt-get update
 ---> Using cache
 ---> 6c3938f4d23d
Step 4 : RUN apt-get install -y mercurial ca-certificates-java build-essential
 ---> Using cache
 ---> e3f99b5a3bd3
Step 5 : RUN cd /tmp &&   hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 &&   cd openjdk9 &&   sh ./get_source.sh
 ---> Using cache
 ---> 26cfaf16b9fa
Step 6 : RUN apt-get install -y wget &&   wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
 ---> Using cache
 ---> 696889250fed
Step 7 : RUN tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt
 ---> Using cache
 ---> c25cc9201c1b
Step 8 : RUN cd /tmp/openjdk9 &&   bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45
 ---> Using cache
 ---> 4e425de379e6
Step 9 : RUN cd /tmp/openjdk9 &&   make clean images
 ---> Using cache
 ---> 2d9e17c870be
Step 10 : RUN cd /tmp/openjdk9 &&   cp -a build/linux-x86_64-normal-server-release/images/jdk     /opt/openjdk9
 ---> Using cache
 ---> 9250fac9b500
Step 11 : RUN cd /tmp/openjdk9 &&   find /opt/openjdk9 -type f -exec chmod a+r {} + &&   find /opt/openjdk9 -type d -exec chmod a+rx {} +
 ---> Using cache
 ---> d0c597d045d4
Step 12 : ENV PATH /opt/openjdk9/bin:$PATH
 ---> Using cache
 ---> 3965c3e47855
Step 13 : ENV JAVA_HOME /opt/openjdk9
 ---> Using cache
 ---> 5877e8efd939
Successfully built 5877e8efd939

The above action creates an image which is stored in your local repository (use docker images to enlist the images in the repo). If you want to load the image into a container, and access the files it has built or see anything else, do the below:

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 /bin/bash

this will take you to a bash prompt into the container and you can run any of your linux commands and access the file system.

Explaining docker run

$ sudo docker run -it --name openjdk9 neomatrix369/openjdk9 java -version

will show you this

openjdk version "1.9.0-internal"
OpenJDK Runtime Environment (build 1.9.0-internal-_2015_06_04_06_46-b00)
OpenJDK 64-Bit Server VM (build 1.9.0-internal-_2015_06_04_06_46-b00, mixed mode)

Here’s a breakdown of the docker run command:

docker run	The command to create and start a new Docker container.
-it	To run in interactive mode, so you can see the after running the container.
neomatrix369/openjdk9	This is a reference to the image tag by name (which we created above).
java -version	Runs the java command asking its version, inside the containing, which is assisted by the two environment variables PATH and JAVA_HOME which was set in the Dockerfile above.

Footnotes

You might have noticed I grouped very specific instructions with each step, especially the RUN commands, its because, each time I got one of these wrong, it would re-execute the step again, including the steps that ran fine and didn’t need re-executing. Not only is this unnecessary its not using our resources efficiently which is what Docker brings us. So any addition, edition or deletion to any step will only result in that step being executed, and not the other steps that are fine.

So one of the best practises is to keep the steps granular enough and pre-load files and data beforehand and give it to docker. It has amazing caching and archiving mechanisms built in.

Save our work

As we know if we do not save the container into the image, our changes are lost.

If I didn’t use the docker build command I used earlier I could have, after the build process was completed and image created, used the below command:

$ sudo docker commit [sha of the image] neomatrix369/openjdk9

Sharing your docker image on Docker hub

Once you are happy with your changes, and want to share it with community at large, do the below:

$ sudo docker push neomatrix369/openjdk9

and you will see these depending on which of your layers have been found in the repo and which ones are new (this one is an example snapshot of the process):

The push refers to a repository [neomatrix369/openjdk9] (len: 1)
5877e8efd939: Image already exists 
3965c3e47855: Image already exists 
d0c597d045d4: Image already exists 
9250fac9b500: Image already exists 
2d9e17c870be: Buffering to Disk
.
.
.

There is plenty of room for development and improvement of this Docker script. So happy hacking and would love to hear your feedback or contributions from you.

BIG Thanks

Big thanks to the below two who proof-read my post and added value to it, whilst enjoying the #Software #Craftsmanship developer community (organised and supported by @LSCC):
Oliver Nautsch – @ollispieps (JUG Switzerland)
Amir Bazazi (@Codurance) – @amirbazazi

Special thanks to Roberto Cortez (@radcortez) for your Docker posts, these inspired and helped me write my first Docker post.

Resources

[1] Docker
[2] Get into Docker – A Guide for Total Newbies
[3] Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images
[4] Docker posts on Voxxed
[5] OpenJDK
[6] Building OpenJDK
[7] Building OpenJDK on Linux, MacOs and Windows
[8] Virtual Machines (OpenJDK)
[9] Build your own OpenJDK
[10] Vagrant script (OpenJDK)
[11] YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it
[12] Dockerfile on github
[13] Adopt OpenJDK: Getting Started Kit
[14] London Java Community

Why not build #OpenJDK 9 using #Docker ? – Part 1 of 2

Introduction

I think I have joined the Docker [1] party a bit late but that means by now everyone knows what Docker is and all the other basic fundamentals which I can very well skip, but if you are still interested, please check these posts Get into Docker – A Guide for Total Newbies [2] and Docker for Total Newbies Part 2: Distribute Your Applications with Docker Images [3]. And if you still want to know more about this widely spoken topic, check out these Docker posts on Voxxed [4].

Why ?

Since everyone has been doing some sort of provisioning or spinning up of dev or pre-prod or test environments using Docker [1] I decided to do the same but with my favourite project i.e. OpenJDK [5].

So far you can natively build OpenJDK [6] across Linux, MacOs and Windows [7], or do the same things via virtual machines or vagrant instances, see more on then via these resources Virtual Machines, [8] Build your own OpenJDK [9] and this vagrant script [10]. All part of the Adopt OpenJDK initiative lead by London Java Community [14] and supported by JUGs all over the world.

Requirements

Most parts of post is for those using Linux distributions (this one was created on Ubuntu 14.04). Linux, MacOS and Windows users please refer to Docker‘s  Linux, MacOS and Windows instructions respectively.

Hints: MacOS and Windows users will need to install Boot2Docker and remember to run the below two commands (and check your Docker host environment variables):

$ boot2docker init
$ boot2docker up 
$ boot2docker shellinit 

For the MacOS, if the above throw FATA[…] error messages, please try the below:

$ sudo boot2docker init
$ sudo boot2docker up 
$ sudo boot2docker shellinit 

For rest of the details please refer to the links provided above. Once you have the above in place for the Windows or MacOS platform, by merely executing the Dockerfile using the docker build and docker run commands you can create / update a container and run it respectively.

*** Please refer to the above links and ensure Docker works for you for the above platforms – try out tutorials or steps proving that Docker run as expected before proceeding further. ***

Building OpenJDK 9 using Docker

Now I will show you how to do the same things as mentioned above using Docker.

So I read the first two resource I shared so far (and wrote the last ones). So lets get started, and I’m going to walk you through what the Dockerfile looks like, as I take you through each section of the Dockerfile code.

*** Please note the steps below are not meant to be executed on your command prompty, they form an integral part of the Dockerfile which you can download from here at the end of this post. ***

You have noticed unlike everyone else I have chosen a different OS image i.e. phusion/baseimage, why? Read YOUR DOCKER IMAGE MIGHT BE BROKEN without you knowing it [11], to learn more about it.

FROM phusion/baseimage:latest

Each of the RUN steps below when executed becomes a Docker layer in isolation and gets assigned a SHA like this i.e. 95e30b7f52b9.

RUN \
  apt-get update && \
  apt-get install -y \
    libxt-dev zip pkg-config libX11-dev libxext-dev \
    libxrender-dev libxtst-dev libasound2-dev libcups2-dev libfreetype6-dev && \
  rm -rf /var/lib/apt/lists/*

The base image is updated and a number of dependencies are installed i.e. Mercurial (hg) and build-essential.

RUN \
  apt-get update && \
  apt-get install -y mercurial ca-certificates-java build-essential

Clone the OpenJDK 9 sources and download the latest sources from mercurial. You will notice that each of these steps are prefixed by this line cd /tmp &&, this is because each instruction is run in its own layer, as if it does not remember where it was when the previous instruction was run. Nothing to worry about, all your changes are still intact in the container.

RUN \
  cd /tmp && \
  hg clone http://hg.openjdk.java.net/jdk9/jdk9 openjdk9 && \
  cd openjdk9 && \
  sh ./get_source.sh

Install only what you need when you need them, see below I downloaded wget and then the jdk binary. I also learnt how to use wget by passing the necessary params and headers to make the server give us the binary we request. Finally un-tar the file using the famous tar command.

RUN \
  apt-get install -y wget && \
  wget --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" \ 
http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

RUN \
  tar zxvf jdk-8u45-linux-x64.tar.gz -C /opt

Run configure with the famous –with-boot-jdk=/opt/jdk1.8.0_45 to set the bootstrap jdk to point to jdk1.8.0_45.

RUN \
  cd /tmp/openjdk9 && \
  bash ./configure --with-cacerts-file=/etc/ssl/certs/java/cacerts --with-boot-jdk=/opt/jdk1.8.0_45

Now run the most important command:

RUN \  
  cd /tmp/openjdk9 && \
  make clean images

Once the build is successful, the artefacts i.e. jdk and jre images are created in the build folder.

RUN \  
  cd /tmp/openjdk9 && \
  cp -a build/linux-x86_64-normal-server-release/images/jdk \
    /opt/openjdk9

Below are some chmod ceremonies across the files and directories in the openjdk9 folder.

RUN \  
  cd /tmp/openjdk9 && \
  find /opt/openjdk9 -type f -exec chmod a+r {} + && \
  find /opt/openjdk9 -type d -exec chmod a+rx {} +

Two environment variable i.e. PATH and JAVA_HOME are created with the respective values assigned to them.

ENV PATH /opt/openjdk9/bin:$PATH
ENV JAVA_HOME /opt/openjdk9

You can find the entire source for the entire Dockerfile on github [12].

…more of this in the next post, Why not build #OpenJDK 9 using #Docker ? – Part 2 of 2, we will use the docker build, docker run commands and some more docker stuff.

How is Java / JVM built ? Adopt OpenJDK is your answer!

Introduction & history

As some of you may already know, starting with Java 7, OpenJDK is the Reference Implementation (RI) to Java. The below time line gives you an idea about the history of OpenJDK:

OpenJDK history (2006 till date)

See Adopt OpenJDK – the past present and future [1] for some more details.

If you have wondered about the JDK or JRE binaries that you download from vendors like Oracle, Red Hat, etcetera, then the clue is that these all stem from OpenJDK. Each vendor then adds some extra artefacts that are not open source yet due to security, proprietary or other reasons.

What is OpenJDK made of ?

OpenJDK is made up of a number of repositories, namely corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn. Between OpenjJDK8 and OpenJDK9 there have been no new repositories introduced, but lots of new changes and restructuring, primarily due to Jigsaw – the modularisation of Java itself [2] [3] [4] [5].

repo composition, language breakdown (metrics are estimated)

Recent history

OpenJDK Build Benchmarks – build-infra (Nov 2011) by Fredrik Öhrström, ex-Oracle, OpenJDK hero!

Fredrik Öhrström visited the LJC [16] in November 2011 where he showed us how to build OpenJDK on the three major platforms, and also distributed a four page leaflet with the benchmarks of the various components and how long they took to build. The new build system and the new makefiles are a result of the build system being re-written (build-infra).

Below are screen-shots of the leaflets, a good reference to compare our journey:

The New Build System [23]

The New Makefiles [24]

Build Benchmarks page 1 [25]

Build Benchmarks page 2 [26]

How has Java the language and platform built over the years ?

Java is built by bootstrapping an older (previous) version of Java – i.e. Java is built using Java itself as its building block. Where older components are put together to create a new component which in the next phase becomes the building block. A good example of bootstrapping can be found at Scheme from Scratch [6] or even on Wikipedia [7].

OpenJDK8 [8] is compiled and built using JDK7, similarly OpenJDK9 [9] is compiled and built using JDK8. In theory OpenJDK8 can be compiled using the images created from OpenJDK8, similarly for OpenJDK9 using OpenJDK9. Using a process called bootcycle images – a JDK image of OpenJDK is created and then using the same image, OpenJDK is compiled again, which can be accomplished using a make command option:

$ make bootcycle-images       # Build images twice, second time with newly built JDK

make offers a number of options under OpenJDK8 and OpenJDK9, you can build individual components or modules by naming them, i.e.

$ make [component-name] | [module-name]

or even run multiple build processes in parallel, i.e.

$ make JOBS=<n>                 # Run <n> parallel make jobs

Finally install the built artefact using the install option, i.e.

$ make install

Some myths busted

OpenJDK or Hotspot to be more specific isn’t completely written in C/C++, a good part of the code-base is good ‘ole Java (see the composition figure above). So you don’t have to be a hard-core developer to contribute to OpenJDK. Even the underlying C/C++ code code-base isn’t scary or daunting to look at. For example here is an extract of a code snippet from vm/memory/universe.cpp in the HotSpot repo –

http://hg.openjdk.java.net/jdk6/jdk6/hotspot/raw-file/a541ca8fa0e3/src/share/vm/memory/universe.cpp [10]:

.
.
.
Universe::initialize_heap()

if (UseParallelGC) {
    #ifndef SERIALGC
    Universe::_collectedHeap = new ParallelScavengeHeap();
    #else // SERIALGC
        fatal("UseParallelGC not supported in this VM.");
    #endif // SERIALGC

} else if (UseG1GC) {
    #ifndef SERIALGC
    G1CollectorPolicy* g1p = new G1CollectorPolicy();
    G1CollectedHeap* g1h = new G1CollectedHeap(g1p);
    Universe::_collectedHeap = g1h;
    #else // SERIALGC
        fatal("UseG1GC not supported in java kernel vm.");
    #endif // SERIALGC

} else {
    GenCollectorPolicy* gc_policy;

    if (UseSerialGC) {
        gc_policy = new MarkSweepPolicy();
    } else if (UseConcMarkSweepGC) {
        #ifndef SERIALGC
        if (UseAdaptiveSizePolicy) {
            gc_policy = new ASConcurrentMarkSweepPolicy();
        } else {
            gc_policy = new ConcurrentMarkSweepPolicy();
        }
        #else // SERIALGC
            fatal("UseConcMarkSweepGC not supported in this VM.");
        #endif // SERIALGC
    } else { // default old generation
        gc_policy = new MarkSweepPolicy();
    }

    Universe::_collectedHeap = new GenCollectedHeap(gc_policy);
}
.
.
.

(please note that the above code snippet might have changed since published here)

The things that appears clear from the above code-block are, we are looking at how pre-compiler notations are used to create Hotspot code that supports a certain type of GC i.e. Serial GC or Parallel GC. Also the type of GC policy is selected in the above code-block when one or more GC switches are toggled i.e. UseAdaptiveSizePolicy when enabled selects the Asynchronous Concurrent Mark and Sweep policy. In case of either Use Serial GC or Use Concurrent Mark Sweep GC are not selected, then the GC policy selected is Mark and Sweep policy. All of this and more is pretty clearly readable and verbose, and not just nicely formatted code that reads like English.

Further commentary can be found in the section called Deep dive Hotspot stuff in the Adopt OpenJDK Intermediate & Advance experiences [11] document.

Steps to build your own JDK or JRE

Earlier we mentioned about JDK and JRE images – these are no longer only available to the big players in the Java world, you and I can build such images very easily. The steps for the process have been simplified, and for a quick start see the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents. For detailed version of the same steps, please see the Adopt OpenJDK home page [13]. Basically building a JDK image from the OpenJDK code-base boils down to the below commands:

(setup steps have been made brief and some commands omitted, see links above for exact steps)

 $ hg clone http://hg.openjdk.java.net/jdk8/jdk8 jdk8  (a)...OpenJDK8

or

$ hg clone http://hg.openjdk.java.net/jdk9/jdk9 jdk9  (a)...OpenJDK9

$ ./get_source.sh                                     (b)

$ bash configure                                      (c)

$ make clean images                                   (d)

(setup steps have been made brief and some commands omitted, see links above for exact steps)

To explain what is happening at each of the steps above:

(a) We clone the openjdk mercurial repo just like we would using git clone ….

(b) Once we have step (a) completed, we change into the folder created, and run the get_source.sh command, which is equivalent to a git fetch or a git pull, since the step (a) only brings down base files and not all of the files and folders.

(c) Here we run a script that checks for and creates the configuration needed to do the compile and build process

(d) Once step (c) is success we perform a complete compile, build and create JDK and JRE images from the built artefacts

As you can see these are dead-easy steps to follow to build an artefact or JDK/JRE images [step (a) needs to be run only once].

Benefits

– contribute to the evolution and improvement of the Java the language & platform

– learn about the internals of the language and platform

– learn about the OS platform and other technologies whilst doing the above

– get involved in F/OSS projects

– stay on top the latest changes in the Java / JVM sphere

– knowledge and experience that helps professionally but also these are not readily available from other sources (i.e. books, training, work-experience, university courses, etcetera).

– advancement in career

– personal development (soft skills and networking)

Contribute

Join the Adopt OpenJDK [13] and Betterrev [15] projects and contribute by giving us feedback about everything Java including these projects. Join the Adoption Discuss mailing list [14] and other OpenJDK related mailing lists to start with, these will keep you updated with latest progress and changes to OpenJDK. Fork any of the projects you see and submit changes via pull-requests.

Thanks and support

Adopt OpenJDK [13] and umbrella projects have been supported and progressed with help of JCP [21], the Openjdk team [22], JUGs like London Java Community [16], SouJava [17] and other JUGs in Brazil, a number of JUGs in Europe i.e. BGJUG (Bulgarian JUG) [18], BeJUG (Belgium JUG) [19], Macedonian JUG [20], and a number of other small JUGs. We hope in the coming time more JUGs and individuals would get involved. If you or your JUG wish to participate please get in touch.

—-

Credits

Special thanks to +Martijn Verburg (incepted Adopt OpenJDK),+Richard Warburton, +Oleg Shelajev, +Mite Mitreski, +Kaushik Chaubal and +Julius G for helping improve the content and quality of this post, and sharing their OpenJDK experience with us.

—-

How to get started ?

Join the Adoption Discuss mailing list [14], go to the Adopt OpenJDK home page [13] to get started, followed by referring to the Adopt OpenJDK Getting Started Kit [12] and Adopt OpenJDK Intermediate & Advance experiences [11] documents.

Please share your comments here or tweet at @theNeomatrix369.

Resources

[1] Adopt OpenJDK – the past present and future

[2] http://mreinhold.org/blog/on-the-next-train

[3] http://mreinhold.org/blog/jigsaw-phase-two

[4] http://mreinhold.org/blog/late-for-the-train-qa

[5] http://openjdk.java.net/projects/jigsaw/

[6] Scheme from Scratch

[7] Bootstrapping on Wikipedia

[8] OpenJDK8

[9] OpenJDK9

[10] Source src/share/vm/memory/universe.cpp in hotspot

[11] Adopt OpenJDK Intermediate & Advance experiences

[12] Adopt OpenJDK Getting Started Kit

[13] Adopt OpenJDK home page

[14] Adoption Discuss

[15] Betterrev

[16] London Java Community

[17] SouJava

[18] BGJUG (Bulgarian JUG)

[19] BeJUG (Belgium JUG)

[20] Macedonian JUG

[21] Java Community Process – JCP

[22] OpenJDK team

Our sister project – Adopt-a-JSR

[23] The New Build System

[24] The New Makefiles

[25] Build Benchmarks page 1

[26] Build Benchmarks page 2

---

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

Hotspot™ is in focus again -aka- Hacking Hotspot™ in Eclipse Juno under Ubuntu 12.04!

News, introduction and a bit of History

Hotspots, sunspots, solar flares et al have been in the news since the last few months, first a major one in November 2012, followed by another one in January 2013. Whilst I have your attention on this topic, did you know the JVM (Java Virtual Machine) also goes by the name Hotspot™, find out why it is called so! HotSpot. Unlike the sun’s solar hotspots, the JVM i.e. Hotspot™ is a much gentler entity! With the new build system i.e. build-infra it is now possible to build and run Hotspot from within Eclipse in a matter of minutes. Further you can even launch your own java-based program using gamma – the Hotspot launcher.

The steps are similar to that of building it via the command-line interface (CLI) – but with the advantage of being able to Run and Debug any line in the Hotspot source-code (it is a C/C++ based component of the OpenJDK project). You can also pass in the same arguments as you would to the java command-line.

---

In this blog we will cover the following in two parts, a simple import-build-run steps for busy readers (first 8 to 9 sections) followed by a simple hack-build-run-examine (the additional sections) for ones with more spare time in hand:

Building the latest version of OpenJDK locally
Installing Eclipse Juno under Ubuntu 12.04
Importing the C/C++ Hotspot project into Eclipse Juno
Downloading the Hotspot ready-to-go project files (and supporting script files)
Applying the downloaded files to the imported Eclipse project files
Running Eclipse with the Hotspot project loaded
Building Hotspot from within Eclipse
Running Hotspot from within Eclipse

Hacking the java.c program – add your own code to it
Running Hotspot (and loading a simple Demo class) from within Eclipse
Putting breakpoints in java.c within Eclipse
Tracing code and inspecting variables in java.c within Eclipse
Examining the different logs generated during the build and Run/Debug launching processes

Most of these sections will have screen-shots to illustrate the actions to be taken to achieve the results. We cover only basic aspects of the topic so advanced topics won’t be covered here. Having said that it is a step up from the Old Build system, as in the new Build-infra incremental builds have made things faster.

---

Pre-requisites

The below programs and environments are required, including basic operational knowledge and understanding of them:

Eclipse Juno
Eclipse CDT 6.0
Ubuntu 12.04 (CLI & GUI familiarity)
OpenJDK 8 (a version that builds)
OpenJDK 7 or Java / Javac 7.0 (JRE/JDK 1.7.013 – at the time of writing)
Any class or jar that can be run using the java command.
Basic knowledge of C/C++ (if you are going to make changes to Hotspot)
Basic knowledge of Bash scripting

---

Building the latest version of OpenJDK locally

In order to download and install the latest OpenJDK system, follow build instructions at http://java.net/projects/adoptopenjdk/pages/AdoptOpenJDKVMBuild.

---

Installing Eclipse Juno under Ubuntu 12.04

Please ensure you do not have a previous installation of Eclipse on your system, if you do uninstall it before performing the below instructions.

Find out whether you are running a 32- or 64-bits system with the below command:

$ file /bin/bash

Sample output indicating the platform of the system
/bin/bash: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0xf199a4a89ac968c2e0e99f2410600b9d7e995187, stripped

Quick method

Download the script that does that automatically for us:
$ wget http://bit.ly/YBS8AR

Run it with the correct platform parameter using the feedback from the previous step:
$ bash installEclipseJunoCDTForUbuntu12.04.sh 32
or
$ bash installEclipseJunoCDTForUbuntu12.04.sh 64

To create a icon in the Unity Dashboard, etc… please refer to the last section of the original blog

— OR —

Step-by-step method

Go to the downloads folder under your $HOME directory:

$ cd ~/Downloads

Download the necessary Eclipse Juno CDT package

32 bits
$ wget http://bit.ly/11QStqT
– or –
64 bits
$ wget http://bit.ly/WSadhy

Untar the downloaded file:

32 bits
$ tar -zxvf eclipse-cpp-juno-SR1-linux-gtk.tar.gz
– or –
64 bits
$ tar -zxvf eclipse-cpp-juno-SR1-linux-gtk-x86_64.tar.gz

This results in a folder named “eclipse”, which should be copied to /opt with

$ sudo mv “eclipse” /opt

Now add a link to the executable in /usr/bin to “/opt/eclipse/eclipse” for easier access.

$ sudo ln -s “/opt/eclipse/eclipse” /usr/bin/eclipse

To do the rest of the optional steps i.e. create a icon in the Unity Dashboard, etc… please refer to the last section of the original blog.

---

Importing the C/C++ Hotspot project into Eclipse Juno

Run Eclipse

Make it a point to create your own Eclipse workspace for OpenJDK projects calling it something like ‘Eclipse_OpenJDK_Projects’ (without quotes).

File > Import… > select a import source > Existing Code as Makefile Project
Select the hotspot project from the hotspot sub-folder
Select ‘Cross GCC’ as the Toolchain

Once imported, the hotspot project appears in the Project Explorer panel at the left-hand-side.

Now shutdown Eclipse.

---

Downloading the Hotspot ready-to-go project files (and supporting script files)

Go to the command-prompt and navigate to where the ‘hotspot’ folder is situated (i.e. ~/sources/jdk8_tl/hotspot).

Run this command at the CLI:

$ wget http://bit.ly/Vreesk

Download takes under a minute and the download progress bar should show 100%.

---

Applying the downloaded files to the imported Eclipse project files

Run the below command from within the hotspot folder:

$ bash downloadEclipseProjectFiles.sh ~/Eclipse_OpenJDK_Projects ~/Eclipse_OpenJDK_Projects/

Once all the scripts and supporting files are downloaded the following visible and hidden files and folders should become available:

.cproject (hidden)
.project (hidden)
.settings (hidden)
.metadata (moved to destination)
eclipseScripts

The local copies of .cproject and .project will be overwritten by the above action, along with that the .metadata folder in the Eclipse workspace for the project will also be updated.

Navigate to the eclipseScripts folder and run:

$ bash runEclipseForHotspot.sh ~/Eclipse_OpenJDK_Projects

This should launch Eclipse and take you to your imported project (always launch Eclipse via the script file).

---

Building Hotspot from within Eclipse

Select the project in the Project Explorer and select the menu option Project > Build Project (in case of first-time build from within Eclipse, please do a Clean Build from within Eclipse before doing a full-build).
Now shutdown Eclipse.

Restart Eclipse using the script mentioned in the above section (see Applying the downloaded…), always launch Eclipse via this script file.

Select the project in the Project Explorer and select the menu option Project > Build Project.
A successful build will result in messages in the build output console:

---

Running Hotspot from within Eclipse

Select the ‘hotspot’ project from the Project Explorer and select the menu option Run > Run.

Basically running ‘gamma’ through the Eclipse Run/Debug launcher should print out the gamma usage screen which looks like the below (snapshot) and exactly the same as the ‘java –help’ command run from the command prompt:

[Loaded sun.misc.Launcher$AppClassLoader$1 from shared objects file]
[Loaded java.lang.SystemClassLoaderAction from shared objects file]
Usage: gamma [-options] class [args...]
           (to execute a class)
   or  gamma [-options] -jar jarfile [args...]
           (to execute a jar file)

where options include:
    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
.
.
.
[Loaded java.lang.Shutdown from shared objects file]
[Loaded java.lang.Shutdown$Lock from shared objects file]

See section “Examining all kinds of logs…” for a detailed version of the above snapshot of the log.

_______________________________________________________________________________________________________________________________________________________________________________

Hacking the java.c program – add your own code to it

The C/C++ code in Hotspot, in this case java.c (found in the hotspot/src/share/launcher folder) isn’t as scary as C/C++ code can be deemed to be – I hope it removes such a phobia about system-level languages. Locate hotspot/src/share/launcher/java.c in Eclipse, open it and go to the section of the code between line numbers 388 and 389 and insert the below block of code:

      printf("**********************************\n");
      printf("* Simple java.c hack \n");
      printf("**********************************\n");
      printf("* jre path: %s \n", jrepath);
      printf("* jvm path: %s \n", jvmpath);
      printf("* Jarfile: %s \n", jarfile);
      printf("* Classname: %s \n", classname);
      printf("**********************************\n");

which on completion should look like:

---

Running Hotspot (and loading a simple Demo class) from within Eclipse

If you don’t have a sample class or jar to hand, create one, here is a snapshot of the code behind the demo HelloWorld.java program:

public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("************************");
        System.out.println("*     Hello, World     *");
        System.out.println("************************");
    }
}

Save this program say in the $HOME folder, then edit ../hotspot/eclipseScripts/updateEnvVarsForEclipseForHotspot.sh to enable passing the HelloWorld or any other class to the hotspot program’s Run/Debug Launcher – instructions in the file, look for the below block of code (comment and uncomment the relevant lines). If another name or location is chosen for the demo class or program, then make the necessary amendment in the above script file – comments available in the file to guide you (see below).

# Uncomment this line when you have a sample class or jar to pass to gamma
# export DEMOCLASS_OR_JAR_ARG="-cp $HOME HelloWorld"

# This will invoke gamma to display the usage screen
export DEMOCLASS_OR_JAR_ARG=""
}

Restart Eclipse using the script mentioned in the above section (see Applying the downloaded…), always launch Eclipse via this script file.

The above lines should produce the below output (snapshot) between two blocks of verbose messages from gamma, which also contains the print messages to the console that were inserted into the java.c unit:

Using java runtime at: /usr/lib/jvm/java-7-openjdk-i386/jre/
**********************************
* Simple java.c hack
**********************************
* jre path: /usr/lib/jvm/java-7-openjdk-i386/jre/
* jvm path: /home/saiubuntu/sources/jdk8_tl/build/linux-x86-normal-server-release/hotspot/linux_i486_compiler2/product/libjvm.so
* Jarfile: (null)
* Classname: HelloWorld
**********************************
[Loaded java.lang.Object from shared objects file]
[Loaded java.io.Serializable from shared objects file]
.
.
.
************************
*     Hello, World     *
************************
[Loaded java.lang.Shutdown from shared objects file]
[Loaded java.lang.Shutdown$Lock from shared objects file]

See the below section on “Examining all kinds of logs…” for a detailed version of the above snapshot.

---

Putting breakpoints in java.c within Eclipse

Open java.c in the Eclipse editor, position the cursor on line 392 (or any other line) and double click on the left-hand-side border/bevel of the editor window:

---

Tracing code and inspecting variables in java.c within Eclipse

Now select the project in the Project Explorer and run the program in Debug mode by clicking on the menu option Run > Debug to launch the Debug perspective:

Adding a Watch Expression and Tracing through the lines of code shows the current value being populated in the Watch Expression window (top right corner). A watch expression can be added by merely select the field or variable in the editor, right-mouse click and selecting Watch Expression from the pop-up menu:

---

Examining the different logs generated during the build and Run/Debug launching processes

A number of log files were created during the whole process and can be examined, see below (the names of the files describe their contents):

hotspot.eclipse.clean.build.log (console messages generated on Clean Hotspot Build action)
hotspot.eclipse.full.build.log (console messages generated on Full Hotspot Build action)
hotspot.eclipse.incremental.build.log (console messages generated on Incremental Hotspot Build action)
gamma.run_or_debug.Usage.output.log (console messages generated on running gamma in verbose mode, and not launching any program)
gamma.run_or_debug.HelloWorld.Verbose.output.log (console messages generated on running gamma in verbose mode, and launching the HelloWorld program)

---

A number of advanced hacks (assignment for readers!)

1) Insert debug-level log messages into java.c throughout the unit, rebuild gamma and run the Demo class or any other java-based.
2) Refactor java.c and insert debug-level log messages throughout the unit, rebuild gamma and run the Demo class or any other java-based program.
3) After step 2) above, load a low-latency, GC-tuned java based program, with GC-logs enabled and examine the GC-logs produced, to see if there is any change in performance (for performance tuning buffs).
4) Apply the Elvis operator to javac (a good way to get exposure to ‘how to modify javac?’) and compile a java program with the Elvis operator implemented in it.
5) GC-fun: replace the existing garbage collector(s) with a custom one. Resurrect PermGen or iCMS in the existing code. Add your change you always wanted to, to the existing version of Hotspot (for GC buffs).
6) Change javac to be able to parse and compile new language features or understand another dialect of JVM-based languages or maybe even older programming languages like C, Assembly, Scheme or Smalltalk.
7) Replace the built-in class-loader with your custom version.

---

References

(1) OpenJDK: README for the New Build System
(2) Hotspot Runtime Overview
(3) HotSpot Internals
(4) Hotspot Docs
(5) OpenJDK build instructions (old build)
(6) Hacking Hotspot in Eclipse – Roman Kennke (old build system)
(7) Biased Locking in HotSpot
(8) HotSpot source: command line arguments
(9) Memory leak profiling with netbeans
(10) HotSpot Tools – HotSpot Internals for OpenJDK – Oracle Wiki
(11) HotSpot development on Linux with NetBeans
(12) Building Hotspot in Eclipse under Ubuntu 12.04 (Old Build system)

---

Related references

Read about the Adopt OpenJDK program at the the Adopt OpenJDK java.net project website. Join the Adopt OpenJDK mailing list at the google group or send an email to adopt-openjdk@googlegroups.com to subscribe to the mailing list. Find out how to join a Google group.
@adoptopenjdk – follow us on twitter, get the latest Adopt OpenJDK community news!

Read about the Adopt-a-JSR program at the java.net project website. Join the Adopt-a-JSR mailing list at the java.net mailing list. Send an email to members@adoptajsr.java.net to subscribe to the mailing list.
@adoptajsr – follow us on twitter, get the latest Adopt-a-JSR community news!

---

Finally

These instructions will be converted to much compact ‘wiki’ instructions without any of the narrations explanations as mentioned in the blog above – “post any feedback and improvements from the users of the instructions!”. You can post them at the bottom of this blog or tweet them to me at @theNeomatrix369.

‘Thank you’ to those who have helped in the process (@karianna, @RichardWarburto, Girish Balakrishnan, @teozaurus, @SamirTalwar, @sandromancuso and @stPundit). Any (constructive) comments or request for changes are welcome.

I have learnt a lot about the topics covered here i.e. Java, Eclipse and Ubuntu! For those who wish to dive deeper in Hotspot and Java, make use of all the links in the blog, study the bash scripts written to automate the process, above all look into the hotspot folder and take interest in all the files it contains.

Feel free to approach the list of ‘assignment’ hacks, I’ll be more than happy to add links from here to your page if you accomplish any one of them. You will also be mentioned in our community news relayed regularly.

Post #fosdem, #jfokus – lots more #java, #jsr & #openjdk news from all over!

In the last couple of weeks since FOSDEM 2013, Jfokus 2013 and, events and hackdays organised by LJC JUG members, we have had a lots of exciting news and resources to share with you. The source of the information have been mailing lists / forums, meetup events, twitter and other sources.

Plenty were spoken about #java, #jsr and #openjdk, the topics covered by the #adoptajar and #adoptopenjdk programs.

FOSDEM 2013

• The OpenJDK and JSR topics get good coverage at FOSDEM 2013! JUG members participating!
• FOSDEM 2013 Lightening talk videos

Speaker interviews:

• https://fosdem.org/2013/interviews/
• https://fosdem.org/2013/news/2013-01-31-last-set-of-speaker-interviews/

---

Jfokus 2013

#Jfokus coverage of #java, a snapshot by Kevin Farnham:

• Monday at Jfokus 2013: Tutorials and Hands-On Labs
• Tuesday at Jfokus 2013: Intro, a New Java Champion, Jfokus Embedded Launch, Keynotes, Technical Sessions, Buffet, and BOFs
• Wednesday at Jfokus 2013: Then Was Then, Now Is Now, It’s Time to Be Smart (Jonas Bonér and the Final Jfokus Embedded Sessions)
• Video & archive for 2013 to feature soon !

---

@steveonjava – Nighthacking!

Watch all the recordings of Steve’s Nighthacking from FOSDEM 2013 through to Jfokus 2013!

---

LJC events & hackdays

Garbage collection – The useful parts

• Meetup event
• Recorded presentation
• Slides

---

WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353)

• Meetup event
• Recorded presentation
• Slides
• Github (JSR-353-JSON artefact: https://github.com/Adopt-a-JSR/JSR-353-JSON; JSR-356-WebSocketAPI to follow soon!)
• David’s github link:  https://github.com/davidillsley/json-workshop
• JSR-353 (JSON): http://glassfish.java.net/adoptajsr/jsr353.html (http://json-processing-spec.java.net/)
  
• JSR-356 (Websocket): http://glassfish.java.net/adoptajsr/jsr356.html (http://websocket-spec.java.net/)

---

Bring your Performance Problems Panel

• Meetup event
• Recorded presentation

---

@adoptajsr news feed

– Plenty of updates on polls, JSRs, presentations, github projects, etc… were discussed on twitter.
– WebSocket & JSON Hack Day (covering implementation for JSR 356 & JSR 353) – see above!
– Suggestion to extend / improve the Java API – Thanks Hildeberto Mendonça for coming forward to contribute!
– Modernize Connector/MDB — Vote and comment – Thanks Richard Kolb for support such initiatives!
– CDI 1.1 applications you can work with – Thanks Luigi for the contribution!

---

@adoptopenjdk news feed

– Updates on latest changes and developments in the OpenJDK world that were discussed amongs members on twitter.
– Potential plans to deprecate SPARC V8 support in HotSpot!
– StringBuffer to StringBuilder again – discussions rekindled!

---

Upcoming events and meetings

Adopt-a-JSR online meeting – February 27
Further hackdays, discussion panels and events to be organised by LJC JUG members.
26-27 March 2013, Devoxx UK

—-

Read about the Adopt OpenJDK program at the the Adopt OpenJDK java.net project website.

Join the Adopt OpenJDK mailing list at the google group. Find out how to join a Google group or send an email to adopt-openjdk@googlegroups.com to subscribe to the mailing list.

Read about the Adopt-a-JSR program at the java.net project website.

Join the Adopt-a-JSR mailing list at the java.net mailing list. Send an email to members@adoptajsr.java.net to subscribe to the mailing list.

The OpenJDK and JSR topics get good coverage at FOSDEM 2013! JUG members participating!

Its FOSDEM (Free and Open source Software Developers’ European Meeting) time again, and this year we have very good coverage on the OpenJDK and JSR topics at FOSDEM 2013. A number of speakers and events are taking place throughout the two days covering various topics on Java.

Adopt-a-JSR and JSRs

The speakers and events for these topics will speak on the present state and the future of Java, technical topics like InvokeDynamics, Zero, and the Shark compiler. The new JSR for Hotspot/JVM that will affect the Android world i.e. Dalvik users! Below are a list of speakers and events covered under this topic.

Speakers

Heather VanCura

Jérôme Pilliet

Martijn Verburg

Events

JCP state of the nation and future directions

Android 292

Invokedynamic: Tales from the Trenches

Return of the Shark

Adopt OpenJDK / OpenJDK

The OpenJDK topic has got massive coverage with 17 speakers, speaking and holding events covering various topics. In summary the topics covered will be:

– Past and future of OpenJDK, new releases
– Q & A session on OpenJDK
– The OpenJDK journey and what has been learnt so far
– Porting OpenJDK to PowerPC/AIX, AArch64
– Running Nashorn and JavaJFX
– OpenJDK7u progress
– Highlight on OpenJDK Lambda
– Using Shark again

Below are a list of speakers and events covered under this topic.

Speakers

Andrew Haley

Volker Simonis

Martijn Verburg

Georges Saab

Doug Lea

Andrew Dinn

Paul Sandoz

Xerxes Rånby

Mark Reinhold

Steve Poole

Roman Kennke

Sean Coffey

Goetz Lindenmaier

Dalibor Topić

Mario Torre

Ben Evans

Richard Sands

Events

OpenJDK Governing Board Q&A

The State of OpenJDK

Adopt OpenJDK. What we’ve learned, where we’re going.

Free Java devroom

Power to the people – the OpenJDK PowerPC/AIX port

OpenJDK Lambda the Ultimate

Porting OpenJDK to AArch64

Getting up and running with Nashorn & OpenJFX

OpenJDK7u, progress and highlights

Return of the Shark

Java gets its own space in the Free Java (Dev Room).

Members of the LJC JUG, CEJUG, Brussels JUG and other JUGs worldwide will be present at this event!

Follow the event on Twitter by following the below handles:

@fosdem

@adoptopenjdk

@adoptajsr

FOSDEM website: https://fosdem.org/2013/

Tracks: https://fosdem.org/2013/schedule/tracks/

Events schedule: https://fosdem.org/2013/schedule/events/

Speaker Interview: https://fosdem.org/2013/interviews/

Ben Evans’ post on FOSDEM 2011

Read about the Adopt OpenJDK program at the the Adopt OpenJDK java.net project website.

Join the Adopt OpenJDK mailing list at the google group or send an email to adopt-openjdk@googlegroups.com to subscribe to the mailing list. Find out how to join a Google group.

Read about the Adopt-a-JSR program at the java.net project website.

Join the Adopt-a-JSR mailing list at the java.net mailing list. Send an email to members@adoptajsr.java.net to subscribe to the mailing list.

(function(){
var b, a;
b=document.createElement(“script”);
b.src=”//a1.vdna-assets.com/analytics.js”;
b.async=true;
a=document.getElementsByTagName(“head”)[0];
a.insertBefore(b,a.firstChild);
this.VDNA=this.VDNA||{};
this.VDNA.queue=this.VDNA.queue||[];
}());

VDNA.queue.push({ apiKey : “individual1370625757899”, method : “reportPageView”});

How to build JTReg in Eclipse (for Ubuntu 12.04 LTS)

After taking inspiration from doing a whole lot of work with Eclipse and OpenJDK projects I took off on the journey to experiment with JTReg (http://openjdk.java.net/jtreg/) and attempt to build it in an IDE of my choice i.e. Eclipse. These instructions specify how you can build the JTReg using the Eclipse IDE running under Ubuntu (using OpenJDK’s Old build and Infrabuild systems).

I have kept the structure of this blog similar to the previous one so to maintain uniformity in my approach.

Firstly what is JTReg – it is a regression test harness used by the OpenJDK test framework. The different projects and packages in the OpenJDK system can be individually or wholely tested via JTReg.

Below is a list of systems and versions under which the installations and configurations were performed to come up with these instructions:

Synaptic Package Manager 0.75.9

Eclipse Indigo 3.7.2
—Eclipse JDT (Java) 3.7.2
—Eclipse CDT (C/C++)
——6.0.0.dist (C/C++ Remote debug launcher)
——1.0.2.dist (CDT GCC Cross Compiler)
——7.0.0.dist (GDB Common)

Ubuntu 12.04 LTS
Ant 1.8.4
JavaHelp 2.0
Java Test Harness 4.4
JUnit 4-4.8
OpenJDK 8
Java/Javac 1.7

You need to have an environment with OpenJDK installed and setup (as described in the How to build OpenJDK projects in Eclipse (for Ubuntu 12.04 LTS) – just follow the parts from the start upto the section ‘Preparing projects, folders and files’, as hereafter the ‘meaty OpenJDK shabhang’ starts and we don’t need it for the current work.

If you wish to just download the JTReg binary run tests from the command line, follow the instructions on the
Adopt OpenJDK’s Install JTReg page – configurations for both the Old build and the Infrabuild systems are available there.

At this point you should be able to manually run JTReg tests from the command-line interface (terminal).

Build and run JTReg in Eclipse

Downloading and setting up dependencies

Change the current folder location to the $HOME directory folder and download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:
Download the JTReg source files from http://hg.openjdk.java.net/code-tools/jtreg using the below command:

 $
 $ hg clone http://hg.openjdk.java.net/code-tools/jtreg
 $ 

this will create a folder by the name jtreg in the current location with the JTReg source files in it.

Alternative but less updated location: http://download.java.net/openjdk/jtreg/). If this route is used, then run the below command in the terminal mode, to download the zip file stored at the above location:

 cd ~
 wget http://www.java.net/download/openjdk/jtreg/promoted/4.1/b04/jtreg-4.1-src-b04_14_mar_2012.zip

Create a folder say under the Home directory ~/ called ‘jtreg’, unzip the zip file into this folder.

JavaHelp
Install Javahelp – required to build jtreg

 sudo apt-get install javahelp2

JavaTest Harness
Download the Javatest Harness 4.4 from http://download.java.net/jtharness/download.html

Run the below commands to download it off the web into ~

 cd ~
 wget http://download.java.net/jtharness/4.4.1/Rel/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Unzip the zip file into /opt/jtharness/4.4 by doing the below:

 cd /opt
 sudo mkdir jtharness
 sudo mkdir jtharness/4.4
 cd jtharness/4.4
 sudo unzip ~/jtharness-4_4_1-MR1-bin-b13-20_dec_2011.zip

Apache Ant
Download Apache Ant 1.8.4 from http://archive.apache.org/dist/ant/binaries/ using the below commands:

 cd ~
 wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.4-bin.zip   

 cd /opt
 sudo unzip ~/apache-ant-1.8.4-bin.zip

JUnit
Download JUnit4 jar file from Maven Central (http://search.maven.org/#search and search for junit4) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/junit4
 sudo mv junit-4.nn.jar /opt/junit4

TestNG
Download TestNG jar file from Maven Central (http://search.maven.org/#search and search for testng) and copy it into the /opt folder by performing the following commands:

 sudo mkdir /opt/testng
 sudo mv testng-n.n.jar /opt/testng

Now make changes to the build.properties file to amend paths for JUnit4, TestNG JavaHelp and JavaTest Harness.

Amending the Ant build properties file

As the build.properties file is protected, you will need to invoke it using the below commands:

 cd ~/jtreg/make
 sudo gedit build.properties

Make the following changes to the build.properties file in the ~/jtreg/make/ folder

jtreg.build.resources=/opt
 jhhome = /usr/share/java
 
 # JTHarness or JavaTest (should be should be 4.3 or better)
 #javatest.home = ${jtreg.build.resources}/jtharness/4.3
 javatest.home = ${jtreg.build.resources}/jtharness/4.4
 
 # JavaHelp (should be version 2.0 or better)
 javahelp.home = ${jhhome}
 #jhall.jar = ${javahelp.home}/javahelp/lib/jhall.jar
 jhall.jar = ${javahelp.home}/jhall.jar
 #jh.jar = ${javahelp.home}/javahelp/lib/jh.jar
 jh.jar = ${javahelp.home}/jh.jar
 
 # JUnit (should be 4.5 or better)
 #junit.jar = ${jtreg.build.resources}/junit/4.5/junit-4.5.jar
 junit.jar = ${jtreg.build.resources}/junit4-n.n.n.jar
  
 # TestNG (should be 6.5 or better)
 #testng.jar = ${jtreg.build.resources}/testng/6.7/other-jars/testng-6.7-dist.jar
 testng.jar = ${jtreg.build.resources}/testng/testng-n.n.jar
 
 # Ant ((should be version 1.6.5 or better)
 #ant.home = ${jtreg.build.resources}/ant/1.7.1
 ant.home = ${jtreg.build.resources}/apache-ant-1.8.4
 ant.jar = ${ant.home}/lib/ant.jar

Ensure comments are placed against the old settings (just in case you need to revert). In order to find the correct location of the jh.jar and jhall.jar files use the ‘find’ command in terminal mode.

Setting up environment variables

Add the below two lines to the ~/.bashrc file (Infrabuild)

     export JT_HOME=$HOME/jtreg                         ............. [1]
     
     or 
     
     export JT_HOME=$HOME/jtreg/dist/jtreg              ..............[2]

[1] Applies if the jtreg binary has been downloaded directly from the JTReg website.
[2] Applies if the jtreg source has been downloaded directly from the JTReg repo and a binary is built locally.

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-x64-normal-server-release/images/j2sdk-image

Note: the name of the sub-folder under $SOURCE_CODE/jdk8_tl/build/ folder can have other variants i.e. linux-x86_64-normal-server-release, so please check for this in your build environment before applying the above env settings.

Or the below two lines to the ~/.bashrc file (Old build)

     export PRODUCT_HOME=/home/openjdk/sources/jdk8_tl/build/linux-amd64_backup/j2sdk-image

Now source the .bashrc file using the below command:

source ~/.bashrc

Link folder /jtreg/src to the existing project using the Configure Build Path option.

Add the /opt/apache-ant-1.8.4/lib/ant.jar to the project via the Build Path > Configure Build Path > Libraries option.

Add External JARS…, navigate to the /opt/apache-ant-1.8.4/lib/ folder, select ant.jar and click on Okay.

Importing project via Ant Build file

Run Eclipse.
Select File > New > Other > Java Project from Existing Ant build
Once done select the folder ~/jtreg/make/ and then select the build.xml.

Selecting this file populates all the ant tasks, select the option Link to the buildfile in the file system.

Now select the build.xml file, right mouse click and select the Run As > Ant Build… option, and do the below:
1) Select Main, and add -verbose to the Arguments section
2) Select Targets, and select clean from the list of targets and set the order as clean, build (if you want clean builds each time). Do skim through the list of Ant tasks, to learn the different actions that can be performed pre- and post- builds.
3) Select ClassPath and set the ClassPath for Ant Home to Apach Ant 1.8.4 by selecting Ant Home from the list, and then clicking on the Ant Home button on the bottom right hand corner and select /opt/apache-ant-1.8.4/ from the directory tree.
4) Select JRE, and select java-7-openjdk-amd64 from the list. If it does not exist on the list, click on Installed JREs… and install one by adding a JRE from /usr/lib/jvm/java-7-openjdk-amd64.

Now run the ant script by selecting Run As > Ant build.

See sign of success below (last 30 odd lines of echos from the Ant console)

dist-jtreg:
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/lib/junit.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
      [zip] Building zip: /home/openjdk/jtreg/dist/jtreg-no-junit.zip
      [zip] adding entry jtreg/COPYRIGHT
      [zip] adding entry jtreg/LICENSE
      [zip] adding entry jtreg/README
      [zip] adding entry jtreg/doc/jtreg/faq.html
      [zip] adding entry jtreg/doc/jtreg/tag-spec.txt
      [zip] adding entry jtreg/doc/jtreg/usage.txt
      [zip] adding entry jtreg/legal/jtharness/copyright.txt
      [zip] adding entry jtreg/legal/jtharness/license.txt
      [zip] adding entry jtreg/lib/javatest.jar
      [zip] adding entry jtreg/lib/jh.jar
      [zip] adding entry jtreg/lib/jtreg.jar
      [zip] adding entry jtreg/linux/bin/jtdiff
      [zip] adding entry jtreg/linux/bin/jtreg
      [zip] adding entry jtreg/solaris/bin/jtdiff
      [zip] adding entry jtreg/solaris/bin/jtreg
      [zip] adding entry jtreg/win32/bin/jtdiff
      [zip] adding entry jtreg/win32/bin/jtreg
build:
BUILD SUCCESSFUL
Total time: 37 seconds

Run Configuration

TODO – able to run OpenJDK tests on one or multiple projects / packages from within Eclipse – anyone dares to go this path?

Newly created Eclipse projects

Now you have a number of ready created Eclipse projects in your Eclipse workspace.

Go to the workspace folder where Eclipse creates the project files and settings i.e. into the /home/openjdk/workspace folder.

There is a folder (called jtreg) representing the project you are currently working in, there are atleast two to four hidden files and folders in this location containing the below:

 .project
 .classpath
 .settings

Importing project via Make file

TODO – importing the JTReg Make file and being able to do the same thing as above – anyone wishes to contribute?

Build and run JTReg from CLI (terminal)

The steps mentioned in sections Downloading and setting up dependencies, Amending the Ant build properties file and Setting up environment variables must be performed before going further.

Once the above settings are in place, build JTReg from the CLI via the following commands

 cd ~/sources/jtreg/make
 ant

This should give the same output as illustrated above under Eclipse section:

 jar-jtreg:
      [jar] Building jar: /home/saiubuntu/sources/jtreg/dist/jtreg/lib/jtreg.jar
 
 -dist-jtreg.doc:
     [copy] Copying 1 file to /home/saiubuntu/sources/jtreg/dist/jtreg/doc/jtreg
 
 -dist-jtreg.bin:
 
 -dist-jtreg.legal:
 
 dist-jtreg:
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/jtreg.zip
      [zip] Building zip: /home/saiubuntu/sources/jtreg/dist/open-jtreg.zip
 
 build:
 
 BUILD SUCCESSFUL
 Total time: 9 seconds

Once a build is successful, a number of distribution files are created in the following folders under the jtreg folder structure:

 dist
 ├── jtreg
 │   ├── lib
 │   │   ├── javatest.jar
 │   │   ├── jh.jar
 │   │   ├── jtreg.jar
 │   │   └── junit.jar
 │   ├── linux
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   ├── solaris
 │   │   └── bin
 │   │       ├── jtdiff
 │   │       └── jtreg
 │   └── win32
 │       └── bin
 │           ├── jtdiff
 │           └── jtreg
 ├── jtreg.zip
 └── open-jtreg.zip

The files under each of the OS names i.e. linux, solaris, windows are bash script files that execute the jtreg.jar file, for e.g.

 ./dist/jtreg/linux/bin/jtreg

or

 java -jar dist/jtreg/lib/jtreg.jar 

the above two will do the exact same thing and expects parameters, see usage screen (only a snapshot of the entire text):

 Documentation Options
                Options for additional documentation
    -h [words...] | -help [words...] | -usage [words...]
                    Command line help. Give words to see help info containing
                    those or use "-help all" to see all available help.
    -n | -relnote   Release notes
    -onlineHelp [words...]
                    Show the online help. You can also show the online help from
                    the desktop Help menu.
    -t | -tagspec   Tag specification supported by this implementation
    -version        Give information about the version of jtreg in use.
 ... [multi-page usage text]

The most simplest syntax to run jtreg is

  [absolute/relative path]/jjtreg [options] [folder]

for example

  [absolute/relative path]/jtreg -verbose:fail [absolute/relative path]/jdk/test/java/lang/invoke/

[absolute/relative path] – applies depending on where you position in the OpenJDK folder structure.

It is also possible to build jtreg via the make command but that would require additional installation of dependent components and configuration. Also have a look at Adopt OpenJDK’s Install JTReg page to see how tests are run via the CLI.

Credit and kudos

My sincere thanks to Jonathan Gibbon’s from OpenJDK’s JTReg team who has been kindly responding to my queries and has walked me through the last important steps in the process!

Thanks to the Adopt OpenJDK initiative led by Martijn Verburg and others – its a pleasure to support our community!

Big thanks to Rabea Gransberger for spending the time and meticulously going through the steps and reporting issues and suggestions.

Come and join us and take on a project (like I did) and make something out of the project and yourself!

Please provide any feedback on areas that work differently for you or does not work for you at all. If you have fixed any issue please let us know so that we can update the information here.

I will continue to refine the instructions above and also include your useful feedback as and when I receive them.