Tapestry Training -- From The Source

Let me help you get your team up to speed in Tapestry ... fast. Visit howardlewisship.com for details on training, mentoring and support!

Saturday, October 31, 2009

Maven: Throwing out the bath water, keeping the baby

... in other words, a first step towards using Maven for dependency management but NOT builds. That's the irony of Maven ... they've conflated two things (dependency management and builds) in such as way that they make the useful one (dependency management) painful because the build system is so awful.

As an interrum step between full Maven and (most likely) Gradle, I've been looking at a way to use Maven for dependencies only in a way that is compatible with Eclipse ... without using the often flakey and undependable M2Eclipse plugin.

In any case, rather than assuming that dependencies might change at any point in time at all, let's assume that when I change dependencies (by manually editing pom.xml) I know it and am willing to run a command to bring Eclipse (and my Ant-based build) in line.

First, my pom.xml:

<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"
  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.example</groupId>
  <artifactId>myapp</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <dependencies>
    <dependency>
      <groupId>org.apache.tapestry</groupId>
      <artifactId>tapestry-hibernate</artifactId>
      <version>${tapestry-version}</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.tapestry</groupId>
      <artifactId>tapestry-test</artifactId>
      <version>5.2.0-SNAPSHOT</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>javax.servlet</groupId>
      <artifactId>servlet-api</artifactId>
      <version>2.4</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.lucene</groupId>
      <artifactId>lucene-core</artifactId>
      <version>2.4.1</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.hibernate</groupId>
      <artifactId>hibernate-search</artifactId>
      <version>3.1.1.GA</version>
    </dependency>
    <dependency>
      <groupId>commons-lang</groupId>
      <artifactId>commons-lang</artifactId>
      <version>2.4</version>
    </dependency>
    <dependency>
      <groupId>commons-beanutils</groupId>
      <artifactId>commons-beanutils</artifactId>
      <version>1.8.0</version>
    </dependency>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>0.9.17</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.5.8</version>
      <type>jar</type>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>postgresql</groupId>
      <artifactId>postgresql</artifactId>
      <version>8.4-701.jdbc4</version>
    </dependency>
    <dependency>
      <groupId>xerces</groupId>
      <artifactId>xercesImpl</artifactId>
      <version>2.4.0</version>
      <type>jar</type>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.howardlewisship</groupId>
      <artifactId>tapx-datefield</artifactId>
      <version>${tapx-version}</version>
    </dependency>
    <dependency>
      <groupId>com.howardlewisship</groupId>
      <artifactId>tapx-prototype</artifactId>
      <version>${tapx-version}</version>
    </dependency>
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>5.9</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <id>tapestry360-snapshots</id>
      <url>http://tapestry.formos.com/maven-snapshot-repository/</url>
    </repository>
    <repository>
      <id>repository.jboss.org</id>
      <url>http://repository.jboss.org/maven2/</url>
    </repository>
  </repositories>
  <properties>
    <tapestry-version>5.1.0.5</tapestry-version>
    <lucene-version>2.4.1</lucene-version>
    <tapx-version>1.0.0-SNAPSHOT</tapx-version>
  </properties>
</project>

(This was adapated from one of my client's POMs).

Next, an Ant build file that compiles this, runs tests and builds a WAR:

<project name="example" xmlns:mvn="urn:maven-artifact-ant">

  <property name="classes.dir" value="target/classes" />
  <property name="test.classes.dir" value="target/test-classes" />
  <property name="web.lib.dir" value="target/web-libs" />
  <property name="webapp.dir" value="src/main/webapp" />
  <property name="webinf.dir" value="${webapp.dir}/WEB-INF" />

  <path id="compile.path">
    <fileset dir="lib/provided" includes="*.jar"/>
    <fileset dir="lib/runtime" includes="*.jar" />
  </path>

  <path id="test.path">
    <path refid="compile.path" />
    <pathelement path="${classes.dir}" />
    <fileset dir="lib/test" includes="*.jar" />
  </path>

  <target name="clean" description="Delete all derived files.">
    <delete dir="target" quiet="true" />
  </target>

  <!-- Assumes that Maven's Ant library is installed in ${ANT_HOME}/lib/ext. -->

  <target name="-setup-maven">
    <typedef resource="org/apache/maven/artifact/ant/antlib.xml" uri="urn:maven-artifact-ant" />
    <mvn:pom id="pom" file="pom.xml" />
  </target>

  <macrodef name="copy-libs">
    <attribute name="filesetrefid" />
    <attribute name="todir" />
    <sequential>
      <mkdir dir="@{todir}" />
      <copy todir="@{todir}">
        <fileset refid="@{filesetrefid}" />
        <mapper type="flatten" />
      </copy>
    </sequential>
  </macrodef>

  <macrodef name="rebuild-lib">
    <attribute name="base" />
    <attribute name="scope" />
    <attribute name="libs.id" default="@{base}.libs" />
    <attribute name="src.id" default="@{base}.src" />
    <sequential>
      <mvn:dependencies pomrefid="pom" filesetid="@{libs.id}" sourcesFilesetid="@{src.id}" scopes="@{scope}" />
      <copy-libs filesetrefid="@{libs.id}" todir="lib/@{base}" />
      <copy-libs filesetrefid="@{src.id}" todir="lib/@{base}-src" />
    </sequential>
  </macrodef>

  <target name="refresh-libraries" depends="-setup-maven" description="Downloads runtime and test libraries as per POM.">
    <delete dir="lib" quiet="true" />
    <rebuild-lib base="provided" scope="provided"/>
    <rebuild-lib base="runtime" scope="runtime,compile" />
    <rebuild-lib base="test" scope="test" />
    <echo>
      
*** Use the rebuild-classpath command to update the Eclipse .classpath file.</echo>
  </target>

  <target name="compile" description="Compile main source code.">
    <mkdir dir="${classes.dir}" />
    <javac srcdir="src/main/java" destdir="${classes.dir}" debug="true" debuglevel="lines,vars,source">
      <classpath refid="compile.path" />
    </javac>
  </target>

  <target name="compile-tests" depends="compile" description="Compile test sources.">
    <mkdir dir="${test.classes.dir}" />
    <javac srcdir="src/test/java" destdir="${test.classes.dir}" debug="true" debuglevel="lines,vars,source">
      <classpath refid="test.path" />
    </javac>
  </target>

  <target name="run-tests" depends="compile-tests" description="Run unit and integration tests.">
    <taskdef resource="testngtasks" classpathref="test.path" />
    <testng haltonfailure="true">
      <classpath>
        <path refid="test.path" />
        <pathelement path="${test.classes.dir}" />
      </classpath>

      <xmlfileset dir="src/test/conf" includes="testng.xml" />

    </testng>
  </target>


  <target name="war" depends="run-tests,-setup-maven" description="Assemble WAR file.">

    <!-- Copy and flatten the libraries ready for packaging. -->

    <mkdir dir="${web.lib.dir}" />
    <copy todir="${web.lib.dir}" flatten="true">
      <fileset dir="lib/runtime" />
    </copy>
    <jar destfile="${web.lib.dir}/${pom.artifactId}-${pom.version}.jar" index="true">
      <fileset dir="src/main/resources" />
      <fileset dir="${classes.dir}" />
    </jar>

    <war destfile="target/${pom.artifactId}-${pom.version}.war">
      <fileset dir="${webapp.dir}" />
      <lib dir="${web.lib.dir}" />
    </war>
  </target>
</project>

The key target here is refresh-libraries, which deletes the lib directory then repopulates it. It creates a sub folder for each scope (lib/provided, lib/runtime, lib/test) and another sub folder for source JARs (lib/provided-src, lib/runtime-src, etc.).

So how does this help Eclipse? Ruby to the rescue:

#!/usr/bin/ruby
# Rebuild the .classpath file based on the contents of lib/runtime, etc.

# Probably easier using XML Generator but don't have the docs handy

def process_scope(f, scope)
 # Now find the actual JARs and add an entry for each one.
 
 dir = "lib/#{scope}"

 return unless File.exists?(dir)
 
 Dir.entries(dir).select { |name| name =~ /\.jar$/ }.sort.each do |name|
   f.write %{  <classpathentry kind="lib" path="#{dir}/#{name}"}
   
   srcname = dir + "-src/" + name.gsub(/\.jar$/, "-sources.jar") 

   if File.exist?(srcname)
      f.write %{ sourcepath="#{srcname}"}
   end
   
   f.write %{/>\n}
 end
end

File.open(".classpath", "w") do |f|
  f.write %{<?xml version="1.0" encoding="UTF-8"?>
<classpath>
  <classpathentry kind="src" path="src/main/java"/>
  <classpathentry kind="lib" path="src/main/resources"/>
  <classpathentry kind="src" path="src/test/java"/>
  <classpathentry kind="lib" path="src/test/resources"/>
  <classpathentry kind="output" path="target/classes"/>
  <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
}

 process_scope(f, "provided")
 process_scope(f, "runtime")
 process_scope(f, "test")
 
 f.write %{
</classpath>
}
    
end

That's pretty good for half an hour's work. This used to be much more difficult (in Maven 2.0.9), but the new scopes attribute on the Maven dependencies task makes all the difference.

Using this you are left with a choice: either you don't check in .classpath and the contents of the lib folder, in which case you need to execute the target and script in order to be functional ... or you simply check everything in. I'm using GitHub for my project repository ... the extra space for a few MB of libraries is not an issue and ensures that I can set up the exact classpath needed by the other developers on the project with none of the usual Maven guess-work. I'm looking forward to never having to say "But it works for me?" or "What version of just about anything do you have installed?" or "Try a clean build, and close and reopen your project, and remove and add Maven dependency support, then sacrifice a small goat" every again.

Next up? Packaging most of this into an Ant library so that I can easily reuse it across projects ... or taking the time to learn Gradle and let it handle most of this distracting garbage.

7 comments:

Andreas Andreou said...

It would seem like http://maven.apache.org/plugins/maven-eclipse-plugin/ is enough for your needs then...

Buddy Casino said...

Spot on. I hate Maven, but I need it for various reasons atm.
Suppose I want to switch, why Gradle and not Buildr or something else?

Howard said...

maven-eclipse-plugin updates .classpath to point to the location of JARS in the Maven respository, which means that any other bulid you do in Ant is complicated.

Ivan Luzyanin said...

So the problem is about how bad Eclipse is working with it. I'm sorry to say that, but it doesn't sounds like it is Maven's fault.

Maven build system works exactly how it was designed to work. If you don't accept this approach then why are you saying it is "awful"? It just doesn't match you needs. If so - choose the the right tool for the job.

If you are looking for dependency management only - use Ivy.

Jon Scott Stevens said...

Howard, I just don't get it. Maybe I'm missing something here. As far as
I can tell, all of that over complicated Ant and Maven build juju will
result in failed builds 10 years from now when those repositories are no
longer available and nobody can figure out what you did. You are using 3
different 'languages' to just create a simple .war file and get things
into Eclipse. This is a classic case of KISS
fail!

Just check the jars into svn (alexandria), create a jars.properties file. List the .jar files in there:

<property name="alexandria.dir" location="../Alexandria" />
<property file="${alexandria.dir}/jar.properties" />

ex: servlet.jar=servlet-1.2.3.jar

Then in your ant build script... reference them...

<fileset id="war-thirdparty.lib" dir="${alexandria.dir}/thirdparty">
<include name="${joda-time-jsptags.jar}" />
<include name="${jstl.jar}" />
<include name="${standard.jar}" />
<include name="${tagonist.jar}" />
<include name="${flexjson.jar}" />
</fileset>

<target>
<war destfile="${build.dir}/website.war"
webxml="${base.dir}/website/src/main/webapp/WEB-INF/web.xml">
<lib refid="war-thirdparty.lib" />
</war>
</target>

This system works great. It is super simple, easy to understand, easy to extend and most importantly, your builds will work forever. You can check out your projects from svn and know full well that they will build any time from now or 100 years from now.

With regards to the .classpath file, just check it into svn with the project. Being able to setup a new developer hire in minutes by just pointing them at a repo and telling them to check it out is invaluable. No dependencies on Ruby or buggy eclipse plugins.

Howard said...

It's funny how polarizing Maven is!

Actually, under the system I describe, I do check in my JARs and .classpath ... I just find it handy that as I change or introduce dependencies, I can have all the ugly work (download it, put it in the right place, get rid of the old version, update the .classpath) semi-automated.

Curious Attempt Bunny said...

You can also try using the Maven ant tasks - Pom & DependenciesTask.

The Gant for this is not all that complicated:

ant.taskdef(name: "pom", classname: "org.apache.maven.artifact.ant.Pom")
ant.taskdef(name: "dependencies", classname: "org.apache.maven.artifact.ant.DependenciesTask")

pom(id: "maven", file: "pom.xml")
dependencies(pomrefid: "maven", pathId: compileDependencyClasspath, useScope: "compile")
// etc.