Choosing the Right Web Framework

Thank you Google Alerts, for pointing out this article on choosing a Java web framework. It's over a year old, but I think the things that make Tapestry special have only gotten stronger in the intervening time.

Git on Mac OS X: Don't ignore case!

By default, Mac OS X uses a case insensitive file system, and Git seems to honor that. The problem is, most programming languages, especially Java, are case sensitive. Class "JavaScriptSupport" needs to be in file "JavaScriptSupport.java" and not "JavascriptSupport.java". This is even worse when sharing code via a repository since some other developers may check out code on a case sensitive file system.

I was just renaming some classes, from things like "JavascriptStack" to "JavaScriptStack" (because the language is called "JavaScript" not "Javascript") ... and I was dismayed that Git saw that as an in-place update to a file, not a rename of the file.

Unfortunately, it's not as simple as git config core.ignorecase false to make Git do the right thing. That's an essential part of it, but Git still sees changes to the original naming of the file as a change, not a deletion.

I had to use the trick of one commit renaming JavascriptStack.java --> JSStack.java, then a second commit renaming JSStack.java --> JavaScriptStack.java.

Tapestry 5 Training in London: Oct 5 - 8

I'm once again partnering with SkillsMatter to teach my full Tapestry workshop. This is an expanded version of the class, which is growing from three days up to four; the additional day will ensure that we have time for all the existing materials, and add a new section on testing using TestNG, Selenium and Groovy. It will also give us more time to explore student directed ideas, such as security and meta-programming.

The class will be taught at SkillsMatter's offices in London, from October 5th through the 8th.

Everyone out of the Pool! Tapestry goes singleton!

Tapestry applications are inherently stateful: during and between requests, information in Tapestry components, value stored in fields, stick around. This is a great thing: it lets you program a web application in a sensible way, using stateful objects full of mutable properties and methods to operate on those properties.

It also has its downside: Tapestry has to maintain a pool of page instances. And in Tapestry, page instances are big: a tree of hundreds or perhaps thousands of interrelated objects: the tree of Tapestry structural objects that forms the basic page structure, the component and mixin objects hanging off that tree, the binding objects that connect parameters of components to properties of their containing component, the template objects that represents elements and content from component templates, and many, many more that most Tapestry developers are kept unawares of.

This has proven to be a problem with biggest and busiest sites constructed using Tapestry. Keeping a pool of those objects, checking them in and out, and discarded them when no longer needed is draining needed resources, especially heap space.

So that seems like an irreconcilable problem eh? Removing mutable state from pages and components would turn Tapestry into something else entirely. On the other hand, allowing mutable state means that applications, especially big complex applications with many pages, become memory hogs.

I suppose one approach would be to simply create a page instance for the duration of a request, and discard it at the end. However, page construction in Tapestry is very complicated and although some effort was expended in Tapestry 5.1 to reduce the cost of page construction, it is still present. Additionally, Tapestry is full of small optimizations that improve performance ... assuming a page is reused over time. Throwing away pages is a non-starter.

So we're back to square one ... we can't eliminate mutable state, but (for large applications) we can't live with it either.

The best solution would be to require that all those mutable fields be, instead, ThreadLocal objects, and to change all the logic that accesses them to instead read and write values to the ThreadLocal. Oh, and clean up each and every one at the end of the request, so that information doesn't bleed through to the next request. That would be an incredible imposition on Tapestry developers.

Fortunately, Tapestry has lots of options for meta-programming Tapestry component classes.

Tapestry has already been down this route: the way persistent fields are handled gives the illusion that the page is kept around between requests. You might think that Tapestry serializes the page and stores the whole thing in the HttpSession. In reality, Tapestry is shuffling just the individual persistent field values in to and out of the session. To both the end user and the Tapestry developer, it feels like the entire page is live between requests, but it's really a bit of a shell game, providing an equivalent page instance that has the same values in its fields.

What's going on in trunk (Tapestry 5.2 alpha) right now is extrapolating that concept from just persistent fields to all mutable fields. Every access to every mutable field in a Tapestry page is converted, as part of the class transformation process, into an access against a per-thread Map of keys and values. Each field gets a unique identifying key. The Map is discarded at the end of the request.

The end result is that a single page instance can be used across multiple threads without any synchronization issues and without any field value conflicts.

This idea was suggested in years past, but the APIs to accomplish it (as well as the necessary meta-programming savvy) just wasn't available. However, as a side effect of rewriting and simplifying the class transformation APIs in 5.2, it became very reasonable to do this.

Let's take an important example: the handling of typical, mutable fields. This is the responsibility of the UnclaimedFieldWorker class, part of Tapestry component class transformation pipeline. UnclaimedFieldWorker finds fields that have not be "claimed" by some other part of the pipeline and converts them to read and write their values to the per-thread Map. A claimed field may store an injected service, asset or component, or be a component parameter.

public class UnclaimedFieldWorker implements ComponentClassTransformWorker
{
    private final PerthreadManager perThreadManager;

    private final ComponentClassCache classCache;

    static class UnclaimedFieldConduit implements FieldValueConduit
    {
        private final InternalComponentResources resources;

        private final PerThreadValue<Object> fieldValue;

        // Set prior to the containingPageDidLoad lifecycle event
        private Object fieldDefaultValue;

        private UnclaimedFieldConduit(InternalComponentResources resources, PerThreadValue<Object> fieldValue,
                Object fieldDefaultValue)
        {
            this.resources = resources;

            this.fieldValue = fieldValue;
            this.fieldDefaultValue = fieldDefaultValue;
        }

        public Object get()
        {
            return fieldValue.exists() ? fieldValue.get() : fieldDefaultValue;
        }

        public void set(Object newValue)
        {
            fieldValue.set(newValue);

            // This catches the case where the instance initializer method sets a value for the field.
            // That value is captured and used when no specific value has been stored.

            if (!resources.isLoaded())
                fieldDefaultValue = newValue;
        }

    }

    public UnclaimedFieldWorker(ComponentClassCache classCache, PerthreadManager perThreadManager)
    {
        this.classCache = classCache;
        this.perThreadManager = perThreadManager;
    }

    public void transform(ClassTransformation transformation, MutableComponentModel model)
    {
        for (TransformField field : transformation.matchUnclaimedFields())
        {
            transformField(field);
        }
    }

    private void transformField(TransformField field)
    {
        int modifiers = field.getModifiers();

        if (Modifier.isFinal(modifiers) || Modifier.isStatic(modifiers))
            return;

        ComponentValueProvider<FieldValueConduit> provider = createFieldValueConduitProvider(field);

        field.replaceAccess(provider);
    }

    private ComponentValueProvider<FieldValueConduit> createFieldValueConduitProvider(TransformField field)
    {
        final String fieldName = field.getName();
        final String fieldType = field.getType();

        return new ComponentValueProvider<FieldValueConduit>()
        {
            public FieldValueConduit get(ComponentResources resources)
            {
                Object fieldDefaultValue = classCache.defaultValueForType(fieldType);

                String key = String.format("UnclaimedFieldWorker:%s/%s", resources.getCompleteId(), fieldName);

                return new UnclaimedFieldConduit((InternalComponentResources) resources,
                        perThreadManager.createValue(key), fieldDefaultValue);
            }
        };
    }
}

That seems like a lot, but lets break it down bit by bit.

    public void transform(ClassTransformation transformation, MutableComponentModel model)
    {
        for (TransformField field : transformation.matchUnclaimedFields())
        {
            transformField(field);
        }
    }

    private void transformField(TransformField field)
    {
        int modifiers = field.getModifiers();

        if (Modifier.isFinal(modifiers) || Modifier.isStatic(modifiers))
            return;

        ComponentValueProvider<FieldValueConduit> provider = createFieldValueConduitProvider(field);

        field.replaceAccess(provider);
    }

The transform() method is the lone method for this class, as defined by ComponentClassTransformWorker. It uses a method on the ClassTransformation to locate all the unclaimed fields. TransformField is the representation of a field of a component class during the transformation process. As we'll see it is very easy to intercept access to the field.

Some of those fields are final or static and are just ignored. A ComponentValueProvider is a callback object: when the component (whatever it is) is first instantiated, the provider will be invoked and the return value stored into a new field. A FieldValueConduit is an object that takes over responsibility for access to a TransformField: internally, all read and write access to the field is passed through the conduit object.

So, what we're saying is: when the component is first created, use the callback to create a conduit, and change any read or write access to the field to pass through the created conduit. If a component is instantiated multiple times (either in different pages, or within the same page) each instance of the component will end up with a specific FieldValueConduit.

Fine so far; it comes down to what's inside the createFieldValueConduitProvider() method:

    private ComponentValueProvider<FieldValueConduit> createFieldValueConduitProvider(TransformField field)
    {
        final String fieldName = field.getName();
        final String fieldType = field.getType();

        return new ComponentValueProvider<FieldValueConduit>()
        {
            public FieldValueConduit get(ComponentResources resources)
            {
                Object fieldDefaultValue = classCache.defaultValueForType(fieldType);

                String key = String.format("UnclaimedFieldWorker:%s/%s", resources.getCompleteId(), fieldName);

                return new UnclaimedFieldConduit((InternalComponentResources) resources,
                        perThreadManager.createValue(key), fieldDefaultValue);
            }
        };
    }

Here we capture the name of the field and its type (expressed as String). Inside the get() method we determine the initial default value for the field: typically just null, but may be 0 (for a primitive numeric field) or false (for a primitive boolean field).

Next we build a unique key used to store and retrieve the field's value inside the per-thread Map. The key includes the complete id of the component and the name of the field: thus two different component instances, in the same page or across different pages, will have their own unique key.

We use the PerthreadManager service to create a PerThreadValue for the field. You can think of a PerThreadValue as a specialized kind of ThreadLocal that automatically cleans itself up at the end of the request.

Lastly, we create the conduit object. Let's look at the conduit in more detail:

    static class UnclaimedFieldConduit implements FieldValueConduit
    {
        private final InternalComponentResources resources;

        private final PerThreadValue<Object> fieldValue;

        // Set prior to the containingPageDidLoad lifecycle event
        private Object fieldDefaultValue;

        private UnclaimedFieldConduit(InternalComponentResources resources, PerThreadValue<Object> fieldValue,
                Object fieldDefaultValue)
        {
            this.resources = resources;

            this.fieldValue = fieldValue;
            this.fieldDefaultValue = fieldDefaultValue;
        }

We use the special InternalComponentResources interface because we'll need to know if the page is loading, or in normal operation (that's coming up). We capture our initial guess at a default value for the field (remember: null, false or 0) but that may change.

        public Object get()
        {
            return fieldValue.exists() ? fieldValue.get() : fieldDefaultValue;
        }

Whenever code inside the component reads the field, this method will be invoked. It checks to see if a value has been stored into the PerThreadValue object this request; if so the stored value is returned, otherwise the field default value is returned.

Notice the distinction here between null and no value at all. Just because the field is set to null doesn't mean we should switch over the the default value (assuming the default is not null).

The last hurdle is updates to the field:

      public void set(Object newValue)
        {
            fieldValue.set(newValue);

            // This catches the case where the instance initializer method sets a value for the field.
            // That value is captured and used when no specific value has been stored.

            if (!resources.isLoaded())
                fieldDefaultValue = newValue;
        }

The basic logic is just to stuff the value assigned to the component field into the PerThreadValue object. However, there's one special case: a field initialization (whether it's in the component's constructor, or at the point in the code where the field is first defined) turns into a call to set(). We can differentiate the two cases because that update occurs before the page is marked as fully loaded, rather than in normal use of the page.

And that's it! Now, to be honest, this is much more detail than a typical Tapestry developer ever needs to know. However, it's a good demonstration of how Tapestry's class transformation APIs make Java code fluid; capable of being changed dynamically (under carefully controlled circumstances).

Back to pooling: how is this going to affect performance? That's an open question, and putting together a performance testing environment is another task at the top of my list. My suspicion is that the new overhead will not make a visible difference for small applications (dozens of pages, reasonable number of concurrent users) ... but for high end sites (hundreds of pages, large numbers of concurrent users) the avoidance of pooling and page construction will make a big difference!

Tapestry 5.2: Improved Query Parameter Support

I just checked in some very nice changes for Tapestry 5.2; you can now easily store data about a page in the URL as query parameters:

  @ActivationRequestParameter
  private String name;

By annotating a page (not a component!) field this way, the field will be mapped to the query parameter "name". When a page render link or component event link for the page is created, the current value of the field will be added as parameter "name". When that link is triggered to form a request, the parameter will be read and the field updated from the query parameter value.

It isn't limited to strings ... it uses the whole ValueEncoder machinery so that you can encode numbers or even Hibernate entities (represented in the URL as their primary key).

Cool stuff, if I do say so myself. Even I'm still learning how to flex the massive amount of meta-programming muscle that Tapestry provides. It turns out that the combination of component method advice with custom events triggered on the page can do some really sophisticated things!

Who Wants The Func? Gotta Have That Func!

I've been entranced by the concept of laziness since I first really considered it while teaching myself a bit of Haskell. Laziness is the idea that no computation takes place until it is actually needed ... an idea that is common in the functional programming world and one that works best with immutable data.

Why immutable? This has been covered extensively elsewhere, but the gist is that when you have any kind of mutable data (any field that can ever change its value), you add time as an invisible input to your expressions. Literally, the time that any single expression is evaluated relative to other changes to mutable state will affect the outcome of the expression, often in non-predictable ways. In the mathematical world, a function will always return the same value for the same inputs ... in the fuzzy, dirty world of Object Orientation, a method invocation may return different values at different times based on mutable state. Not necessarily mutable state in the object being invoked, but in some other object, somewhere, that the invoked object depends on.

This is why parallel programming in the OO world seems so hard. It requires locks on mutable data, which brings its own problems, such as deadlocks. It can feel like a tottering house of cards.

But remove mutability from this picture and an entirely different world emerges. Functions do behave as functions; same inputs: same result. Side effects disappear, because there's no mutable state. Evaluation of expressions is no longer linked to time: it can be evaluated in parallel threads, or can be deferred until absolutely needed.

That last bit is the laziness. Laziness is a way to bootstrap your code up to a simpler, clearer expression of your algorithms ... once you embrace laziness, you can see that a good amount of the code you write (using mutable data especially) is a case of premature optimization.

Back to Tapestry; as far back as Tapestry 4.0 (where HiveMind and the use of Inversion of Control and Dependency Injection where introduced), Tapestry's internal code has had many functional characteristics. The base unit of work in the Tapestry IoC container is an interface, not a function ... but often those interfaces have a single method. That makes the interfaces look a lot like a function, ready for the kind of composition possible in a functional programming language. Sure, it's a bit clumsy compared to a real functional programming language ... but the power is still there.

Tapestry 5 uses these features to handle a lot of Aspect Oriented Programming concepts; for instance, services are lazily instantiated, and they can be decorated and advised to provide cross-cutting concerns. In fact, Tapestry uses functional composition extensively for all kinds of meta-programming.

Meanwhile outside the realm of Tapestry, my exposure to Clojure has really sold me on the functional approach, and I take to immutable data structures like a warm, comforting blanket. I miss all that when I'm working with ordinary Lists and Sets from Collections API.

Given that Tapestry does a lot of complex things, I started work on a simple functional library. What I've created is not nearly as complex as Functional Java; I think it does less, but does it more cleanly. It's more focused.

The idea is that you'll create a Flow from some source (usually, a Collection). You can then map, filter, reduce, append, concatenate, and iterate the values inside the Flow. Further, the Flows are lazy (as with Haskell and Clojure); all evaluation is deferred until absolutely necessary, and it's thread safe. You can also have infinite flows.

It all starts in a static class F (for Functional) that has the initial factories for Flows. This example uses the F.range() method to create a Flow for a range of integers:

System.out.println(F.range(1, 100).filter(F.gt(10)).map(
  new Mapper() {
   public Integer map(Integer value) {
    return value * 2;
   }
  }).take(3).toList());

When executed, this code prints the following: [22, 24, 26]. That is, it dropped the values less than or equal to 10; it then multiplied each remaining value by 2 and converted the first three to a list.

• F.range() creates a lazy Flow of integers in the range (from 1 to 99; the upper range is non-inclusive).
• filter() is a Flow method that keeps only some values, based on a Predicate
• F.gt() is a static factory method, it creates a Predicate that compares a Number value from the Flow against the provided value
• map() is a Flow method that is applied to each value in the Flow
• take() takes a limited number of values from the front of the Flow
• toList() converts the Flow into a non-modifiable List

Here we mapped from Integer to Integer, but it would have been possible to map to a different type. At each stage, a new (immutable) Flow object is created.

What about laziness? Well if we modify the code a bit:

System.out.println(F.range(1, 100).filter(F.gt(10)).map(
  new Mapper() {
   public Integer map(Integer value) {
    System.out.println("** mapping " + value);
    return value * 2;
   }
  }).take(3).toList());

The new output is:

** mapping 11
** mapping 12
** mapping 13
[22, 24, 26]

... in other words, although we write code that makes it appear that the entire Flow is transformed by the map() call, the reality is that individual values from the original flow are mapped, just once, as needed. The code we write focuses on the flow of transformations, from the input, to the final result: "start with the range, retain the values greather than 10, multiply each by two, keep just the first three".

Does this make a difference? Not with trivial cases like this example. The functional code could be rewritten in standard Java as:

List<Integer> result = new ArrayList<Integer>();
for (int i = 1; i < 100; i++)
{
  if (i >= 10)
  {
    result.add(i * 2);
  }
}

result = result.subList(0, 3);

System.out.println(result);

Yes, this code is shorter, but it does more work (computing many doubles values that are not needed). We could do some extra work to keep a count of result values and end the loop earlier, but that increases the cyclomatic complexity even further. The extra work is not a big deal here, but if the transformations were more expensive (say, re-drawing images in different sizes, or reading data from a database) the work not done unnecessarily would become quite significant.

And is the traditional Java code really shorter? What if we create a reusable factory function:

public static Mapper<Integer,Integer> multiplyBy(final int multiplicand)
{
  return new Mapper<Integer, Integer>() 
  {
    public Integer map(Integer value)
    {
      return value * multiplicand;
    }
  };
}

Then our original Flow expressions becomes:

System.out.println(F.range(1, 100).filter(F.gt(10)).map(multiplyBy(2)).take(3).toList());

... meaning that, once we have a good collection of these Mapper and Predicate factory methods, we can have the efficient, lazy code and it will be more concise and readable.

Anyway, tapestry-func is a work in progress, but it's very promising, and already being used in both Tapestry 5.2, and in some of my clients' code.

Tapestry Happenings: 5.2 beta soon?, New Committers, Dynamic component

There's lots and lots going on with Tapestry right now. We're gearing up to bring Tapestry 5.2 into a beta phase ... hopefully a short one before a final GA release.

We've been busy adding new committers to the team. Not every vote has been successful, but that just shows that the system works.

Lots of people are working on a terrific refresh of the Tapestry home page and documentation. Not just new text, but a new more timely approach (based on Confluence) and a terrific new layout. Just lots of energy going on there. I can't wait for it to be ready ... it really makes Tapestry looks like the first-class citizen it is.

Meanwhile, I've been busy for my clients, writing some useful code.

I've managed to get the nightly builds of the TapX working again ... just in time. I've created an exciting new component: Dynamic.

Dynamic is targeted for use in skinning applications. Tapestry's structure makes skinning an application a little tricky, especially if you want to take a standard application and customize it for different customers. Tapestry templates are expected to be uniform across all instances of a component (in the same way that a classes's methods and fields are fixed across all instances). This makes it tricky to create a component that renders differently for different users, which is the essence of skinning. Often, you end up either simulating a JSP include and injecting a blob of raw markup content into the Tapestry DOM ... or you end up with an evil nest of conditionals and indirection.

The Tapx Dynamic component exists to mix a dynamic template file with live Tapestry components; The dynamic template is any well-formed XML document that can be represented as a Resource; mostly, it is output as-is ... until Tapestry hits an element, such as a <div>, with a special id. When the id is of the form param:block-name, the Dynamic component replaces the dynamic template element with the content of the Block parameter passed to the Dynamic component. We're really weaving Tapestry and the external template together here.

I'm working on a good, simple example here ... but anyone who has struggled with this issue will appreciate what Dynamic accomplishes. It's not unlike running SiteMesh at the Tapestry component level.

Inform 7's Birthday

Long, long ago, some of my first programs were interactive fictions. Really simple stuff, "go north", "kill rat", that kind of thing. While I was hacking that kind of thing together in Basic, others had gone much further: the masters of Infocom. This system, written way back in the late 70's and early 80's, predates Java but has many similar features; including a bytecode-based runtime portable across different operating system and hardware platforms.

Sure games have gone in a different direction with incredible 3d graphics, but there's still a certain joy in playing interactive fiction games; it really is like playing a short story. The games have evolved from classic dungeon crawls into something more, with many of the best games eschewing puzzles and focusing on interaction between the player and the non-player characters.

The modern way to write these games, if you are so inclined, is in the Inform 7 environment. The Inform team try to come up with new releases every year, on the "birthday"; this month is the 17th birthday of the original Inform language (Inform was originally a more C-like objected oriented language that's evolved over the years into its current state).

Here's a getting started screencast:

Inform 7 Introductory Screencast from Aaron Reed on Vimeo.

I've gone a bit deep with Inform in the past, and hope to do more in the future. It's a truly amazing piece of software ... the language is generally a natural language (I call it "the mother of all DSLs") with features combining object oriented, rules based and even aspect oriented programming. In fact, the next release, due shortly, even includes map and reduce operations! The language is very powerful, allowing for concise ways to deal with deep cross-cutting concerns, allowing for human ambiguity

More than that, the IDE is truly full and integrated. It has extensive documentation (both a front to back manual and a cookbook), and error messages include hyperlinks to your code and to the manual pages. It includes hundreds of short examples that can be pasted directly into the editor with a single click. It has built in testing features (shown in the screen cast above). It provides an incredible cross-reference of your project (integrated with the built-in libraries) ... even an automatic map of your game world. It's truly a labor of love, and I wish any of the tools I work with day to day showed so much innovation and usefulness.

If you've ever wanted to write interactive fiction, or just want to play with a really fascinating alternative language, give Inform a try.

Setting up committer access Git for Tapestry 5

Given the problems I'm having, I decided to set up a new local Git repository for futher work. Here's how to do it:

$ git clone git://git.apache.org/tapestry5.git

This sets up a new working folder, tapestry5. It takes it a while to download the necessary Git repository objects.

$ cd tapestry5
$ curl http://git.apache.org/authors.txt -o .git/authors.txt
$ git config svn.authorsfile .git/authors.txt

This fetches the current list of authors so that proper names appear in various Git reports, then configures Git to make use of the file.

$ git svn init --prefix=origin/ --trunk=trunk https://svn.apache.org/repos/asf/tapestry/tapestry5

This tells Git where to sync from and to.

$ git svn rebase

And that finishes things up, ensuring that you have all the most recent revisions.

From here on in, the two commands you need the most are git svn rebase (to pull in repository changes) and git svn dcommit (to push deltas back to Subversion). You should always rebase before a dcommit.

Perhaps that's not quite complete; I generally create local Git topic branches; so I start my work with git co -Blocal to create (or overwrite) my local branch, do my work there as a series of commits, then: git co trunk ; git rebase local to move those commits back over to trunk before git svn dcommit. This helps a lot if you ever have to deal with a merge.

When I'm fixing particular bugs, I often create a branch names after the bug id.

Update: Removed the --tags and --branches arguments ... mostly because of how horribly Git SVN works with branches (don't try it!), and to make the init step nice and fast.

Git & Svn : Not Always A Match Made In Heaven

Apache is stuck using Subversion ... so I've been using the Git/Svn integration built into Git for a while now. The good news is that most of the Git workflow comes with it ... you can create private branches, do local commits to your local repository, and build up a series of changes to dcommit ("delta commit") into SVN.

But that doesn't always work. For reasons I don't understand (given that there have been no other commits to SVN since I started work in my private branch), I keep getting the following error:

Applying: Provide the missing asset request handler for the virtual "stack" folder.
Committing to https://svn.apache.org/repos/asf/tapestry/tapestry5/trunk ...
 M tapestry-core/src/main/java/org/apache/tapestry5/internal/services/ClasspathAssetAliasManagerImpl.java
 M tapestry-core/src/main/java/org/apache/tapestry5/services/LibraryMapping.java
 M tapestry-core/src/test/java/org/apache/tapestry5/internal/services/ClasspathAssetAliasManagerImplTest.java
 A tapestry-core/src/test/java/org/apache/tapestry5/services/LibraryMappingTest.java
Committed r935569
W: 86533530aac8673a9e107e323de5201b7187270f and refs/remotes/origin/trunk differ, using rebase:
:040000 040000 9c78596ee3f916f012c51d8927b4aa31d497f17b 8eb2b9b4f28e825e223c736eaa664bb53018258e M tapestry-core
Current branch trunk is up to date.
# of revisions changed  
before:
 07b37e03cbc17012247d2221e795023c564d8228
0830b5f383dc94ae16088185efefac2e1358cf30
0bf378bcafc3f5372b67edc50d7de5bed8713cd0
95c87c5d7a2435df6bfced0d858bfdcb6ff26f22
3cd4ea4d9b225fd5013e1ce72cb9bac6d5b3e5e2
7b95d935099ebbbeb81845c1b8170a89d6ca6421
af310cfa9a5552aab2574c1e345b3beb049fb040
20805630fc67b83b4ca946b942716aeba4c80bef
b3ef5e069942a30e2dce45a35e4be16382c0108d
1be75a15b7f203c927bc2aa34f43dda59ca968e3
555d94ebab122a688b7a1c0af253bf73609f88f5
729757eb3c35e14e126cb6ef16f5032d95d1cc4a
79dcfa32b291454bf9c652d635374d60638b8fb8
304d12f9d7d040f4dc231d213df663fcdf3863b6
0d626a7b0648735ab83bc7a2fd241390eb92e4e2 

after:
 86533530aac8673a9e107e323de5201b7187270f
07b37e03cbc17012247d2221e795023c564d8228
0830b5f383dc94ae16088185efefac2e1358cf30
0bf378bcafc3f5372b67edc50d7de5bed8713cd0
95c87c5d7a2435df6bfced0d858bfdcb6ff26f22
3cd4ea4d9b225fd5013e1ce72cb9bac6d5b3e5e2
7b95d935099ebbbeb81845c1b8170a89d6ca6421
af310cfa9a5552aab2574c1e345b3beb049fb040
20805630fc67b83b4ca946b942716aeba4c80bef
b3ef5e069942a30e2dce45a35e4be16382c0108d
1be75a15b7f203c927bc2aa34f43dda59ca968e3
555d94ebab122a688b7a1c0af253bf73609f88f5
729757eb3c35e14e126cb6ef16f5032d95d1cc4a
79dcfa32b291454bf9c652d635374d60638b8fb8
304d12f9d7d040f4dc231d213df663fcdf3863b6
0d626a7b0648735ab83bc7a2fd241390eb92e4e2 
 If you are attempting to commit  merges, try running:
  git rebase --interactive --preserve-merges  refs/remotes/origin/trunk 
Before dcommitting
~/work/t5-project
$

I did the right things; git co trunk followed by git svn rebase, then git rebase revised-assets-12apr2010. It claimed to replay my branch changes on top of the trunk branch, but regardless, the dcommit failed.

Doing some hunting around with Google, I found a partial explanation, that at least gives me a way forward. I'd still like to know how I got into this predicament.

At this point I just keep blindly entering the command: git reset --hard 705ccfb1e27d303a9db62de755b2fcfcca9a02f6 ; git svn rebase; git svn dcommit and get one Git commit further each time (that's the Git hash code for my final change in my original branch). Joy.

An extended stay in London

If you were unable to attend my "In the Brain Of" talk last Tuesday, it's now available at SkillsMatter ... voice, video and slides. This was a fun session, even if I was jet lagged to the point of being dizzy.

Meanwhile, I'm still hanging out in London until this at least the end of this week. I expect to work from my hotel room, visit a couple of clients, and maybe have a bit of fun. Drop me a line.

In London, looking for a Tapestry job? I've been contacted by a recruiter who is looking for you! They're building a team of perhaps 20 developers (I'm not yet sure who the actual client is). Drop me a line and I'll hook you up!

Meta-Programming Java with Tapestry

A significant amount of what Tapestry does is meta programming: code that modifies other code. Generally, we're talking about adding behavior to component classes, which are transformed as they are loaded into memory. The meta-programming is the code that sees all those annotations on methods and fields, and rebuilds the classes so that everything works at runtime.

Unlike AspectJ, Tapestry does all of its meta-programming at runtime. This fits in better with live class reloading, and also allows for loaded libraries to extend the meta-programming that's built-in to the framework.

All the facilities Tapestry has evolved to handle meta-programming make it easy to add new features. For example, I was doing some work with the Heartbeat enviromental object. Heartbeat allows you to schedule part of your behavior for "later". First off, why would you need this?

A simple example is the relationship between a Label component and a form control component such as TextField. In your template, you may use the two together:

  <t:label for="email"/>
  <t:textfield t:id="email"/>

The for parameter there is not a simple string, it is a component id. You can see that in the source for the Label component:

    @Parameter(name = "for", required = true, allowNull = false, defaultPrefix = BindingConstants.COMPONENT)
    private Field field;

Why does for="email" match agains the email component, and not some property of the page named email? That's what the defaultPrefix annotation attribute does: it says "pretend there's a component: prefix on the binding unless the programmer supplies an explicit prefix."

So you'd think that would wrap it up, we just need to do the following in the Label code:

  writer.element("label", "for", field.getClientId());

Right? Just ask the field for its client-side id and now all is happy.

Alas, that won't work. The Label component renders before the TextField, and the clientId property is not set until the TextField renders. What we need to do is wait until they've both rendered, and then fill in the for attribute after the fact.

That's where Heartbeat comes in. A Heartbeat represents a container such as a Loop or a Form. A Heartbeat starts, and accumulates deferred commands. When the Heartbeat ends, the deferred commands are executed. Also, Heartbeats can nest.

Using the Heartbeat, we can wait until the end of the current heartbeat after both the Label and the TextField have rendered and then get an accurate view of the field's client-side id. Since Tapestry renders a DOM (not a simple text stream) we can modify the Label's DOM Element after the fact.

Without the meta-programming, it looks like this:

    @Environmental
    private Heartbeat heartbeat;

    private Element labelElement;

    boolean beginRender(MarkupWriter writer)
    {
        final Field field = this.field;

        decorator.beforeLabel(field);

        labelElement = writer.element("label");

        resources.renderInformalParameters(writer);

        Runnable command = new Runnable()
        {
            public void run()
            {
                String fieldId = field.getClientId();

                labelElement.forceAttributes("for", fieldId, "id", fieldId + "-label");

                decorator.insideLabel(field, labelElement);          
            }
        };
        
        heartbeat.defer(command);

        return !ignoreBody;
    }

See, we've gotten the active Heartbeat instance for this request and we provide a command, as a Runnable. We capture the label's Element in an instance variable, and force the values of the for (and id) attributes. Notice all the steps: inject the Heartbeat environmental, create the Runnable, and pass it to defer().

So where does the meta-programming come in? Well, since Java doesn't have closures, it has a pattern of using component methods for the same function. Following that line of reasoning, we can replace the Runnable instance with a method call that has special semantics, triggered by an annotation:

    private Element labelElement;

    boolean beginRender(MarkupWriter writer)
    {
        final Field field = this.field;

        decorator.beforeLabel(field);

        labelElement = writer.element("label");

        resources.renderInformalParameters(writer);

        updateAttributes();

        return !ignoreBody;
    }

    @HeartbeatDeferred
    private void updateAttributes()
    {
        String fieldId = field.getClientId();

        labelElement.forceAttributes("for", fieldId, "id", fieldId + "-label");

        decorator.insideLabel(field, labelElement);
    }

See what's gone on here? We invoke updateAttributes, but because of this new annotation, @HeartbeatDeferred, the code doesn't execute immediately, it waits for the end of the current heartbeat.

What's more surprising is how little code is necessary to accomplish this. First, the new annotation:

@Target(ElementType.METHOD)
@Retention(RUNTIME)
@Documented
@UseWith(
{ COMPONENT, MIXIN, PAGE })
public @interface HeartbeatDeferred
{

}

The @UseWith annotation is for documentation purposes only, to make it clear that this annotation is for use with components, pages and mixins ... but can't be expected to work elsewhere, such as in services layer objects.

Next we need the actual meta-programming code. Component meta-programming is accomplished by classes that implement the ComponentClassTransformationWorker interface.

public class HeartbeatDeferredWorker implements ComponentClassTransformWorker
{
  private final Heartbeat heartbeat;

  private final ComponentMethodAdvice deferredAdvice = new ComponentMethodAdvice()
  {
    public void advise(final ComponentMethodInvocation invocation)
    {
      heartbeat.defer(new Runnable()
      {

        public void run()
        {
          invocation.proceed();
        }
      });
    }
  };

  public HeartbeatDeferredWorker(Heartbeat heartbeat)
  {
    this.heartbeat = heartbeat;
  }

  public void transform(ClassTransformation transformation, MutableComponentModel model)
  {
    for (TransformMethod method : transformation.matchMethodsWithAnnotation(HeartbeatDeferred.class))
    {
      deferMethodInvocations(method);
    }
  }

  void deferMethodInvocations(TransformMethod method)
  {
    validateVoid(method);

    validateNoCheckedExceptions(method);

    method.addAdvice(deferredAdvice);

  }

  private void validateNoCheckedExceptions(TransformMethod method)
  {
    if (method.getSignature().getExceptionTypes().length > 0)
      throw new RuntimeException(
          String
              .format(
                  "Method %s is not compatible with the @HeartbeatDeferred annotation, as it throws checked exceptions.",
                  method.getMethodIdentifier()));
  }

  private void validateVoid(TransformMethod method)
  {
    if (!method.getSignature().getReturnType().equals("void"))
      throw new RuntimeException(String.format(
          "Method %s is not compatible with the @HeartbeatDeferred annotation, as it is not a void method.",
          method.getMethodIdentifier()));
  }
}

It all comes down to method advice. We can provide method advice that executes around the call to the annotated method.

When advice is triggered, it does not call invocation.proceed() immediately, to continue on to the original method. Instead, it builds a Runnable command that it defers into the Heartbeat. When the command is executed, the invocation finally does proceed and the annotated method finally gets invoked.

That just leaves a bit of configuration code to wire this up. Tapestry uses a chain-of-command to identify all the different workers (theres more than a dozen built in) that get their chance to transform component classes. Since HeartbeatDeferredWorker is part of Tapestry, we need to extend contributeComponentClassTransformWorker() in TapestryModule:

  public static void contributeComponentClassTransformWorker(
      OrderedConfiguration<ComponentClassTransformWorker> configuration
  {
  
    ...
    
    configuration.addInstance("HeartbeatDeferred", HeartbeatDeferredWorker.class, "after:RenderPhase");
  }      

Meta-programming gives you the ability to change the semantics of Java programs and eliminate boiler-plate code while you're at it. Because Tapestry is a managed environment (it loads, transforms and instantiates the component classes) it is a great platform for meta-programming. Whether your concerns are security, caching, monitoring, parallelization or something else entirely, Tapestry gives you the facilities to you need to move Java from what it is to what you would like it to be.

In The Brain Of Howard Lewis Ship

While I'm in London for three days of Tapestry 5 Training, I'll also be giving an evening In The Brain Of talk ... on Tapestry, because there's not that much else rattling around my brain lately. Whereas Ben's talk was about lessons learned at the tail end of a Tapestry project, my talk gives you a point of reference on what Tapestry is all about, and why you want to start using it.

Swing by, take in the talk, and come on out for a pint or two! The talk is Tuesday, April 13th at 18:30.

Getting Ready for SkillsMatter Training

Talk about putting too much on your plate ... I'm working to update my existing Tapestry workshop to release 5.2, even though it is an alpha snapshot. The new features, especially live class reloading in the services layer, are just too useful to ignore.

Improvements to Live Class Reloading in 5.2

I just spent the morning making some improvements and fixes to live class reloading. The umbrella for reloading has now extended to any class that Tapestry instantiates ... whether is by ObjectLocator.proxy() or any variation of Configuration.addInstance(). In all those cases, when Tapestry is doing the instantiating, and there's a service interface, what gets created is a reloadable proxy ... as long as there is a class file available.

The interesting part is that for development you get the live class reloading behavior. In production, you will not, because all the classes will be packaged up in JARs, not on the file system.

Tapestry Training Returns to London: April 14 - 16, 2010

I'll be returning to SkillsMatter to teach my Tapestry workshop. The course runs from April 14th to the 16th. I had a great time teaching the course back in February: SkillsMatter has a great facility, just perfect for a hands-on class like this one.

Procrastination and JavaOne 2010: See you in 2011!

Well, that's what I get for waiting until the last day ... by the time I had a chance to put together a submission or two for JavaOne 2010, the site was down (from about 10pm on). Kind of frustrating, I was looking forward to talking about Tapestry and Clojure (my main speaking staples) in front of another big crowd.

Live Service Reloading in Tapestry 5.2

A common question I get during Tapestry training sessions is: Why can't Tapestry reload my services as well as my pages and components?. It does seem odd that I talk about how agile Tapestry is, with the live class reloading, and how nicely OO it is, what with services ... but when you move common logic to a service, you lose the agility because services do not live reload.

This came up yet again, during my latest training session, in London.

I've considered this before, and I've been opposed to live service reloading for a couple of reasons. First, live reloading requires creating new class loaders, and that causes conflicts with other frameworks and libraries. You get those crazy ClassCastExceptions even though the class name is correct (same name, different class loader, different class). Further, in Tapestry IoC, services can be utilized to make contributions to other services ... changing one service implementation, or one module, can cause a ripple effect across an untraceable number of other services. How do you know what needs to be reloaded or re-initialized?

When I last really considered this, back in the HiveMind days, my conclusion was that it was not possible to create a perfect reloading process: one that would ensure that the live-reloaded Registry (and all of its services with all their internal state) would be an exact match for what configuration you'd get by doing a cold restart.

So I shelved the idea, thinking that simply redeploying the WAR containing the application (and the services and modules) would accomplish the desired effect.

But as they say, The Perfect Is The Enemy Of The Good. One very sharp student, Andreas Pardeike, asked: Why not just reload the service implementations?.

Why not indeed? Why not limit the behavior to something understandable, trackable, and not very expensive. Most of the infrastructure was already present, used for reloading of component classes. What about ClassCastExceptions? In Tapestry, service implementations are already buried under multiple layers of dynamically generated proxies that implement the service interface. The underlying service implementation is never automatically exposed.

A few hours of work later ... and we have live service reloading. Each reloadable service gets its own class loader, just to load the service implementation class. When Tapestry is periodically checking for updated files, it checks each reloadable service. If necessary, the existing instance, class and class loader is discarded and a new class loader created for the updated .class file.

This is going to make a big difference for me, and for most Tapestry developers. Both applications I'm working on have enough Hibernate entities and other clutter to take some time (20 - 30 seconds) to restart, and most functionality is hidden past a login page. Being able to change a service, for example to tweak a Hibernate query, with the same speed with which I can tweak a template or component class, is just one more thing to keep me in the flow and super productive.

Give it a try ... it's one more step towards making Tapestry so compelling, you wouldn't think of using anything else!

Why Eclipse leaves me wanting

I think I've come to understand why Eclipse leaves me always feeling a bit frustrated. Yes, it is more stable than IDEA, uses less memory, has some documentation, and a lot of acceptance ... but even so, it just leaves me cold (and I was an early adopter, signed up for the beta way back in 2000!).

Keystrokes are not modal

The fact that I can type a common keystroke into an Eclipse window and not know what it will do is painful. How a keystroke is interpreted depends on what perspective is active, what view or editor has focus, and what kind of data is being edited in the editor. That's dead wrong; keystrokes are about muscle memory, and muscle memory remembers motion, not context. The end result is that I get frustrated hitting keystrokes and seeing nothing happening. It doesn't help that I cycle between Mac and a PC on most days.

You can't have it your way

A tool as powerful and extensible as Eclipse walks the tightrope of offering lots of features and customizations without overwhelming the user. Alas, Eclipse is lying in a broken heap fifty feet below that tightrope. Eclipse has an unending set of options and defaults for things I don't care about, but anything I do care about seems to never be presents. Here's a few ideas of the top of my head:

• Stop running launches when I close the project (I often have to kill them from the command line)
• Give me a quick way to stop all running launches
• Why so many steps to implement an interface? It's the second most common thing I do!
• How about a button to quickly relaunch the current running launch?
• Why are the available refactorings so paltry and where are the 3rd party ones?

Who's eating their own dog food?

When I used IDEA, I was constantly struck by little details that showed that the IDE developers were also its prime users. For example, it has open-type and open-resource dialogs much like Eclipse ... but each recognizes the keystroke for the other, so that if you mistakenly activate the open-type dialog, you just hit the normal keystroke to switch over to open-resource. Eclipse makes you cancel the dialog first.

Another example: in IDEA if you rename a field, it notices the getter and setter and will offer to rename those as well.

IDEA also has lots of quick fixes everywhere, such as "implement this interface" and lots of other tiny, cool things I miss every single day I use Eclipse. It's been about a year since I gave up on IDEA and I still miss it.

Is it cultural or organizational? Eclipse gives me the impression that day-to-day developers either have no concept of how the IDE gets used (and what rough spots are causing some serious chafing) OR they are somehow prohibited from fixing things that are obviously wrong.

If you love IDEA so much why don't you marry it?

So why don't I use IDEA anymore? Two main reasons: first, it's become very bloated, to the point that unless you go in and shut off a ton of features, it's unusable on my hardware. Merlyn has the same problem doing GWT work on his MacBook Pro ... all the help it gives you comes at a cost in terms of CPU and memory utilization and some instability.

Secondly, I tried (even before IDEA went open source) to use IDEA in my training labs and I hit a stone wall of non-acceptance. Switching to Eclipse was a benefit to my students since, even running in Ubuntu instead of Windows, it was familiar and easy to navigate. It also out-performs IDEA inside my Ubuntu Virtual Machine. I simply lack the ability to switch between the two on a constant basis without getting completely confused and frustrated. I had to choose one, and I chose Eclipse: stable and accepted, even if it is brain dead.

Why call it Ugly?

One thing I don't get is how many people claim Eclipse is "ugly" and IDEA "beautiful". I found IDEA to be overly chock-full of modal dialogs and a number of improperly resized (or non-resizable) dialogs and windows. It's a real dog's breakfast in terms of UI, and has the classically ugly Swing look and feel.

I've always found Eclipse to look sharp and somewhat elegant. You can have a debate about the technical merits of SWT vs. AWT and Swing, or the ability to tune Swing to look like SWT ... but SWT out of the box is simply a better L&F visually.

On a Mac they both suck at keyboard navigation, though.

There, I've vented. See what going cold-turkey from Twitter can do?

Java Champion

You might call it petty, you might call it vain, but I've aspired to be recognized as a Java Champion for the last couple of years. The process by which you are selected for this is a bit secretive, but I've finally gotten the nod and joined the roster.

My larger goal for Tapestry has always been to create a web application platform so compelling that it would draw developers to the Java programming language, just to be able to use it. Of course, that's not so much a goal as it is a journey. Technologically, I think Tapestry has the chops to embrace that goal (or journey) ... and looking at current discussions and developments in the Tapestry world, I think the other critical areas where Tapestry is lagging (namely, Documentation and Marketing) may come around.

Want to do your part? Blog about Tapestry ... what you like, what you don't like, what's missing, and what's hidden.