LinkedIn Etiquette

I've used LinkedIn for many years now, long before I joined Facebook ... I liked the concept of never losing contact information with business contacts and technologist. It just seemed like a good idea (though I do sometimes wonder if LinkedIn has any particular purpose).

I tend to only connect with people I've met in person, or at least talked to on the phone.

One thing that drives me crazy about LinkedIn is that you aren't forced to customize the message. As far as I'm concerned, the default message is like no message at all, and its a sign that you are just trolling for contacts. Just like you should always write a cover letter for a resume.

So the first rule of LinkedIn is: always customize the message. You chances of a connection go up significantly (at least with me) ... and with the default message, your chances drop down to near zero.

Tapestry Advantages

A summary of a discussion about the advantages of Tapestry over Struts:

• Exceptional exception reporting
• Significantly less code
• Live class reloading
• Sensible defaults, especially for SEO-friendly URLs
• Great community
• Flexibility and customizability

Interestingly, the quality of Tapestry's documentation was mentioned ... favorably! Between the revised home page, and Tapestry JumpStart (and Igor's coming book), I think we're headed in the right direction in terms of documentation going from a liability to an asset.

Tapestry 5.4: Focus on JavaScript

Tapestry 5.3.1 is out in the wild ... and if Tapestry is to stay relevant, Tapestry 5.4 is going to need to be something quite (r)evolutionary.

There was some confusion on the Tapestry developer mailing list in advance of this blog post; I'd alluded that it was coming, and some objected to such pronouncements coming out fully formed, without discussion. In reality, this is just a distillation of ideas, a starting point, and not a complete, finalized solution. If it's more detailed than some discussions of Tapestry's evolution in the past, that just means that the mailing list discussion and eventual implementation will be that much better informed.

In posts and other conversations, I've alluded to my vision for Tapestry 5.4. As always, the point of Tapestry is to allow developers to code less, deliver more, and that has been the focus of Tapestry on the server side: everything drives that point: terseness of code and templates, live class reloading, and excellent feedback are critical factors there. Much of what went into Tapestry 5.3 strengthened those points ... enhancements to Tapestry's meta-programming capabilities, improvements to the IoC container, and reducing Tapestry's memory footprint in a number of ways. I have one client reporting a 30% reduction in memory utilization, and another reporting a 30 - 40% improvement in execution speed.

Interestingly, I think that for Tapestry to truly stay relevant, it needs to shift much, much, more of the emphasis to the client side. For some time, Tapestry has been walking a fine line with regards to the critical question of where does the application execute? Pre-Ajax, that was an easy question: the application runs on the server, with at most minor JavaScript tricks and validations on the client. As the use of Ajax has matured, and customer expectations for application behavior in the browser have expanded, it is no longer acceptable to say that Tapestry is page based, with limited Ajax enhancements. Increasingly, application flow and business logic need to execute in the browser, and the server-side's role is to orchestrate and facilitate the client-side application, as well as to act as a source and sink of data ultimately stored in a database.

As Tapestry's server-side has matured, the client side has not kept sufficient pace. Tapestry does include some excellent features, such as how it allows the server-side to drive client-side JavaScript in a modular and efficient way. However, that is increasingly insufficient ... and the tension caused by give-and-take between client-side and server-side logic has grown with each release.

Nowhere is this more evident than in how Tapestry addresses HTML forms. This has always been a tricky issue in Tapestry, because the dynamic rendering that can occur needs to be matched by dynamic form submission processing. In Tapestry, the approach is to serialize into the form instructions that will be used when the form is submitted (see the store() method of the FormSupport API). These instructions are used during the processing of the form submission request to re-configure the necessary components, and direct them to read their query parameters, perform validations, and push updated values back into server-side objects properties. If you've ever wondered what the t:formdata hidden input field inside every Tapestry forms is about ... well, now you know: it's a serialized stream of Java objects, GZipped and MIME encoded.

However, relative to many other things in Tapestry, this is a bit clumsy and limited. You start to notice this when you see the tepid response to questions on the mailing list such as "how to do cross-field validation?" Doing more complicated things, such as highly dynamic form layouts, or forms with even marginal relationships between fields, can be problematic (though still generally possible) ... but it requires a bit too much internal knowledge of Tapestry, and the in-browser results feel a bit kludgy, a bit clumsy. Tapestry starts to feel like it is getting in the way, and that's never acceptible.

Simply put, Tapestry's abstractions on forms and fields is both leaky and insufficient. Tapestry is trying to do too much, and simply can't keep up with modern, reasonable demands in terms of responsiveness and useability inside the client. We've become used to pages rebuilding and reformatting themselves even while we're typing. For Tapestry to understand how to process the form submission, it needs a model of what the form looks like on the client-side, and it simply doesn't have it. There isn't an effective way to do so without significantly restricting what is possible on the client side, or requiring much more data to be passed in requests, or stored server-side in the session.

The primary issue here is that overall form submission cycle, especially combined with Tapestry's need to serialize commands into the form (as the hidden t:formdata field). Once you add Ajax to this mix, where new fields and rules are created dynamically (on the server side) and installed into the client-side DOM ... well, it gets harder and harder to manage. Add in a few more complications (such as a mix of transient and persistent Hibernate entities, or dynamic creation of sub-entities and relationships) into a form, it can be a brain burner getting Tapestry to do the right thing when the form is submitted: you need to understand exactly how Tapestry processes that t:formdata information, and how to add your own callbacks into the callback stream to accomplish just exactly the right thing at just exactly the right time. Again, this is not the Tapestry way, where things are expected to just work.

Further, there is some doubt about even the desirability of the overall model. In many cases, it makes sense to batch together a series of changes to individual properties ... but in many more, it is just as desirable for individual changes to filter back to the server (and the database) as the user navigates. Form-submit-and-re-render is a green screen style of user interaction. Direct interaction is the expectation now, and that's something Tapestry should embrace.

What's the solution, then? Well, it's still very much a moving target. The goal is to make creating client-side JavaScript libraries easier, to make it easier to integrate with libraries such as jQuery (and its vast library of extensions), make things simpler and more efficient on the client side, and not sacrifice the features that make Tapestry fun and productive in the first place.

Overall Vision

The overall vision breaks down into a number of steps:

• Reduce or remove outside dependencies
• Modularize JavaScript
• Change page initializations to use modules
• Embrace client-side controller logic

Of course, all of these steps depend on the others, so there isn't a good order to discuss them.

Reducing and removing outside dependencies

Tapestry's client-side strength has always been lots of "out of the box" functionality: client-side validation, Zones and other Ajax-oriented behaviors, and a well-integrated system for performing page-level initializations.

However, this strength is also a weakness, since that out of the box behavior is too tightly tied to the Prototype and Scriptaculous libraries ... reasonable choices in 2006, but out-of-step with the industry today. Not just in terms of the momentum behind jQuery, but also in terms of very different approaches, such as Sencha/ExtJS and others.

It was a conscious decision in 2006 to not attempt to create an abstraction layer before I understood all the abstractions. I've had the intermediate time to embrace those abstractions. Now the big problem is momentum and backwards compatibility.

Be removing unnecessary behaviors, such as animations, we can reduce Tapestry's client-side needs. Tapestry needs to be able to attach event handlers to elements. It needs to be able to easily locate elements via unique ids, or via CSS selectors. It needs to be able to run Ajax requests and handle the responses, including dynamic updates to elements.

All of these things are reasonable to abstract, and by making it even easier to execute JavaScript as part of a page render or page update (something already present in Tapestry 5.3), currently built-in features (such as animations) can be delegated to the application, which is likely a better choice in any case.

Modularizing JavaScript

Tapestry has always been careful about avoiding client-side namespace polution. Through release 5.2, most of Tapestry's JavaScript was encapulated in the Tapestry object. In Tapestry 5.3, a second object, T5 was introduced with the intention that it gradually replace the original Tapestry object (but this post represents a change in direction).

However, that's not enough. Too often, users have created in-line JavaScript, or JavaScript libraries that defined "bare" variables and functions (that are ultimately added to the browser's window object). This causes problems, including collisions between components (that provide competing definitions of objects and functions), or behavior that varies depending on whether the JavaScript was added to the page as part of a full-page render, or via an Ajax partial page render.

The right approach is to encourage and embrace some form of JavaScript module architecture, where there are no explicit global variables or functions, and that all JavaScript is evaluated inside a function, allowing for private variables and functions.

Currently, I'm thinking in terms of RequireJS as the way to organize the JavaScript. Tapestry would faciliate organizing its own code into modules, as well as application-specific (or even page-specific) JavaScript modules. This would mean that de-referencing the T5 object would no longer occur (outside of some kind of temporary compatibility mode).

For example, clicking a button inside some container element might, under 5.3, publish an event using Tapestry's client-side publish/subscribe system. In the following example, the click events bubble up from the buttons (with the button CSS class name) to a container element, and are then published under the topic name button-clicked.

Consider this an abbreviated example, as it doesn't explain where the element variable is defined or initialized; the important part is the interaction with Tapestry's client-side library: the reference to the T5.pubsub.publish function.

Under 5.4, using the RequireJS require function, this might be coded instead as:

Here, the t5/pubsub module will be loaded by RequireJS and passed as a parameter into the function, which is automatically executed. So, this supports JavaScript modularization, and leverages RequireJS's ability to load modules on-the-fly, as needed.

Notice the difference between the two examples; in the first example, coding as a module was optional (but recommended), since the necessary publish() function was accessible either way. In the 5.4 example, coding using JavaScript modules is virtually required: the anonymous function passed to require() is effectively a module, but its only through the use of require() (or RequireJS's define()) that the publish() function can be accessed.

This is both the carrot and the stick; the carrot is how easy it is to declare dependencies and have them passed in to your function-as-a-module. The stick is that (eventually) the only way to access those dependencies is by providing a module and declaring dependencies.

Change page initializations to use modules

Tapestry has a reasonably sophisticated system for allowing components to describe their JavaScript requirements as they render, in the form of the JavaScriptSupport environmental (an environmental is a kind of per-thread/per-request service object). Methods on JavaScriptSupport allow a component to request that a JavaScript library be imported in the page (though this is most commonly accomplished using the Import annotation), and to request the initialization functions get executed.

Part of Tapestry's Ajax support is that in an Ajax request, the JavaScriptSupport methods can still be invoked, but a completely different implementation is responsible for integrating those requests into the overall reply (which in an Ajax request is a JSON object, rather than a simple stream of HTML).

Here's an example component from the TapX library:

The @Import annotation directs that a stack (a set of related JavaScript libraries, defined elsewhere) be imported into the page; alternately, the component could import any number of specific JavaScript files, located either in the web application context folder, or on the classpath.

Inside the afterRender() method, the code constructs a JSONObject of data needed on the client side to perform the operation. The call to addInitializerCall references a function by name: this function must be added to the T5.Initializers namespace object. Notice the naming: tapxExpando: a prefix to identify the library, and to prevent collisions with any other application or library that also added its own functions to the T5.initializers object.

The JavaScript library includes the function that will be invoked:

Under 5.4, this would largely be the same except:

• There will be a specific Java package for each library (or the application) to store library modules.
• The JavaScriptSupport environmental will have new methods to reference a function, inside a module, to invoke.
• Stacks will consist not just of individual libraries, but also modules, following the naming and packaging convention.

Embrace client-side controller logic

The changes discussed so far only smooth out a few rough edges; they still position Tapestry code, running on the server, as driving the entire show.

As alluded to earlier; for any sophisticated user interface, the challenge is to coordinate the client-side user interface (in terms of form fields, DOM elements, and query parameters) with the server-side components; this is encoded into the hidden t:formdata field. However, it is my opinion that for any dynamic form, Tapestry is or near the end of the road for this approach.

Instead, it's time to embrace client-logic, written in JavaScript, in the browser. Specifically, break away from HTML forms, and embrace a more dynamic structure, one where "submitting" a form always works through an Ajax update ... and what is sent is not a simple set of query parameters and values, but a JSON representation of what was updated, changed, or created.

My specific vision is to integrate Backbone.js (or something quite similar), to move this logic solidly to the client side. This is a fundamental change: one where the client-side is free to change and reconfigure the UI in any way it likes, and is ultimately responsible for packaging up the completed data and sending it to the server.

When you are used to the BeanEditForm component, this might feel like a step backwards, as you end up responsible for writing a bit more code (in JavaScript) to implement the user interface, input validations, and relationships between fields. However, as fun as BeanEditForm is, the declarative approach to validation on the client and the server has proven to be limited and limiting, especially in the face of cross-field relationships. We could attempt to extend the declarative nature, introducing rules or even scripting languages to establish the relationships ... or we could move in a situation that puts the developer back in the driver's seat.

Further, there are some that will be concerned that this is a violation of the DRY pricipal; however I subscribe to different philosophy that client-side and server-side validation are fundamentally different in any case; this is discussed in an excellent blog post by Ian Bickling.

Certainly there will be components and services to assist with this process, in term of extracting data into JSON format, and converting JSON data into a set of updates to the server-side objects. There's also a number of security concerns that necessitate careful validation of what comes up from the client in the Ajax request. Further, there will be new bundled libraries to make it easier to build these dynamic user interfaces.

Conclusion

In this vision of Tapestry's future, the server-side framework starts to shift from the focus of all behavior to the facilitator: it paints the broad stokes on the server, but the key interactions end up working exclusively on the client.

I'm sure this view will be controversial: after all, on the surface, what the community really wants is just "jQuery instead of Prototype". However, all of the factors described in the above sections are, I feel, critical to keeping Tapestry relevant by embracing the client-side in the way that the client-side demands.

I think this change in focus is a big deal; I think it is also necessary for Tapestry to stay relevant in the medium to long term. I've heard from many individual developers (not necessarily Tapestry users) that what they really want is "just jQuery and a restful API"; I think Tapestry can be that restful API, but by leveraging many of Tapestry's other strengths, it can be a lot more. Building something right on the metal feels empowering ... until you hit all the infrastructure that Tapestry provides, including best-of-class exception reporting, on-the-fly JavaScript aggregation and minimization, and (of course) live class reloading during development. java

I'm eager to bring Tapestry to the forfront of web application development ... and to deliver it fast! Monitor the Tapestry developer mailing list to see how this all plays out.

Review: Gradle Class with Luke Daley

Last week, Luke Daly arrived in Portland to teach a three day Gradle class; the folks at Gradleware were nice enough let me audit the class (so it only cost me a couple of thousand dollars of lost billing revenue to attend). My goals for the class was to gain a deeper understanding of how Gradle works, so that I could write more efficient builds, diagnose problems, and write my own plugins. The class scored very high on all of those counts!

Much of the first day was spent on basics, including a very useful introduction to the Groovy programming language (which is the basis of the Gradle DSL). Even though I have used Groovy pretty extensively for testing purposes over the last couple of years, there were features I've glossed over in the past that turned out to be very useful.

The class is taught largely bottom up: from the basics of declaring tasks, then actions on tasks, and gradually working up towards defining tasks inputs and outputs. Compiling Java came pretty late, which seems curious since that's the primary job of Gradle, but this makes sense from a bottom-up approach as SourceSets can then be described correctly.

The class alternates between lecture and discussion, and short focused labs. At the end of the third day, we worked on tuning up our own builds, passing the video projector cable around ... we had a good time making the Tapestry build more efficient and readable, as well as adding new features to it.

I'm feeling very good about my Gradle skills after attending the course; I've already been able to make further improvements to the Tapestry build subsequently, and I have plans for more involving things going forward.

The return on investment for this class is a bit tricky; Gradle simply does so much straight out of the box, and even without a strong understanding of its structure, you can kind of get it to do what you want just by guesswork and Googling. If you've been using Gradle for a while, I give this a cautious recommendation: getting back the three days of invested time may take a while to pay off. On the other hand, if you are currently dependent on Ant or Maven ... sign up for the next class and get yourself switched over, today!

Hackergarten in PDX - Friday January 20th

Merlyn Albery-Speyer is organizing a Hackergarten while Luke Daley (creator of Geb, and Gradle committer) is in town to run an in-depth Gradle training. I'll be there, working on Tapestry, or Gradle, or a video game, or something.

Please see Meryln's blog to RSVP. I look forward to meeting and coding with more PDX peeps!

Adding "Ajax Throbbers" to Zone updates

A common desire in Tapestry is for Zone updates to automatically include a throbber (or "spinner") displayed while the Ajax update is in process. This is, unfortunately, a space where the built-in Tapestry 5.3 Zone functionality is a bit lacking. Fortunately, it's not too hard to hard it in after the fact.

This solution involves a JavaScript library, two CSS stylesheet files (one is IE specific), plus the "throbber" image. Typically, you'll bind all of these things together in your application's Layout component.

First, the JavaScript. We need to intercept links and forms that update a Zone. When such a request starts, we add a <div> to the top of the Zone's client-side element. When the update from the server arrives, the entire content of the Zone's element will be replaced (so we don't have to worry about clearing the <div> explicitly).

When a form is submitted with Ajax, to update a Zone, Tapestry fires a client-side event on the Form; the Tapestry.FORM_PROCESS_SUBMIT_EVENT constant provides the event name. The primary handler for this event is the code that actually performs the XmlHTTPRequest and sets up a handlers for the response; the above code adds a second handler that adds the Ajax overlay.

Likewise, when a link is used to update a Zone, there's a second client-side event; again, the primary handler for the event does the actual Ajax work, but the same logic allows the Zone to be decorated with the overlay.

The overlay consists of a <div> that will visually mark the entire zone's content and consume any mouse clicks during the Ajax update. The CSS associated with the zone-ajax-overlay CSS class sets up a translucent background color and the spinning Ajax throbber.

Next up is the CSS:

This little bit of CSS is doing quite a bit. Firstly, if the Ajax request is very quick, then there will be an annoying flicker; to combat this, we've set up a simple CSS animation to delay the animation momentarily, long enough that fast requests will just see the new content pop into place. There's probably a bit of room here to tweak the exact timing.

Alas, in the current world, we need to do a bit of work to support both Firefox (the -moz prefix) and WebKit (Safari, Chrome, the -webkit prefix). This is really calling out for a SASSy solution.

You'll also see an animated image for the throbber. I used ajaxload.info to create one.

But what about Internet Explorer? It doesn't understand the animation logic, and it does CSS opacity differently from the others. Fortunately, we can segregate those differences in a separate CSS file.

Lastly, we put all this together inside the application's Layout component:

The @Import annotation does the easy imports of the main CSS and JavaScript.
Tapestry 5.3 supports IE conditional stylesheets ... but this requires just a bit of code as the @Import annotation doesn't support adding a condition, as this is a fairly rare requirement.

Instead, the IE-specific CSS is injected into the page as an Asset object; this can be combined with StylesheetOptions to form a StylesheetLink, which can be imported into the page.

With this in place, every page will include both CSS stylesheets (one as an IE-only conditional comment) and the necessary client-side logic ... and every Zone update will get this uniform treatment.

There's some limitations here; in Tapestry it's possible for the server-side to push updates into multiple Zones. The client-side doesn't even know that's happening until it gets the reply, so there's no universal way to add overlays to multiple zones when the request is initiated.

Secondly, in rare cases, a Zone update may only update other Zones, and leave the initiating Zone's content unchanged. In that case, you may find that the Zone's throbber is still in place after the response is handled! I'll leave it as an exercise to the reader on how to deal with that.

Dissecting a Tapestry Operation Trace

I'm helping out a client who is having a problem using Spock and Tapestry 5.3 together. The Spock/Tapestry integration was created for Tapestry 5.2, and some subtle change in the Tapestry 5.3 IoC container has boned the integration, so running even a simple test results in an exception, with a very big stacktrace:

The part at the top (the numbered entries) are the operation trace: Tapestry tracks what its doing at all times (using a stack stored inside a ThreadLocale) just so it can report this information in case there's an error. That's part of Tapestry's commitment to useful feedback. The operation tracing was enhanced a bit in Tapestry 5.3 to be a bit more verbose and all-reaching.

The operation trace is providing a wealth of information about how Tapestry got to the point where an exception was thrown. This is much more useful than just the huge stack trace (about 400 frames!) since Tapestry, by design, tends to call through the very same methods repeatedly; stack traces are less useful when what counts are the parameters to the methods, rather than the methods themselves.

It takes a while to figure out, but the key operations are:

That operation corresponds to invoking this method:

Translated to English, this code says:

When starting up the Registry (the odd name for the Tapestry IoC container), execute this block of code, that checks to see if early startup of Hibernate is desired and, if so, forces the initialization of Hibernate by invoking the getConfiguration() method (otherwise, the initialization would happen lazily the first time a request needed to use the Hibernate Session).

The @Symbol annotation means that the parameter's value is derived from a Tapestry configuration symbol, which is a flexible, late-binding way to configure services as they are instantiated. In other words, because of the use of a symbol, rather than a constant, the actual value passed in can't be determined until runtime ... which is a good thing; it means a developer can configure the symbol's value locally, but a different default value is in effect for the production application. Sometimes you want early startup, sometimes you don't.

In order to resolve the value of a symbol, Tapestry must instantiate the SymbolSource service; it has its own configuration that depends on other services, including ApplicationDefaults, FactoryDefaults, as well as a few other simple objects that implement the SymbolProvider interface, but are not services.

There's also a hidden dependency here: Starting in Tapestry 5.3, Tapestry will attempt to type-coerce contributed values (in this case, symbol values contributed to ApplicationDefaults or FactoryDefaults) from their actual type, to the expected type. That shows up in operation 14:

Those integers and booleans need to be converted to Strings; Tapestry 5.3 invokes the full machinery of the TypeCoercer service to do this coersion, seen as operation 15.

At operations 21 - 23, Tapestry sees the ThreadLocale service (which stores the active Locale used during processing of the request; something that can vary on a request-by-request basis). The ThreadLocale service uses a special service lifecycle that enforces that the instance is stored as a per-thread singleton, not a per-Registry singleton, and will be discarded at the end of each request.

The ServiceLifecycleSource service is the source for these ServiceLifecycle objects.

At operation 28 - 31, the Spock/Tapestry integration code is getting involved. It adds a special service lifecycle just for values that are part of a Spock specification ... and we're finally reaching the problem point!

The Spock/Tapestry integration is using the addInstance() method, which instantiates a class with dependencies; this is operation 30. This is the problem point, but it's not obvious why its causing an eventual exception.

Because of the use of addInstance(), Tapestry must locate and inject the dependencies of the PerIterationServiceLifecycle class, including the IPerIterationManager service (operation 31).

In Tapestry, there is a mechanism to replace services with overrides; this is the ServiceOverride service and its configuration. It's super handy for extending Tapestry in completely unexpected ways.

That brings us to some code, new in Tapestry 5.3, at operation 38:

And that brings us to the actual cause. Notice the @Symbol annotation ... remember way back to operation 7, that required the TypeCoercer (operation 15) ... well, we're not done with that yet, but this production override code has a @Symbol annotation that requires the TypeCoercer ... which is still in the middle of being instantiated.

Yes, this takes a lot of detective work ... this is something of an Achilles' Heel of Tapestry's IoC container; since much of the functionality of the container is defined in terms of other functionality of the container, you can get into these hidden dependency cycles when tweaking some of the more fundamental aspects of Tapestry, such as TypeCoercer contributions, or adding new service lifecycles. This is unfortunate, since so much else in Tapestry's web framework and IoC container Just Works^TM.

In terms of fixing this ... turns out the Spock/Tapestry integration has some other dependencies on 5.2, making use of internal classes and constructors that no longer exist in 5.3. I'll be forking their code shortly to produce a 5.3 compatible version.

However, my take-away here is: the system works, the emphasis on feedback, and the generation of useful operation traces, makes this detective work even possible. The alternative would have taken far, far longer ... using the debugger to try and work backwards to what Tapestry was trying to do. It's so much better to have Tapestry simply tell you what you need to know!

Mac Tips: Preventing Aperture from Launching

Since I tend to connect and disconnect my Android phone to my Mac pretty often, I got frustrated that it kept launching Aperture every time (it used to do the same with iPhoto, before I switched).

In any case, the solution for this is easy enough; with the phone connected, launch the ImageCapture.app: select the phone, and choose "No application" from the drop down list in the bottom left corner:

This is under Lion; your mileage may vary under earlier versions. Hope it helps!