How we can be more secure with mail-in ballots than touch screens

Yes, it's election day! I didn't wait in line ... I live in Oregon, where balloting is done by mail. That's a much better system.

Of course, people are suspicious of voting electronically, or by mail. Somehow the mechanical interaction reassures people. However, security has been an issue for the entire history of voting. Ballots can be "misplaced", or electronically stored values from modern voting machines can be invisibly tampered with, after the fact. Certainly, as it is election day, we're hearing stories about devices that make it hard to select specific candidates ... and the conspiracy theorists are certain that the fix is in.

It may be, but the cases I've heard of are simply bad touch screens. I wonder what the conspiracy theorists believe is going on when they attempt to withdraw $25 from an ATM and get $10 instead?

Real attempts to fix the elections don't change what you enter; they change what is stored ... possibly long after you have left the voting booth.

I would say that having fewer centralized counting locations would be more secure. First of all, we could use (as we do in Oregon), optical scanners ... the same, basic, trusted technology that's used for the SATs.

Next, the process of opening the secure envelopes and feeding them to the scanners could be more easily monitored if it occurred in a few locations across each state, rather than in every polling location across the entire country. This process is subject to fraud, yes, but is probably more likely to be affected by simple incompetence.

With physical ballots, it is also possible to do a proper recount. Voting on a touch screen is too ephemeral for a recount to have any meaning.

Ultimately, there is no way to ensure that your vote is actually counted. However, I think we can come up with a way in which we can at least ensure that your ballot made it to the processing center.

Imagine if the ballot mailed to your home had a unique bar code on it, as well as a "tear off" tab with the corresponding unique number. In addition, there is a section on the ballot where you could enter a four or six digit PIN code of your own choice.

Later, at home, you visit a web site: you provide your unique, randomly assigned ballot id as well as your own secret, personally-selected, PIN code. The government web site (built using open-source software, of course) could then identify the status of your vote, just like UPS can tell you where your package is.

Since PIN codes are ultimately guessable, I don't think I'd want the system to do more than confirm the processing of the ballot. However, to someone like me, it would go a lot further towards assuring me that, yes, my vote was counted than anything I've used in the past ... and certainly it would be far more reassuring than the buggy, mis-calibrated, poorly designed touch screen devices in use today.

Further, I like the idea of the vote taking shape over the course of a few days, or even a week. Vote by mail from a week before "election day". Maybe the ballots accumulate but are not counted until the end of the election itself ... again, something that the ballot status check outlined above would help to ensure.

The big weakness of this is privacy concern over linking your identity to a specific ballot. Who would be able to view your vote at a later date? The answer to that should be: nobody; that is not only a sacred tradition, but the fact that your private vote can never be retrieved and used against you in any way at a later date is important to keep voting free and uncoerced.

So imagine if your ballot arrived with four or five unique id stickers, each five or ten digits long: you chose, say, three of them, in whatever order you like and place them on the ballot. Nothing else on the ballot could link that specific ballot to you ... but that now partially-random ballot id can still be tracked to the central counting location. Only you would have a link between your identity and the ballot. I leave it to someone smarter than me to determine just how many "stickers" are needed to ensure reasonably anonymity (and prevent any potential ballot id conflicts) but I think the basic concept is valid.

Zeroing in on Tapestry 5.4

I've had just a bit of time this week to devote to furthering the work on Tapestry 5.4, and it feels like I'm near the turning point. You can follow the progress in the 5.4-js-rewrite branch

I've discussed my plans before; this is a bit of a progress update.

Basically, this release will be all about JavaScript:

• Removing the hard-dependence on Prototype
• Creating an abstraction layer, allowing jQuery to be used instead
• Introducing AMD (asynchronous module definition) modules (using RequireJS)
• Switching to a more declarative style of JavaScript; leveraging HTML5 data- attributes

Prototype vs. jQuery

Tapestry has historically used Prototype as what I call its foundation framework. The foundation is responsible for element selection and creation, attaching event handlers, triggering events, and encapsulating Ajax requests.

It is already possible to have both Prototype and jQuery on the same page; I do it regularly. There's a project for this as well, tapestry-jquery. That being said, there's some costs in having multiple foundation frameworks in place:

First of all is size; there's a lot of code in each framework (160Kb for Prototype, 260Kb for jQuery -- much less after minification and compression), and tremendous overlap. This includes the fact that both frameworks have a version of the Sizzle CSS selector engine.

There's also some occasional clashes; I had a nasty problem where some Bootstrap JavaScript was firing a "hide" event when a modal dialog was dismissed. After much tracing and hair pulling, I discovered that jQuery will treat a function attached to an element as an event handler for the event with the matching name. The "hide" event triggered by jQuery found the hide() method added by Prototype, and my modal dialog just winked out of existence, rather than animation the way I wanted.

Finally, its not just an either-or world; there's also YUI, ExtJS, MooTools ... over the last few years, every single time I mentioned my desire to introduce an abstraction layer, I've received the question "will it support X?" and "X" was some new framework, that person's favorite, that I had not previously heard of.

So the abstraction layer will be important for helping ensure a fast download of JavaScript, and to minimize the amount of network and JavaScript execution overhead.

If you are an application developer, there will be nothing to prevent you from using the native framework of your choice, directly. In many cases, this will be the best approach, and it will enable greater efficiency, access to a more complete API, and access (in jQuery's case) to a huge community of plugins.

If you are creating reusable components, it is best to try to use the SPI (the service provider interface; the abstraction layer). This allows your library to be widely used without locking your users into any particular foundation framework.

Modules vs. Libraries

Tapestry 5.3 and earlier have provided some great support for JavaScript in library form. The concept of a JavaScriptStack makes it possible to bundle any number of individual JavaScript libraries together, along with supporting CSS style sheet files. Even better, Tapestry can create a single virtual JavaScript library by concatenating all the stack libraries together. Tapestry does this at runtime, and conditionally ... so during development, you can work with many small JavaScript files and in production, they become one combined, minimized JavaScript file.

However, this JavaScript approach has its own problems. Page initialization logic is very dependent on the stacks (including the foundation frameworks) being present and loaded in a specific order. No page initialization logic can execute until all libraries have been loaded. If you have multiple stacks in production, then all the stacks must load before any logic executes.

All of this affects page load time, especially the time perceived by the end user. Too much is happening sequentially, and too much is happening over-all.

In Tapestry 5.4, the focus is shifting from libraries to modules. Under AMD, a module is a JavaScript function that expresses a dependency on any number of other modules, and exports a value to other modules. The exported value may be a single function, or an object containing multiple functions (or other values).

Here's a snapshot of one of modules, which provides support for the Zone component:

The define() function is used to define a module; the first parameter is an array of module names. The second parameter is a function invoked once all the dependencies have themselves been loaded; the parameters to that function are the exports of the named modules.

This function performs some initialization, attaching event handlers to the document object; it also defines and exports a single named function ("deferredZoneUpdate") that may be used by other modules.

Yes, I have a module named "_" which is Underscore.js. I'll probably add "$" for jQuery. Why not have short names?

The RequireJS library knows how to load individual modules and handle dependencies of those modules on others. Better yet, it knows how to do this in parallel. In fact, in 5.4, the only JavaScript loaded using a traditional <script> tag is RequireJS; all other libraries and modules are loaded through it.

At the end of the day, there will be less JavaScript loaded, and the JavaScript that is loaded will be loaded in parallel. I'm still working on some of the details about how module libraries may be aggregated into stacks.

Declarative JavaScript

Even in earlier forms of Tapestry, JavaScript support has been powerful ... but clumsy. To do anything custom and significant on the client-side you had to:

• Write a JavaScript library for your functionality
• Monkey-patch a function onto the T5.initializers namespace
• Package up your initialization data as a JSONObject ... typically, client element ids, URLs, etc.
• Have you Java component code invoke JavaScriptSupport.addInitializerCall(String, JSONObject) to link it all together.

Tapestry packages up all those addInitializerCall() initializations into one big block that executes at the bottom of the page (and does something similar for Ajax-oriented partial page updates).

Whew! To make this works requires that elements have unique ids (which can be a challenge when there are Ajax updates to the page). On top of that, the typical behavior is to create controller objects and attach event handlers directly to the elements; that works fine for small pages, but if there are hundreds (or thousands) of elements on the page, it turns into quite a burden on the JavaScript environment: lots of objects.

There isn't a specific name for this approach, beyond perhaps "crufty". Let's call it active initialization.

A more modern approach is more passive. In this style, the extra behavior is defined primarily in terms of events an element may trigger, and a top-level handler for that event. The events may be DOM related such as "click", "change", or "focus", or application-specific one triggered on the element directly. The top-level handler, often attached to the document, handles the event when it bubbles up from the element to the top level; it often distinguishes which elements it is interested using a CSS selector that includes references to a data- attribute.

For example, in the core/forms module, we need to track clicks on submit and image buttons as part of the form (this is related to supporting Ajax form submissions).

That little bit of code attaches a "click" event handler to the document, for any submit or image element anywhere on the page ... or ever added to the page later via Ajax. Older versions of Tapestry might have put an individual event handler on each such element, but in 5.4, the single event handler is sufficient.

Inside the event handler, we have access to the element itself, including data- attributes. That means that what may have been done using page initialization in Tapestry 5.3 may, in Tapestry 5.4, be a document-level event handler and data- attributes on the specific element; no JSONObject, no addInitializerCall(). Less page initialization work, smaller pages, fewer objects at runtime.

The flip side, however, is that the cost of triggering the extra events, and the cost of all the extra event bubbling, is hard to quantity. Still, the 5.3 approach introduces a lot of memory overhead at all times, whereas the 5.4 approach should at worst, introduce some marginal overhead when the user is actively interacting.

There's another advantage; by attaching your own event handlers to specific elements, you have a way to augment or replace behavior for specific cases. It's kind of a topsy-turvy version of inheritance, where the behavior of an element is, effectively, partially determined by the elements it is contained within. Kind of crazy ... and kind of just like CSS.

Compatibility

So you've written a fancy application in Tapestry 5.3 and you are thinking about upgrading to 5.4 when it is ready ... what should you be ready for?

On the server side, Tapestry 5.4 introduces a number of new services and APIs, but does not change very much that was already present.

A number of interfaces and services that were deprecated in the past have been removed entirely; even some dependencies, such as Javassist.

All built-in Tapestry components will operate through the SPI; they will work essentially the same regardless of whether you choose to operate using Prototype or jQuery or both.

The look-and-feel is changing from Tapestry's built-in CSS to Twitter Bootstrap. All of the old "t-" prefixed CSS classes are now meaningless.

On the other hand, what if you've built some complex client-side code? Well, if it is based on Prototype directly, that will be fine as well; just keep Prototype in the mix and start migrating your components to use the SPI.

The devil in the details is all about the Tapestry and T5 namespaces. The Tapestry object is the very oldest code; the T5 namespace was introduced in 5.2 in an attempt to organize things on the client-side a bit better.

It is not clear, at this time, just how much of those libraries will be left. Ideally, they will not be present at all unless an explicit compatibility mode is enabled (possibly, enabled by default). That being said, the conflict between the old active/crufty approach, and the new modular and passive/declarative approach is a difficult one to bridge.

In fact, the point of this post is partly to spur a discussion on what level of compatibility is required and realistic.

Conclusion

I've completely pumped about where Tapestry 5.4 already is, and where it is headed. Although the future of web applications is on the client, there is still significant play for hybrid applications: partly page oriented, partly Ajax oriented – and it's nice to have a single tool that integrates both approaches.

Beyond that, even for a Tapestry application that is a single "page", what Tapestry brings to the table is still very useful:

• live class reloading
• JavaScript stacks (with minification, Gzip compression, and client-side caching)
• the coming runtime CoffeeScript-to-JavaScript support
• best-of-breed exception reporting
• staggeringly high performance

These benefits are all as compelling in a single page application as they are in a traditional application, if not more so. I'm looking forward to building even more impressive apps in 5.4 than I could accomplish in 5.3, and I hope a lot of you will join me.

Not going to Clojure/Conj ... need a ticket?

Unfortunately, I will not be going to Clojure/Conj this year. And it is too late for a refund ... so I'm looking for someone to purchase my ticket at cost. The good news is the reason I'm not going ... I have a new project coming that will use Tapestry, Clojure, and perhaps Datomic and I need to be on-site with them the week of the conference.

So if you are looking to attend the conj, you can take over my ticket, for $475. Please contact me via email. Thanks!

Update: Sorry, the ticket is gone.

Stop sending template engines to the browser!

This article about client-side templating by Henrik Joreteg was quite interesting. In summary, it says don't send templates to the browser to be compiled and executed ... convert them to JavaScript functions that can be efficiently transfered to the browser and directly executed there.

Currently, Tapestry does virtually all of its markup rendering on the server ... but with the upcoming changes to Tapestry for 5.4, that may start to shift, with increasing amounts of the presentation being rendered in-browser.

I'm already mid-way through building a module structure for Tapestry and I'm quite pleased with what I have so far. Part of that infrastructure is the ability to compile CoffeeScript to JavaScript on the fly. The tapestry-core module will provide the basic infrastructure, and a new module will provide the actual runtime compiler.

I prefer for these kind of compilations and transformations to occur at runtime, rather than build time. It means no special build rules (in Maven or Gradle), no extra step when changing a file and reloading it in the browser, and it makes it easier in many ways for Tapestry to deliver the optimum bundle of compiled, aggregated, minified, and GZip-compressed JavaScript to the browser ... without having awkward conditionals and naming conventions to choose the correct version of each JavaScript file (not to mention ensuring that the minified version is derived from the correct non-minified version). Yes, there's a one-time startup cost, but that's always been a tradeoff I'm willing to make ... balanced by how much of Tapestry is lazy-loaded as needed.

However, the infrastructure is actually more flexible than that; it would be quite reasonable to do something like Henrick's solution: there's no reason why the infrastructure could not be used to read a template file and convert it to JavaScript source that can be exposed on the client as a module.

That raises another interesting idea; a template-as-a-module would then be a uniform way to package pure data (a JSON object bundle of properties) with the name of a module that renders that specific data into markup. That also opens up some terrific ways to have the server efficiently deliver updates to the client, in a data-focused way (rather than the markup-focused way provided by the Zone component). Combine that with server-push, and you have something really powerful brewing together!

CoffeeScript Cautions

Over the last couple of months, I've coded nearly all my client-side code in CoffeeScript. I prefer CoffeeScript to JavaScript ... but there are a few things to watch out for. I thought I'd share some practical experience, rather than the more typical hype for CoffeeScript or the reactionary vibe against it.

My biggest problems with CoffeeScript has been with indentation, which is no surprise, given the basic value proposition of the language. There are places where I've accidentally deleted a single space and completely changed the meaning of my code.

Here's an adapted example. We'll start with the correct CoffeeScript source code, and JavaScript translation:

Now, notice what happens when we delete a single space.

The change in indentation confused the CoffeeScript compiler; it ended the call to View.extend() and created a new expression for an object literal that is created and thrown away.

And remember, this isn't hypothetical; I really did this, and started seeing very odd errors: undefined properties and events not getting triggered. It took me a bit of time, with the debugger, and adding console logging, and eventually some viewing of the generated JavaScript, to realize what had happened.

Part of my solution to this class of issue, in Emacs, is to automatically increase the font size when editing CoffeeScript files. I may have to consider a switch from spaces to tabs as well, something that I've never considered in the past. I've also noticed that some IDEs, including IntelliJ, draw a vertical line to connect code at the same indentation layer. I don't have enough Emacs hacking experience to accomplish that, but I hope someone does.

Another problem I've hit is with function invocation; CoffeeScript style is to omit the parenthesis following the function name, and just immediately start listing function parameters:

element.setAttribute "width", "100%"

However, if your API allows chaining, you may be tempted to:

element.setAttribute "width", "100%".setAttribute "color", "blue"

Alas, the .setAttribute binds to the immediate value, the string:

element.setAttribute("width", "100%".setAttribute("color", "blue"));

One option is to enclose the first call with parenthesis:

(element.setAttribute "width", "100%").setAttribute "color", "blue"

I want to like that option, as it stays consistent with the normal format for function invocation; it just captures the result for chaining. However, each additional setAttribute() call will require an additional nesting layer of parenthesis. This leaves the following as the solution:

element.setAttribute("width", "100%").setAttribute("color", "blue")

And now we are stuck having to parse two different syntaxes for function invocation, depending on some very specific context.

Another issue is that, once you start using the standard function invocation style, which doesn't use parenthesis, you can easily get tripped up: Here the is null bound tightly to the "div" string literal; the result (true or false) was passed to the find() method, rather than the expected "div".

Because of these subtleties, I seem to find myself using the live CoffeeScript to JavaScript preview to check that my guess at how the JavaScript will be generated is correct. Preview is available on the CoffeeScript home page, and also as the CoffeeConsole plugin to Chrome.

What's interesting is that the above concerns aside, most of the time my first pass at the CoffeeScript does produce exactly what I'd expect, making me feel silly to waste the time to check it! Other times, I find that my first pass is actually too verbose, and can be simplified. Here is an example from some code I've written for Tapestry, a simple DSL for constructing DOM elements on the fly: This is invoking the builder() function, passing a string to define the element type and CSS class, then an object that adds attributes to the element, and lastly some additional values that define the body of the element: an array to define a nested element, and a string that will become literal text. This is converted to JavaScript:

Turns out, we can get rid of the curly braces on the objects without changing the output JavaScript; CoffeeScript is able to figure it all out by context:

Even getting rid of the commas is allowed. The commas are equivalent to the line break and indentation that's already present:

And that's where I start really liking CoffeeScript, because quite often, the code you write looks very close to a bespoke DSL.

Then there's the debugging issue; CoffeeScript scores poorly here. First of all, the code has been translated to JavaScript; line numbers at execution time will no longer line up with source CoffeeScript at all, so the first and most useful bit of information in a stack trace is no longer useful. It's very easy to glance back and forth to match the generated JavaScript back to source CoffeeScript, but it still takes some effort.

Of course, in a production application, you'll likely have minimized your JavaScript, which mangles your JavaScript and your line numbers far more than the CoffeeScript compilation does!

The other weakness is that all functions in CoffeeScript are anonymous; this takes another useful bit of information out of stack traces: the local function name. This leaves the only way to map from compiled JavaScript back to source CoffeeScript as involving a lot of processing between your ears, and that's not exactly ideal, given everything else you have to mentally juggle.

Lastly, I've heard of people who think CoffeeScript is only for programming using objects and classes. I find this quite odd; it's nice to have some optimized syntax for classes in CoffeeScript even if it comes at some price (the generated JavaScript can become quite verbose adding runtime support for the syntax that allows super class methods and constructors to be invoked); however, far beyond 95% of the code I write is purely functional, with what state that's present captured in closures of local variables, they way it should be.

I like Scott Davis' summary: CoffeeScript will make you a better JavaScript programmer. It is, largely, JavaScript with better syntax, plus a number of new features. You should never think of CoffeeScript as an alternative to understanding the occasionally ugly nuances of JavaScript; instead I think of it as a "have your cake and eat it too" situation. I certainly don't consider it noobscript, the way many in the Node.js community seem to.

At the end of the day, the translation from CoffeeScript to JavaScript is usually quite predictable, and turns into the JavaScript you probably should write, but often would not: for instance, JavaScript that always evaluates inside a hygienic function, or where every variable is defined at the top of its containing block.

My final take is that you are trading the very rough edges present in the JavaScript language for some sensible tradeoffs, and the occasional hidden "gotcha". I think that's a completely reasonable trade, and will continue to write as much as I can, client-side and server-side, in CoffeeScript.

Tapestry 5 Book by Igor Drobiasko

Is the one thing holding you back from embracing Tapestry the lack of a current book? Well, that excuse is finally going by the wayside; Igor Drobiazko, a Tapestry contributor (second in activity only to me) is getting ready to finish and self-publish his book on Tapestry 5.3: Tapestry 5: Rapid Web Application Development in Java

Want early access to the book? Join his funding campaign at indiegogo.

Igor hit his goal on day 1, but for as little as $25 you can get access to at least 400 pages of content as soon as the campaign ends, and the complete book when it's ready. You can also help assure that the book will be updated for the great improvements already cooking up in Tapestry 5.4.

More async: using auto() for parallel operations

One of the straw men that people often cite when discussing event-driven programming, ala Node.js, is the fear that complex server-side behavior will take the form of unmanageably complex, deeply nested callbacks.

Although the majority of the code I've so far written with Node is very simple, usually involving only a single callback, my mental image of Node is of a request arriving, and kicking off a series of database operations and other asynchronous requests that all combine, in some murky fashion, into a single response. I visualize such a request as a pinball dropping into some bumpers and bouncing around knocking down targets, until shooting out, back down to the flippers.

Here's an example workflow from the sample application I'm building; I'm managing a set of images used in a slide show; so I have a SlideImage entity in MongoDB (using Mongoose), and each SlideImage references a file stored in Mongo's GridFS.

When it comes time to delete a SlideImage, it is necessary to delete the GridFS file as well. The pseudo-code for such an operation, in a non-event based system, might look something like:

Inside Node, where all code is event-driven and callback oriented, we should be able to improve on the pseudo-code by doing the deletes of the SlideImage document, and the GridFS file, in parallel. Well, that's the theory anyway, but I'd normally stick to a waterfall approach, as tracking when all operations have completed would be tedious and error prone, especially in light of correctly handling any errors that might occur.

Enter async's auto() function. With auto(), you define a set of tasks that have dependencies on each other. Each task is a function that receives a callback, and a results object. auto() figures out when each task is ready to be invoked.

When a task fails, it passes the error to the callback function. When a task succeeds, it passes null and a result value to the callback function. Later executing tasks can see the results of prior tasks in the result object, keyed on the prior tasks's name.

As with waterfall(), a single callback is passed the error object, or the final result object.

Let's see how it all fits together, in five steps:

• Find the SlideImage document
• Open the GridFS file
• Delete the GridFS file
• Delete the SlideImage document
• Send a response (success or failure) to the client

The granularity here is partly driven by the specific APIs and their callbacks.

The code for this is surprisingly tight:

auto() is passed two values; an object that maps keys to arrays, and the final callback. Each array consists of the names of dependencies for the task, followed by the task function (you can just specify the function if a task has no dependencies, but I prefer the consistency of each entry being an array).

So find has no dependencies, and kicks of the overall process. I think it is really significant how consistent Node.js APIs are: the basic callback consisting of an error and a result makes it very easy to integrate code from many different libraries and authors (I think there's a kind of Monad hidden in there). In the code, the callback created by auto(), and passed to find, is perfectly OK to pass into findById. It's all low impedance: no need to write any kind of shim or adapter.

The later tasks take the additional results parameter; results.find is the SlideImage document provided by the find task.

The remove and openFile tasks both depend on find: they will run in no particular order after find; more importantly, their callbacks will be invoked in no predictable order, based on when the various asynchronous operations complete.

Only once all tasks have executed (or one task has passed an error to its callback), does the final callback get invoked; this is what sends a 500 error to the client, or a 200 success (with a { "result" : "ok" } JSON response.

I think this code is both readable, and concise; in fact, I can't imagine it being much more concise. My brain is starting to really go parallel: part of my brain is evaluating everything in terms of Java code and idioms while the rest is embracing JavaScript and CoffeeScript and Node.js idioms; the Java part is impressed by the degree to which these JavaScript solutions eschew complex APIs in favor of working on a specific "shape" of data; if I was writing something like this in Java, I'd be up to my ears in fluid interfaces and hidden implementations, with a ton of code to write and test.

I'm not sure that the application I'm writing will have any processing workflows significantly more complex than this bit, but if it does, I'm quite relieved to have auto() in my toolbox.

A little Gotcha with asynch and Streams

I stumbled across a little gotcha using async with Node.js Streams: you can easily corrupt your output if you are not careful.

Node.js Streams are an abstraction of Unix pipes; they let you push or pull data a little bit at a time, never keeping more in memory than its needed. async is a library used to organize all the asynchronous callbacks used in node applications without getting the kind of "Christmas Tree" deep nesting of callbacks that can occur too easily.

I'm working on a little bit of code to pull an image file, stored in MongoDB GridFS, scale the image using ImageMagick, then stream the result down to the browser.

My first pass at this didn't use ImageMagick or streams, and worked perfectly ... but as soon as I added in the use of async (even before adding in ImageMagick), I started getting broken images in the browser, meaning that my streams were getting corrupted.

Before adding async, my code was reasonable:

However, I knew I was going to add a few new steps here to pipe the file content through ImageMagick; that's when I decided to check out the async module.

The logic for handling this request is a waterfall; each step kicks off some work, then passes data to the next step via an asynchronous callback. The async library calls the steps "tasks"; you pass an array of these tasks to async.waterfall(), along with the end-of-waterfall callback. This special callback may be passed an error provided by any task, or the final result from the final task.

With waterfall(), each task is passed a special callback function. If the callback function is passed a non-null error as the first parameter, then remaining tasks are skipped, and the final result handler is invoked immediately, to handle the error.

Otherwise, you pass null as the first parameter, plus any additional result values. The next task is passed the result values, plus the next callback. It's all very clever.

My first pass was to duplicate the behavior of my original code, but to do so under the async model. That means lots of smaller functions; I also introduced an extra step between getting the opened file and streaming its contents to the browser. The extra step is intended for later, where ImageMagick will get threaded in.

The code, despite the extra step, was quite readable:

My style is to create local variables with each function; so openFile kicks off the process; once the file has been retrieved from MongoDB, the readFileContents task will be invoked ... unless there's an error, in which case errorCallback gets invoked immediately.

Inside readFileContents we convert the file to a stream with file.stream(true) (the true means to automatically close the stream once all of the file contents have been read from GridFS).

streamToClient comes next, it takes that stream and pipes it down to the browser via the res (response) object.

So, although its now broken up into more small functions, the logic is the same, as expressed on the very last line: open the file, read its contents as a stream, stream the data down to the client.

However, when I started testing this before moving on to add the image scaling step, it no longer worked. The image data was corrupted. I did quite a bit of thrashing: adding log messages, looking at library source, guessing, and experimenting (and I did pine for a real debugger!).

Eventually, I realized it came down to this bit of code from the async module:

The code on line 7 is the callback function passed to each task; notice that once it decides what to do, on line 21 it defers the execution until the "next tick".

The root of the problem was simply that the "next tick" was a little too late. By the time the next tick came along, and streamToClient got invoked, the first chunk of data had already been read from MongoDB ... but since the call to pipe() had not executed yet, it was simply discarded. The end result was that the stream to the client was missing a chunk at the beginning, or even entirely empty.

The solution was to break things up a bit differently, so that the call to file.stream() happens inside the same task as the call to stream.pipe().

So that's our Leaky Abstraction for today; what looked like an immediate callback was deferred just enough to change the overall behavior. And that, in Node, anything that can be deferred, will be deferred, since that makes the overall application that much zippier.

You Cannot Correctly Represent Change Without Immutability

The title of this blog post is a quote by Rich Hickey, talking about the Datomic database. Its a beautiful statement, at once illuminating and paradoxical. It drives at the heart of the design of both Clojure and Datomic, and embraces the difference between identity and state.

What is change? That seems like an obvious question, but my first attempt at defining it was "some change to a quantifiable set of qualities about some object." Woops, I used change recursively there ... that's not going to help.

In the real world, things change in ways we can observe; the leaf falls from the tree, the water in the pot boils, the minute hand moves ever forward.

How do we recognize that things have changed? We can, in our memories, remember a prior state. We remember when the leaf was green and attached to a branch; we remember when the water came out of the tap, and we remember looking at the clock a few minutes ago. We can hold both states in our mind at the same time, and compare them.

How do we represent change in traditional, object-oriented technologies? Well, we have fields (or columnus) and we change the state in place:

• leaf.setColor(BROWN).detachFromTree()
• UPDATE LEAVES SET COLOR = 'BROWN' WHERE ID = ?ID
• water.setTemperature(212)
• or we see time advancing via System.currentTimeMillis()

Here's the challenge: given an object, how do you ask it about its prior state? Can you ask leaf.getTreeDetachedFrom()? Generally, you can't unless you've gone to some herculean effort: the new state overwrites the old state in place.

When Rich talks about conflating state with identity, this is what he means. With the identity and state conflated, then after the change in state, the leaf will now-have-always-been fallen from the tree, the water will now-have-always-been boiled, and the clock will now-eternally be at 9:49 AM.

What Clojure does in memory, and Datomic does in the database, is split identity and state. We end up with leaf1 as {:id "a317a439-50bb-4d37-838a-c8eef289e22f" :color :green :attached-to maple-tree} and leaf2 as {:id "a317a439-50bb-4d37-838a-c8eef289e22f" :color :brown :on-ground true}. The id is the same, but the other attributes can vary.

With immutability, changes in state are really new objects; a new version, or "quantifiable set of qualities", that does not affect the original version. It is possible to compare two different iterations of the same object to see the "deltas". In Datomic, you even have more meta-data about when such state changes occur, what else changed within the same transaction, and who is the responsible party for that transaction.

The essence here is not to think of an object as a set of slots you can put new data into. Instead, think of it as a time-line of different configurations of the object. The fact that late in the time-line, the leaf has fallen from the tree does not affect the fact that earlier on the time-line, the leaf was a bud on a branch. The identity of the leaf transcends all those different states.

In the past, I've built systems that required some of the features that Datomic provides; for example, being able to reconstruct the state of the entire database at some prior time, and strong auditing of what changes occurred to what entities at a specific time (or transaction). Rich knows that others have hit this class of problem; part of his selling point is to ask "and who really understands that query" (the one that reconstructs prior state). He knows people have done it, but he also knows no one is very happy about its performance, correctness, or maintainability ... precisely because traditional databases don't understand mutability: they live in that eternal-now, and drag your application into the same world view.

That's why I'm excited by Datomic; it embraces this key idea: separate identity from state by leveraging immutability and from the ensuing design, much goodness is an automatic by-product. Suddenly, we start seeing much of what we take as dogma when developing database-driven applications to be kludges on top of an unstable central idea: mutable state.

For example: read transactions are a way to gain stable view of interrelated data even as the data is being changed (in place); with Datomic, you always have a stable view of all data, because you operate on an immutable view of the entire database at some instance in time. Other transactions may add, change, or replace Datoms in the database, but any code that is reading from the database will be completely unaware of those changes, even as they lazily navigate around the entire database.

Gradle CoffeeScript Compilation

As I'm working on rebooting JavaScript support in Tapestry one of my goals is to be able to author Tapestry's JavaScript as CoffeeScript. Tapestry will ultimately have a runtime option to dynamically compile CoffeeScript to JavaScript, but I don't want that as a dependency of the core library; that means I need to be able to have that compilation (transpilation?) occur at build time.

Fortunately, this is the kind of thing Gradle does well! My starting point for this is the Web Resource Optimizer for Java project, which includes a lot of code, often leveraging Rhino (for JavaScript) or JRuby, to do a number of common web resource processing, including CoffeeScript to JavaScript. WRO4J has an Ant task, but I wanted to build something more idiomatic to Gradle.

Here's what I've come up with so far. This is a external build script, separate from the project's main build script:

The task finds all .coffee files in the input directory (ignoring everything else) and generates a corresponding .js file in the output directory.

The @InputDirectory and @OutputDirectory annotations allows Gradle to decide when the task's action is needed: If any file changed in the directories provided by these methods then the Task must be rerun. Gradle doesn't tell us exactly what changed, or create the directories, or anything ... that's up to us.

Since I only expect to have a handful of CoffeeScript files, the easiest thing to do was to simply delete the output directory and recompile all CoffeeScript input files on any change. The @TaskAction annotation directs Gradle to invoke the doCompile() method when inputs (or outputs) have changed since the previous build.

It's enlightening to note that the visit passed to the closure on line 34 is, in fact, a FileVisitDetails, which makes it really easy to, among other things, work out the output file based on the relative path from the source directory to the input file.

One of my stumbling points was setting up the classpath to pull in WRO4J and its dependencies; the buildscript configuration on line 4 is specific to this single build script, which is very obvious once you've worked it out. This is actually excellent for re-use, as it means that minimal changes are needed to the build.gradle that makes use of this build script. Earlier I had, incorrectly, assumed that the main build script had to set up the classpath for any external build scripts.

Also note line 54; the CompileCoffeeScript task must be exported from this build script to the project, so that the project can actually make use of it.

The changes to the project's build.gradle build script are satisfyingly small:

The apply from: brings in the CompileCoffeeScript task. We then use that class to define a new task, using the defaults for srcDir and outputDir.

The last part is really interesting: We are taking the existing processResources task and adding a new input directory to it ... but rather than explicitly talk about the directory, we simply supply the task. Gradle will now know to make the compileCoffeeScript task a dependency of the processResources task, and add the task's output directory (remember that @OutputDirectory annotation?) as another source directory for processResources. This means that Gradle will seamlessly rebuild JavaScript files from CoffeeScript files, and those JavaScript files will then be included in the final WAR or JAR (or on the classpath when running test tasks).

Notice the things I didn't have to do: I didn't have to create a plugin, or write any XML, or seed my Gradle extension into a repository, or even come up with a fully qualified name for the extension. I just wrote a file, and referenced it from my main build script. Gradle took care of the rest.

There's room for some improvement here; I suspect I could do a little hacking to change the way compilation errors are reported (right now it is just a big ugly stack trace). I could use a thread pool to do compilation in parallel. I could even back away from the delete-the-output-directory-and-recompile-all approach ... but for the moment, what I have works and is fast enough.

Luke Daly leaked that there will be an experimental CoffeeScript compilation plugin coming in 1.1 so I probably won't waste more cycles on this. For only a couple of hours of research and experimentation I was able to learn a lot, and get something really useful put together, and the lessons I've learned mean that the next one of these I do will be even easier!

Getting CrashPlan to work on Mac after JDK 1.7 upgrade

After upgrading my Mac to JDK 1.7, I noticed that CrashPlan stopped working. Given just how much trouble I got in a ways back when I lost a month's worth of pictures and videos of my son I'm now a belt and suspenders kind of backup guy ... the belt may be TimeMachine, but the suspenders are CrashPlan.

After working it out with their technical staff, it turns out CrashPlan is not compatible with JDK 1.7 on Mac which is rather unfortunate. And, having installed JDK 1.7, it was trying to use that version!

Getting it working is pretty easy however; most of the information is available in their recipe for starting and stopping the CrashPlan engine. The key part is that the com.crashplan.engine.plist file includes an entry where you can specify the path to the Java executable; with a bit of experimentation to find the right path, it appears to be working.

The only change I made to the file was to change the path; the first entry for the ProgramArguments key:

You will, of course, need to sudo to edit the file and to stop and relaunch the CrashPlan engine. Now, having set this up (and, coincidentally, renewed by annual subscription) my computer is happily chugging away, making the backups.

Latency numbers every programmer should know

This is interesting stuff; Jonas Bonér organized some general some latency data by Peter Norvig as a Gist, and others expanded on it. What's interesting is how, scaling time up by a billion, converts a CPU instruction cycle into approximately one heartbeat, and yields a disk seek time of "a semester in university".

This is easier read on the original Gist page. Sorry about my blog's narrow formatting.

Another view of this data is as an animated presentation.

Synchronized Considered Harmful

New flash: concurrency is hard. Any time you have mutable data and multiple threads, you are just asking for abuse, and synchronized is simply not going to be your savior ... but the concurrency facilities of JDK 1.5 just might be.

I was recently contacted by a client who was load testing their Tapestry 5.3.3 application; they were using Tomcat 6.0.32 with 500 worker threads, on a pretty beefy machine: Intel Xeon X7460 @ 2.66Ghz, OpenJDK 64-Bit Server VM (14.0-b16, mixed mode). That's a machine with six cores, and 16 MB of L2 cache.

For all that power, they were tapping out at 450 requests per second. That's not very good when you have 500 worker threads ... it means that you've purchased memory and processing power just to see all those worker threads block, and you get to see your CPU utilization stay low. When synchronization is done properly, increasing the load on the server should push CPU utilization to 100%, and response time should be close to linear with load (that is to say, all the threads should be equally sharing the available processing resources) until the hard limit is reached.

Fortunately, these people approached me not with a vague performance complaint, but with a detailed listing of thread contention hotspots.

The goal with Tapestry has always been to build the code right initially, and optimize the code later if needed. I've gone through several cycles of this over the past couple of years, optimizing page construction time, or memory usage, or throughput performance (as here). In general, I follow Brian Goetz's advice: write simple, clean, code and let the compiler and Hotspot figure out the rest.

Another piece of advice from Brian is that "uncontested synchronized calls are very cheap". Many of the hotspots located by my client were, in fact, simple synchronized methods that did some lazy initialization. Here's an example:

In this example, getting the messages can be relatively time consuming and expensive, and is often not necessary at all. That is, in most instances of the class, the getMessages() method is never invoked. There were a bunch of similar examples of optional things that are often not needed ... but can be heavily used in the cases where they are used.

It turns out that "uncontested" really means uncontested: You better be sure that no two threads are ever hitting synchronized methods of the same instance at the same time. I chatted with Brian at the Hacker Bed & Breakfast about this, and he explained that you can quickly go from "extremely cheap" to "asymptotically expensive" when there's any potential for contention. The synchronized keyword is very limited in one area: when exiting a synchronized block, all threads that are waiting for that lock must be unblocked, but only one of those threads gets to take the lock; all the others see that the lock is taken and go back to the blocked state. That's not just a lot of wasted processing cycles: often the context switch to unblock a thread also involves paging memory off the disk, and that's very, very, expensive.

Enter ReentrantReadWriteLock: this is an alternative that allows any number of readers to share a lock, but only a single writer. When a thread attempts to acquire the write lock, the thread blocks until all reader threads have released the read lock. The cost of managing the ReentrantReadWriteLock's state is somewhat higher than synchronized, but has the huge advantage of letting multiple reader threads operate simultaneously. That means much, much higher throughput.

In practice, this means you must acquire the shared read lock to look at a field, and acquire the write lock in order to change the field.

ReentrantReadWriteLock is smart about only waking the right thread or threads when either the read lock or the write lock is released. You don't see the same thrash you would with synchronized: if a thread is waiting for the write lock, and another thread releases it, ReentrantReadWriteLock will (likely) just unblock the one waiting thread.

Using synchronized is easy; with an explicit ReentrantReadWriteLock there's a lot more code to manage:

I like to avoid nested try ... finally blocks, so I broke it out into seperate methods.

Notice the "lock dance": it is not possible to acquire the write lock if any thread, even the current thread, has the read lock. This opens up a tiny window where some other thread might pop in, grab the write lock and initialize the messages field. That's why it is desirable to double check, once the write lock has been acquired, that the work has not already been done.

Also notice that things aren't quite symmetrical: with ReentrantReadWriteLock it is allowable for the current thread to acquire the read lock before releasing the write lock. This helps to minimize context switches when the write lock is released, though it isn't expressly necessary.

Is the conversion effort worth it? Well, so far, simply by converting synchronized to ReentrantReadWriteLock, adding a couple of additional caches (also using ReentrantReadWriteLock), plus some work optimizing garbage collection by expanding the eden space, we've seen some significant improvements; from 450 req/sec to 2000 req/sec ... and there's still a few minor hotspots to address. I think that's been worth a few hours of work!

Things I Learned at Hacker Bed & Breakfast

• You can spend any amount of money on Scotch, and it keeps getting better
• A Java instance field that is assigned exactly once via lazy initialization does not have to be synchronized or volatile (as long as you can accept race conditions across threads to assign to the field); this is from Rich Hickey
• Scotch + Random Internet Images + Speakers == Fun; giving off the cuff talks on silly subjects, with random internet images as your "slides"
• No one's that interested in Java anymore (at least, not at Hacker B&B)
• Only ride a zip line backwards if you are prepared to accept the consequences
• The linkage between honey and fried chicken has not yet penetrated into Research Triangle, despite my best efforts
• The cost of using the synchronized keyword can go asymptotic with enough threads and contention, making ReentrantReadWriteLock much cheaper in comparison
• Stu Halloway's house appears to have been designed by the same team that designed the Tardis, and is also bigger on the inside than the outside
• It is much more efficient to just hand Brian Goetz the $50 buy in, rather than go through the motions of playing poker with him

Yet More Spock Magic: Mocks

Spock's built-in mock object capabilities are just a dream to use ... unlike other systems I've used, it doesn't get in your way, or force you to think backwards or inside out. Once again, some listings. These are for tests of Tapestry IoC's AspectDecorator service, which is used to create a wrapper interceptor around some other object. The test below shows how a supplied MethodAdvice callback object is invoked by the interceptor, if the advice is associated with the invoked method.

TestNG with EasyMock (Java)

Even this example is a bit streamlined, as some of the mock object capabilities, such as methods newMock(), replay() and verify() are being derived from the TestBase base class.

Spock

Spock's wonderful when: / then: blocks organize the behavior into a stimulus followed by a response; using EasyMock, you have to train the mock objects for the response before introducing the stimulus (the method invocation). Further, with EasyMock there's one API for methods that return a fixed response, a second API for methods that throw an exception, and a third API for methods where the result must be calculated; in Spock the value after the >> operator is either a literal value, or a closure that can do what it likes, such as the one attached to MethodAdvice.advice() that checks for the expected method name, and then proceed()s to the delegate mock object's method.

I think that a reasonable developer, even without a thorough understanding of Spock, would get the gist of what this test does (perhaps with a little side note about the interaction system inside the then: block). On the other hand, I've seen when training people with TestNG and EasyMock that it very rarely sinks in immediately, if at all.

Yet Another Bit of Spock Love

I'm gradually converting a back-log of existing tests to Spock ... and some of them convert so beautifully, it hurts. Here's an example:

Before (Java and TestNG)

After (Spock)

What a difference; the data-driven power of the where: block makes this stuff a bit of a snap, and you can see in once place, at a glance, what's going on. IDEA even lines up all the pipe characters automatically (wow!). It's obvious how the tests execute, and easy to see how to add new tests for new cases. By comparison, the TestNG version looks like a blob of code ... it takes a bit more scrutiny to see exactly what it is testing and how.

In addition, the propertyMissing() trick means that any property (including public static fields) of Calendar is accessible without qualification, making things look even nicer. This is what they mean by writing an "executable specification", rather than just writing code.

I can't say this enough: using any other framework for testing Java or Groovy code would simply not be logical.

Node and Callbacks

One of the fears people have with Node is the callback model. Node operates as a single thread: you must never do any work, especially any I/O, that blocks, because with only a single thread of execution, any block will block the entire process.

Instead, everything is organized around callbacks: you ask an API to do some work, and it invokes a callback function you provide when the work completes, at some later time. There are some significant tradeoffs here ... on the one hand, the traditional Java Servlet API approach involves multiple threads and mutable state in those threads, and often those threads are in a blocked state while I/O (typically, communicating with a database) is in progress. However, multiple threads and mutable data means locks, deadlocks, and all the other unwanted complexity that comes with it.

By contrast, Node is a single thread, and as long as you play by the rules, all the complexity of dealing with mutable data goes away. You don't, for example, save data to your database, wait for it to complete, then return a status message over the wire: you save data to your database, passing a callback. Some time later, when the data is actually saved, your callback is invoked, and which point you can return your status message. It's certainly a trade-off: some of the local code is more complicated and bit harder to grasp, but the overall architecture can be lightening fast, stable, and scalable ... as long as everyone plays by the rules.

Still the callback approach makes people nervous, because deeply nested callbacks can be hard to follow. I've seen this when teaching Ajax as part of my Tapestry Workshop.

I'm just getting started with Node, but I'm building an application that is very client-centered; the Node server mostly exposes a stateless, restful API. In that model, the Node server doesn't do anything too complicated that requires nested callbacks, and that's nice. You basically figure out a single operation based on the URL and query parameters, execute some logic, and have the callback send a response.

There's still a few places where you might need an extra level of callbacks. For example, I have a (temporary) API for creating a bunch of test data, at the URL /api/create-test-data. I want to create 100 new Quiz objects in the database, then once they are all created, return a list of all the Quiz objects in the database. Here's the code:

It should be pretty easy to pick out the logic for creating test data at the end. This is normal Node JavaScript but if it looks a little odd, it's because it's actually decompiled CoffeeScript. For me, the first rule of coding Node is always code in CoffeeScript! In its original form, the nesting of the callbacks is a bit more palatable:

What you have there is a count, remaining, and a single callback that is invoked for each Quiz object that is saved. When that count hits zero (we only expect each callback to be invoked once), it is safe to query the database and, in the callback from that query, send a final response. Notice the slightly odd structure, where we tend to define the final step (doing the final query and sending the response) first, then layer on top of that the code that does the work of adding Quiz objects, with the callback that figures out when all the objects have been created.

The CoffeeScript makes this a bit easier to follow, but between the ordering of the code, and the three levels of callbacks, it is far from perfect, so I thought I'd come up with a simple solution for managing things more sensibly. Note that I'm 100% certain that this issue has been tackled by any number of developers previously ... I'm using the excuse of getting comfortable with Node and CoffeeScript as an excuse to embrace some Not Invented Here syndrome. Here's my first pass:

The Flow object is a kind of factory for callback wrappers; you pass it a callback and it returns a new callback that you can pass to the other APIs. Once all callbacks that have been added have been invoked, the join callbacks are invoked after each of the other callbacks have been invoked. In other words, the callbacks are invoked in parallel (well, at least, in no particular order), and the join callback is invoked only after all the other callbacks have been invoked.

In practice, this simplifies the code quite a bit:

So instead of quiz.save (err) -> ... it becomes quiz.save flow.add (err) -> ..., or in straight JavaScript: quiz.save(flow.add(function(err) { ... })).

So things are fun; I'm actually enjoying Node and CoffeeScript at least as much as I enjoy Clojure; which is nice because it's been years (if ever) since I've enjoyed the actual coding in Java (though I've liked the results of my coding, of course).

Plastic: Advanced Example

Plastic is Tapestry's built-in Aspect Oriented Programming library, which primarily operates at the byte code level, but shields you from most byte code level thinking: normally, your code is implemented in terms of having method invocations or field reads and writes passed to callback objects that act as delegates or filters.

Sometimes, though, you need to get a little more low-level and generate the implementation of a method more directly. Plastic includes a fluent interface for this as well: InstructionBuilder.

This is an example from Tapestry's Inversion Of Control (IoC) container code; the proxy instance is the what's exposed to other services, and encapsulates two particular concerns: First, the late instantiation of the actual service implementation, and second, the ability to serialize the proxy object (even though the services and other objects are decidedly not serializable).

In terms of serialization, what actually gets serialized is a ServiceProxyToken object; when a ServiceProxyToken is later de-serialized, it can refer back to the equivalent proxy object in the new JVM and IoC Service Registry. The trick is to use the magic writeReplace() method so that when the proxy is serialized, the token is written instead. Here's the code:

To kick things off, we use the PlasticProxyFactory service to create a proxy that implements the service's interface.

The callback passed to createProxy() is passed the PlasticClass object. This is initially an implementation of the service interface where each interface method does nothing.

The basic setup includes making the proxy implement Serializable and creating and injecting values into new fields for the other data that's needed.

Next, a method called delegate() is created; it is responsible for lazily creating the real service when first needed. This is actually encapsulated inside an instance of ObjectCreator; the delegate() method simply invokes the create() method and casts the result to the service interface.

The methods on InstructionBuilder have a very close correspondence to JVM byte codes. So, for example, loading an instance field involves ensuring that the object containing the field is on the stack (via loadThis()), then consuming the this value and replacing it with the instance field value on the stack, which requires knowing the class name, field name, and field type of the field to be loaded. Fortunately, the PlasticField knows all this information, which streamlines the code.

Once the ObjectCreator is on the stack, a method on it can be invoked; at the byte code level, this requires the class name for the class containing the method, the return type of the method, and the name of the method (and, for methods with parameters, the parameter types). The result of that is the service implementation instance, which is cast to the service interface type and returned.

Now that the delegate() method is in place, it's time to make each method invocation on the proxy invoke delegate() and then re-invoke the method on the late-instantiated service implementation. Because this kind of delegation is so common, its supported by the delegateTo() method.

introduceMethod() can access an existing method or create a new one; for the writeReplace() method, the introduceMethod call creates a new, empty method. The call to changeImplementation() is used to replace the default empty method implementation with a new once; again, loading an injected field value, but then simply returning it.

Finally, because I feel strongly about including a useful toString() method in virtually all objects, this is also made easy in Plastic.

Once the class has been defined, it's just a matter of invoking the newInstance() method on the ClassInstantiator object to instantiate a new instance of the proxy class. Behind the scenes, Plastic has created a constructor to set the injected field values, but another of the nice parts of the Plastic API is that you don't have to manage that: ClassInstantiator does the work.

I'm pretty proud of the Plastic APIs in general; I think they strike a good balance between making common operations simple and concise, but still providing you with an escape-valve to more powerful (or more efficient) mechanisms, such as the InstructionBuilder examples above. Of course, the deeply-nested callback approach can be initially daunting, but that's mostly a matter of syntax, which may be addressed in JDK 8 with the addition of proper closures to the Java language.

I strongly feel that Plastic is a general purpose tool, that goes beyond inversion of control and the other manipulations that are specific to Tapestry ... and Plastic was designed specifically to be reused outside of Tapestry. It seems like it could be used for anything from implementing simple languages and DSLs, to providing all kinds of middleware code in new domains ... I have a fuzzy idea involving JMS and JSON with a lot of wiring and dispatch that could be handled using Plastic. I'd love to hear other people's ideas!

Tests: Auto Launch Application, or Expect Running?

Does your test suite launch your application or expect it to be running already? This question came up while working on a client project; I launched the Selenium-based test suite and everything failed ... no requests got processed and I spent some time tracking down why the test suite was failing to launch the application.

I chatted about this with my client ... and found out that the Selenium test suite expects the application to already be running. It starts up Selenium Server, sure, but the application-under-test should already be running.

And suddenly, I realized I had a Big Blind Spot, dating back to when I first started using Selenium. For years, my testing pattern was that the test suite would start up the application, run the Selenium tests, then shut it down ... and this makes perfect sense for testing the Tapestry framework, where the test suite starts and stops a number of different mini-applications (and has to run headless on a continuous integration server).

But an application is different, and there's a lot of advantages to testing against the running application ... including being able to quickly and easily reproduce any failures in the running application. Also, without the time to start and stop the application, the test suite runs a bit faster.

Certainly, the command line build needs to be able to start and stop the application for headless and hands-off testing. However, the test suite as run from inside your IDE ... well, maybe not.

So ... how do you handle this on your projects?

LinkedIn Etiquette

I've used LinkedIn for many years now, long before I joined Facebook ... I liked the concept of never losing contact information with business contacts and technologist. It just seemed like a good idea (though I do sometimes wonder if LinkedIn has any particular purpose).

I tend to only connect with people I've met in person, or at least talked to on the phone.

One thing that drives me crazy about LinkedIn is that when requesting a connection, you aren't forced to customize the greeting message from the default "I'd like to add you to my professional network."

As far as I'm concerned, this default greeting is like no greeting at all, and it's a sign that you are just trolling for contacts. Just like you should always write a cover letter for a resume, you should always customize this greeting message.

Your chances of successfully making a connection go up significantly if you do customize the message ... and with me, if you don't, your chances drop to near zero.

Tapestry Advantages

A summary of a discussion about the advantages of Tapestry over Struts:

• Exceptional exception reporting
• Significantly less code
• Live class reloading
• Sensible defaults, especially for SEO-friendly URLs
• Great community
• Flexibility and customizability

Interestingly, the quality of Tapestry's documentation was mentioned ... favorably! Between the revised home page, and Tapestry JumpStart (and Igor's coming book), I think we're headed in the right direction in terms of documentation going from a liability to an asset.

Tapestry 5.4: Focus on JavaScript

Tapestry 5.3.1 is out in the wild ... and if Tapestry is to stay relevant, Tapestry 5.4 is going to need to be something quite (r)evolutionary.

There was some confusion on the Tapestry developer mailing list in advance of this blog post; I'd alluded that it was coming, and some objected to such pronouncements coming out fully formed, without discussion. In reality, this is just a distillation of ideas, a starting point, and not a complete, finalized solution. If it's more detailed than some discussions of Tapestry's evolution in the past, that just means that the mailing list discussion and eventual implementation will be that much better informed.

In posts and other conversations, I've alluded to my vision for Tapestry 5.4. As always, the point of Tapestry is to allow developers to code less, deliver more, and that has been the focus of Tapestry on the server side: everything drives that point: terseness of code and templates, live class reloading, and excellent feedback are critical factors there. Much of what went into Tapestry 5.3 strengthened those points ... enhancements to Tapestry's meta-programming capabilities, improvements to the IoC container, and reducing Tapestry's memory footprint in a number of ways. I have one client reporting a 30% reduction in memory utilization, and another reporting a 30 - 40% improvement in execution speed.

Interestingly, I think that for Tapestry to truly stay relevant, it needs to shift much, much, more of the emphasis to the client side. For some time, Tapestry has been walking a fine line with regards to the critical question of where does the application execute? Pre-Ajax, that was an easy question: the application runs on the server, with at most minor JavaScript tricks and validations on the client. As the use of Ajax has matured, and customer expectations for application behavior in the browser have expanded, it is no longer acceptable to say that Tapestry is page based, with limited Ajax enhancements. Increasingly, application flow and business logic need to execute in the browser, and the server-side's role is to orchestrate and facilitate the client-side application, as well as to act as a source and sink of data ultimately stored in a database.

As Tapestry's server-side has matured, the client side has not kept sufficient pace. Tapestry does include some excellent features, such as how it allows the server-side to drive client-side JavaScript in a modular and efficient way. However, that is increasingly insufficient ... and the tension caused by give-and-take between client-side and server-side logic has grown with each release.

Nowhere is this more evident than in how Tapestry addresses HTML forms. This has always been a tricky issue in Tapestry, because the dynamic rendering that can occur needs to be matched by dynamic form submission processing. In Tapestry, the approach is to serialize into the form instructions that will be used when the form is submitted (see the store() method of the FormSupport API). These instructions are used during the processing of the form submission request to re-configure the necessary components, and direct them to read their query parameters, perform validations, and push updated values back into server-side objects properties. If you've ever wondered what the t:formdata hidden input field inside every Tapestry forms is about ... well, now you know: it's a serialized stream of Java objects, GZipped and MIME encoded.

However, relative to many other things in Tapestry, this is a bit clumsy and limited. You start to notice this when you see the tepid response to questions on the mailing list such as "how to do cross-field validation?" Doing more complicated things, such as highly dynamic form layouts, or forms with even marginal relationships between fields, can be problematic (though still generally possible) ... but it requires a bit too much internal knowledge of Tapestry, and the in-browser results feel a bit kludgy, a bit clumsy. Tapestry starts to feel like it is getting in the way, and that's never acceptible.

Simply put, Tapestry's abstractions on forms and fields is both leaky and insufficient. Tapestry is trying to do too much, and simply can't keep up with modern, reasonable demands in terms of responsiveness and useability inside the client. We've become used to pages rebuilding and reformatting themselves even while we're typing. For Tapestry to understand how to process the form submission, it needs a model of what the form looks like on the client-side, and it simply doesn't have it. There isn't an effective way to do so without significantly restricting what is possible on the client side, or requiring much more data to be passed in requests, or stored server-side in the session.

The primary issue here is that overall form submission cycle, especially combined with Tapestry's need to serialize commands into the form (as the hidden t:formdata field). Once you add Ajax to this mix, where new fields and rules are created dynamically (on the server side) and installed into the client-side DOM ... well, it gets harder and harder to manage. Add in a few more complications (such as a mix of transient and persistent Hibernate entities, or dynamic creation of sub-entities and relationships) into a form, it can be a brain burner getting Tapestry to do the right thing when the form is submitted: you need to understand exactly how Tapestry processes that t:formdata information, and how to add your own callbacks into the callback stream to accomplish just exactly the right thing at just exactly the right time. Again, this is not the Tapestry way, where things are expected to just work.

Further, there is some doubt about even the desirability of the overall model. In many cases, it makes sense to batch together a series of changes to individual properties ... but in many more, it is just as desirable for individual changes to filter back to the server (and the database) as the user navigates. Form-submit-and-re-render is a green screen style of user interaction. Direct interaction is the expectation now, and that's something Tapestry should embrace.

What's the solution, then? Well, it's still very much a moving target. The goal is to make creating client-side JavaScript libraries easier, to make it easier to integrate with libraries such as jQuery (and its vast library of extensions), make things simpler and more efficient on the client side, and not sacrifice the features that make Tapestry fun and productive in the first place.

Overall Vision

The overall vision breaks down into a number of steps:

• Reduce or remove outside dependencies
• Modularize JavaScript
• Change page initializations to use modules
• Embrace client-side controller logic

Of course, all of these steps depend on the others, so there isn't a good order to discuss them.

Reducing and removing outside dependencies

Tapestry's client-side strength has always been lots of "out of the box" functionality: client-side validation, Zones and other Ajax-oriented behaviors, and a well-integrated system for performing page-level initializations.

However, this strength is also a weakness, since that out of the box behavior is too tightly tied to the Prototype and Scriptaculous libraries ... reasonable choices in 2006, but out-of-step with the industry today. Not just in terms of the momentum behind jQuery, but also in terms of very different approaches, such as Sencha/ExtJS and others.

It was a conscious decision in 2006 to not attempt to create an abstraction layer before I understood all the abstractions. I've had the intermediate time to embrace those abstractions. Now the big problem is momentum and backwards compatibility.

Be removing unnecessary behaviors, such as animations, we can reduce Tapestry's client-side needs. Tapestry needs to be able to attach event handlers to elements. It needs to be able to easily locate elements via unique ids, or via CSS selectors. It needs to be able to run Ajax requests and handle the responses, including dynamic updates to elements.

All of these things are reasonable to abstract, and by making it even easier to execute JavaScript as part of a page render or page update (something already present in Tapestry 5.3), currently built-in features (such as animations) can be delegated to the application, which is likely a better choice in any case.

Modularizing JavaScript

Tapestry has always been careful about avoiding client-side namespace polution. Through release 5.2, most of Tapestry's JavaScript was encapulated in the Tapestry object. In Tapestry 5.3, a second object, T5 was introduced with the intention that it gradually replace the original Tapestry object (but this post represents a change in direction).

However, that's not enough. Too often, users have created in-line JavaScript, or JavaScript libraries that defined "bare" variables and functions (that are ultimately added to the browser's window object). This causes problems, including collisions between components (that provide competing definitions of objects and functions), or behavior that varies depending on whether the JavaScript was added to the page as part of a full-page render, or via an Ajax partial page render.

The right approach is to encourage and embrace some form of JavaScript module architecture, where there are no explicit global variables or functions, and that all JavaScript is evaluated inside a function, allowing for private variables and functions.

Currently, I'm thinking in terms of RequireJS as the way to organize the JavaScript. Tapestry would faciliate organizing its own code into modules, as well as application-specific (or even page-specific) JavaScript modules. This would mean that de-referencing the T5 object would no longer occur (outside of some kind of temporary compatibility mode).

For example, clicking a button inside some container element might, under 5.3, publish an event using Tapestry's client-side publish/subscribe system. In the following example, the click events bubble up from the buttons (with the button CSS class name) to a container element, and are then published under the topic name button-clicked.

Consider this an abbreviated example, as it doesn't explain where the element variable is defined or initialized; the important part is the interaction with Tapestry's client-side library: the reference to the T5.pubsub.publish function.

Under 5.4, using the RequireJS require function, this might be coded instead as:

Here, the t5/pubsub module will be loaded by RequireJS and passed as a parameter into the function, which is automatically executed. So, this supports JavaScript modularization, and leverages RequireJS's ability to load modules on-the-fly, as needed.

Notice the difference between the two examples; in the first example, coding as a module was optional (but recommended), since the necessary publish() function was accessible either way. In the 5.4 example, coding using JavaScript modules is virtually required: the anonymous function passed to require() is effectively a module, but its only through the use of require() (or RequireJS's define()) that the publish() function can be accessed.

This is both the carrot and the stick; the carrot is how easy it is to declare dependencies and have them passed in to your function-as-a-module. The stick is that (eventually) the only way to access those dependencies is by providing a module and declaring dependencies.

Change page initializations to use modules

Tapestry has a reasonably sophisticated system for allowing components to describe their JavaScript requirements as they render, in the form of the JavaScriptSupport environmental (an environmental is a kind of per-thread/per-request service object). Methods on JavaScriptSupport allow a component to request that a JavaScript library be imported in the page (though this is most commonly accomplished using the Import annotation), and to request the initialization functions get executed.

Part of Tapestry's Ajax support is that in an Ajax request, the JavaScriptSupport methods can still be invoked, but a completely different implementation is responsible for integrating those requests into the overall reply (which in an Ajax request is a JSON object, rather than a simple stream of HTML).

Here's an example component from the TapX library:

The @Import annotation directs that a stack (a set of related JavaScript libraries, defined elsewhere) be imported into the page; alternately, the component could import any number of specific JavaScript files, located either in the web application context folder, or on the classpath.

Inside the afterRender() method, the code constructs a JSONObject of data needed on the client side to perform the operation. The call to addInitializerCall references a function by name: this function must be added to the T5.Initializers namespace object. Notice the naming: tapxExpando: a prefix to identify the library, and to prevent collisions with any other application or library that also added its own functions to the T5.initializers object.

The JavaScript library includes the function that will be invoked:

Under 5.4, this would largely be the same except:

• There will be a specific Java package for each library (or the application) to store library modules.
• The JavaScriptSupport environmental will have new methods to reference a function, inside a module, to invoke.
• Stacks will consist not just of individual libraries, but also modules, following the naming and packaging convention.

Embrace client-side controller logic

The changes discussed so far only smooth out a few rough edges; they still position Tapestry code, running on the server, as driving the entire show.

As alluded to earlier; for any sophisticated user interface, the challenge is to coordinate the client-side user interface (in terms of form fields, DOM elements, and query parameters) with the server-side components; this is encoded into the hidden t:formdata field. However, it is my opinion that for any dynamic form, Tapestry is or near the end of the road for this approach.

Instead, it's time to embrace client-logic, written in JavaScript, in the browser. Specifically, break away from HTML forms, and embrace a more dynamic structure, one where "submitting" a form always works through an Ajax update ... and what is sent is not a simple set of query parameters and values, but a JSON representation of what was updated, changed, or created.

My specific vision is to integrate Backbone.js (or something quite similar), to move this logic solidly to the client side. This is a fundamental change: one where the client-side is free to change and reconfigure the UI in any way it likes, and is ultimately responsible for packaging up the completed data and sending it to the server.

When you are used to the BeanEditForm component, this might feel like a step backwards, as you end up responsible for writing a bit more code (in JavaScript) to implement the user interface, input validations, and relationships between fields. However, as fun as BeanEditForm is, the declarative approach to validation on the client and the server has proven to be limited and limiting, especially in the face of cross-field relationships. We could attempt to extend the declarative nature, introducing rules or even scripting languages to establish the relationships ... or we could move in a situation that puts the developer back in the driver's seat.

Further, there are some that will be concerned that this is a violation of the DRY pricipal; however I subscribe to different philosophy that client-side and server-side validation are fundamentally different in any case; this is discussed in an excellent blog post by Ian Bickling.

Certainly there will be components and services to assist with this process, in term of extracting data into JSON format, and converting JSON data into a set of updates to the server-side objects. There's also a number of security concerns that necessitate careful validation of what comes up from the client in the Ajax request. Further, there will be new bundled libraries to make it easier to build these dynamic user interfaces.

Conclusion

In this vision of Tapestry's future, the server-side framework starts to shift from the focus of all behavior to the facilitator: it paints the broad stokes on the server, but the key interactions end up working exclusively on the client.

I'm sure this view will be controversial: after all, on the surface, what the community really wants is just "jQuery instead of Prototype". However, all of the factors described in the above sections are, I feel, critical to keeping Tapestry relevant by embracing the client-side in the way that the client-side demands.

I think this change in focus is a big deal; I think it is also necessary for Tapestry to stay relevant in the medium to long term. I've heard from many individual developers (not necessarily Tapestry users) that what they really want is "just jQuery and a restful API"; I think Tapestry can be that restful API, but by leveraging many of Tapestry's other strengths, it can be a lot more. Building something right on the metal feels empowering ... until you hit all the infrastructure that Tapestry provides, including best-of-class exception reporting, on-the-fly JavaScript aggregation and minimization, and (of course) live class reloading during development. java

I'm eager to bring Tapestry to the forfront of web application development ... and to deliver it fast! Monitor the Tapestry developer mailing list to see how this all plays out.