Back from Tapestry 5 Training / Madison, WI

Just got back last night from training in Madison WI. This was primarily an intro to Tapestry 5 for Tapestry 4 developers (and a couple of new-to-web-programming developers) and a lot of hands on pair programming to get them going with their conversion work. It was a lot of fun, and as usual, I learned a bit.

I have a couple of weeks respite now before OSCON which will be followed with training in York, UK, then Estonia, then London.

I'm continuing to hold off on code changes to Tapestry 5, though I hit enough minor annoyances that I'm considering a 5.1 bug fix release before starting 5.2 in earnest.

I have a backlog of work related to Tapestry360 and then my main emphasis is going to be Documentation, Screencasts, Articles, Documentation and Documentation, plus some additional Documentation. It may be a while before I start with new features for 5.2.

Upcoming Clojure Talks

I'll be speaking about Clojure at Open Source Bridge on June 17th here in Portland.

In addition, I'll be speaking on Clojure at OSCON, on July 23rd. OSCON has left its traditional home, Portland, in favor of San Jose, alas. If you are thinking of attending, use my friends, co-workers and family discount code, os09fos, and save 20%.

Tapestry Road Show: Madison, Wisconsin

This week, I'm in Madison, Wisconsin doing some on-site Tapestry training and mentoring. I've been here before and remember it as a pretty cool little college town. In any case, if you want to get together one night this week and chat Tapestry, Java, Open Source, Clojure or anything like that, drop a line!

Vendor Client Relationship Video

This has been making the rounds on YouTube:

As a vendor, I can identify with this ... but I think a lot of clients out there (those not using Formos, for example) are ordering and charged for the Filet and receive the Tacos. Seriously ... multi-year development efforts are a different beast than purchasing a DVD, which is why short iterations and agile development keep everyone's expectations (of both cost and functionality) inline and satisified.

Back from Training and Portland Code Camp

It's been a rough week ... I still had my sore throat (noticeable during the webinar) when I arrived to do four days of accelerated Tapestry training in Michigan. Returning after midnight on Friday, I had morning and afternoon slots at Portland Code Camp on Saturday to talk about Clojure and Tapestry. I think it was a good little conference, and 75 minute time slots are just barely enough time to say something meaningful.

I attended a nice introduction to jQuery (once again confirming that I backed the wrong horse when selecting Prototype over jQuery for Tapestry), and another session on coding for iPhone.

The only other session I attended was iPhone Development from an ex Softie by Rory Blyth. It was entertaining in an unusual way, since Rory is very glib in a stream of consciousness kind of way, but he spent all but five minutes of his time ranting against Objective-C and iPhone toolkits. Literally he had five minutes for the core of his talk!

I was one of a few people in the audience who knew Objective-C (though it's more than ten years since I coded in it) and found many of his objections quite unreasonable. Basically, he wants Objective-C to look like every other language derived from C, which is missing the point of what Objective-C actually is: a melding of concepts from C and Smalltalk designed to operate on the very constrained hardware available, even for desktops, in the late 80's. It obligates developers to do something unreasonable by today's standards (a cumbersome retain count mechanism, rather than garbage collection), and the (optional) type syntax (such as (NSString *)) reveals its C heritage (as Smalltalk doesn't deal with declared types).

I even made this point to him; that Objective-C may be a natural fit for the constrained devices such as mobile platforms. His response to any challenge from any audience member was that we were afflicted with "Stockholm syndrome".

Strangely, a few minutes after I pointed out the "constrained device" theory (which he dismissed, disjointedly citing Windows smart phones as a "success") he then talked about ... the constraints of the iPhone in terms of memory, battery and CPU utilization.

Basically, Rory is unable to wrap his head around anything unfamiliar or to understand how a difference in philosophy can inform how a language syntax evolves, as well as the terminology (i.e., Objective-C's "receivers", "messages" and "selectors") used to describe that language.

There's a quote from the book Freakonomics, roughly (from memory):

Morality is how we think we should live our lives. Economics reveals how we actually do.

Rory has a kind of "language morality" that states the objects should be listed first, with periods separating member access, such as method invocation, and that languages that deviate from this are failed and broken. Unfortunately for that argument, the explosive success of the iPhone and the iPhone app market indicates that Objective-C is a tremendous development platform for the kind of intuitive, focused, responsive applications that dominate the market.

It was a shame, because his style was entertaining, if very "slacker" styled, and if he organized his thoughts a bit and kept track of the clock, his valid criticisms of the iPhone development environment would hold a bit more weight and reach a wider, more receptive audience.

Tapestry 5 Support in IDEA Maia

The next generation of IDEA, Maia will include Tapestry 5 support; this is according to Hugo Palma, who wrote the initial version of the IDEA Tapestry plugin that has been expanded for inclusion in Maia. Cool!

Tapestry & Clojure @ Portland Code Camp

Portland Code Camp is a local (in Portland) free conference on development tools and languages ... and it's coming up fast (this Saturday). I'll most likely be talking about Tapestry and Clojure ... I actually won't know until they make their last minute session selection tomorrow (it's based on how many attendees express interest). If you're in the Portland area and want to learn a bit more about Tapestry or Clojure, sign up and vote!

Tapestry Road Show: Ann Arbor, Michigan

This week, I'll be in Ann Arbor, MI to do a bit (aka, four grueling days) of on-site Tapestry training. If you're in Ann Arbor and would like to get together for a drink and a chat about Tapestry, Open Source, or anything along those lines, please drop a line!

Last call: Tapestry Webinar (Thu May 21)

Formos is sponsoring a live Tapestry Webinar on May 21st. I'll be presenting a 35 - 40 minute introduction to Tapestry 5, followed by live Q&A. I'll be focusing on the big-picture issues about what makes Tapestry useful, unique and fun. If you've been curious about what Tapestry is, or want a chance to ask a question about Tapestry directly, this is a great chance to do it!

Click a link below to register for the Webinar:

• Thu, May 21, 2009 8:00 AM - 9:00 AM PDT
• Thu, May 21, 2009 1:00 PM - 2:00 PM PDT

Tapestry 5 Performance Study

Ben Gidley has done a detailed analysis of Tapestry 5 performance He used some quite reasonable assumptions, including running tests for over an hour to gauge memory use over time. His conclusion is that Tapestry performance is excellent, which is a relief if not a surprise.

What I'm really pleased with is that he's demonstrated that Tapestry will perform well under load; more importantly, if a Tapestry application gets "Slashdotted", it may slow down, but it recovers once the wave of requests dies down.

Ben compared a Tapestry application to a Struts application with similar behavior and found that T5 out-performed Struts pretty handily, despite the fact that T5 doubles the number of requests (due to its automatic redirect-after-post behavior).

Thanks for the great work Ben!

Clojure @ JavaWorld

Another nice introduction to Clojure is now online: Clojure: Challenge your Java Assumptions at JavaWorld. An introductory Tapestry article is due at JavaWorld shortly as well.

Apache Tapestry Community Map

Inspired by a map for Clojure, I created a map for Tapestry users:

View Apache Tapestry Users Worldwide in a larger map

Want to add yourself? The map is wide open, something I'll probably regret later.

Update: People have been adding themselves over the weekend, and trends are starting to appear. It seems like adoption is heavier in Europe than in the US (at least, among the narrow set of people who use Tapestry, know about the map, and have actually added themselves). Still, if that sample is true, it begs the question: why? Is it more to do with the European developers, or due to features of Tapestry (such as localization support).

I'm tending towards the former: talking with many Europeans shows that managers have a more "hands off" attitude towards project leaders, who are given more free reign to use tools of choice as long as they deliver. But that's also a horrible generalization from limited data. In fact, a contrary opinion was offered by some developers in Oslo last september ... in Norway, a small number of developers make strong vocal choices (lately, Spring and SpringMVC) and the whole community follows along. But even Europe is a big place which room for many different developer cultures.

Tapestry 5 Webinar: May 21st

Formos is sponsoring a live Tapestry Webinar on May 21st. I'll be presenting a 35 - 40 minute introduction to Tapestry 5, followed by live Q&A. I'll be focusing on the big-picture issues about what makes Tapestry useful, unique and fun. If you've been curious about what Tapestry is, or want a chance to ask a question about Tapestry directly, this is a great chance to do it!

Click a link below to register for the Webinar:

• Thu, May 21, 2009 8:00 AM - 9:00 AM PDT
• Thu, May 21, 2009 1:00 PM - 2:00 PM PDT

Apache Tapestry 5.1 Final Release - 5.1.0.5

The stable release for Tapestry 5.1, release 5.1.0.5, is now available for download.

Tapestry 5.1 includes a large number of improvements to the speed and scalability of Tapestry without sacrificing backwards compatibility. However, you should still consult the upgrade notes that identify a couple of minor "gotchas".

Major improvements from Tapestry 5.0 to Tapestry 5.1:

• Performance optimizations that speed rendering of very complex pages, and reduce memory usage
• Static JavaScript libraries will now be dynamically combined into a single request
• Automatic GZIP compression of static and dynamic content, for clients that support it
• Significant improvements to Tapestry's property expression language
• Partial page updates (via Ajax) may now update multiple client-side Zones
• Many new features and improvements to Tapestry template files
• Improved client- and server-side validation of numeric input values
• User's preferred locale is now encoded into the URL, rather than stored in a cookie
• A new ProgressiveDisplay component for incremental page loading
• Vastly improved Quickstart Archetype
• New Tapestry IoC features to make it easier to override services and contributions to service configurations
• Many new localizations of validation messages
• Component reference documentation now identifies events triggered by components
• Really slick new client-side logging facility (based on Blackbird)
• New API for URL rewriting (to provide improved control over Tapestry URLs)

... and many, many more bug fixes and improvements; full details are available in the release notes.

Download Tapestry 5.1.0.5

Clojure: Turns out, parsing is somewhat iterative, not functional

... at least for me. I'm working on some more real-world code in Clojure, and I'm struggling with turning a stream of tokens (representing XML content parsed from a stream) into a DOM-like hierarchy. The tokens represent open tags, attributes, close tags, and text content.

I've written dozens of recursive decent compilers in the past, in Pascal, C, PL/1 and Java. The problem is, that style of compiler works by consuming a stream (of characters or tokens) ... a destructive operation.

Logically, I want code such as:

List parseTemplate(tokens)
{
  result = []

  while (tokens not empty)
  {
    token = pop(tokens)
    if (token.type == text)
    {
      add text token to result;
      continue;
    }

    if (token.type == element)
    {
      e = parse_element (token, tokens)
      add e to result;
      continue;
  }
}

Now, in Clojure I can write a (parse-element) function that takes a bunch of tokens, and it can work its way through the tokens using Clojure's (loop) and (recur) forms ... but to the calling function, tokens will not have changed.

One approach might be to pass along some pretty complex state; literally a state machine (inside a map) so that each function can keep push and popping values into and out of the state before invoking each. This would involve recursing on each token. Anyway, I have a rough idea of what that would look like and it seems complicated.

Instead, I've been pounding my head, looking at Clojure's support for monads. Monad are a way to combine certain types of functions together in a way that appears as if there were mutable state. You provide some base types of monadic functions and, within the context of a monad definition, it streamlines the way those basic monadic functions can be strung together to form more complex monadic functions.

Still trying to wrap my head around the concept however. It gets really confusing because the real power of monads appears when the monadic values, the things returned from a monadic function, are themselves functions. Yep, that's where I get lost too.

Update: passed the initial Monad hurdle; I'm finally beginning to understand them, perhaps even enough to explain them to other people.

Excellent Clojure Introduction

R. Mark Volkmann has put together a terrific Clojure Introduction. It starts with basic syntax and quickly works through all the core functions, broken out by categories such as "Collections", "Iteration" or "Concurrency".

Very brisk, but full of functions (from the core) that I didn't know of.

For example, here's a function I never noticed before, (get-in), as defined in the Clojure API

(get-in m ks)
returns the value in a nested associative structure, where ks is a sequence of keys

That's accurate, but really encourages you to read the source or experiment in the REPL to see exactly how it works. Worse, its mixed in with hundreds of other functions and really easy to miss (I never noticed it before!).

Here's what the Clojure Introduction says:

To demonstrate this we'll create a map that describes a person. It has a key whose value describes their address using a map. It also has a key whose value describes their employer which has its own address map.

(def person {
  :name "Mark Volkmann"
  :address {
    :street "644 Glen Summit"
    :city "St. Charles"
    :state "Missouri"
    :zip 63304}
  :employer {
    :name "Object Computing, Inc."
    :address {
      :street "12140 Woodcrest Executive Drive, Suite 250"
      :city "Creve Coeur"
      :state "Missouri"
      :zip 63141}}})

The get-in function takes a map and a key sequence. It returns the value of the nested map key at the end of the sequence. The -> macro and the reduce function can also be used for this purpose. All of these are demonstrated below to retrieve the employer city which is "Creve Coeur".

(get-in person [:employer :address :city])
(-> person :employer :address :city) ; explained below
(reduce get person [:employer :address :city]) ; explained below

This is a document to keep linked and to print out and study. Thanks Mr. Volkmann!

Ruby and CouchDB

I've started looking into CouchDB of late; I think it may be a great solution to some of our common issues around collecting "survey" style data from users. Our current approach uses a big nasty hierarchy of Hibernate entities.

I wanted to have some real, and real-world, data. Since I'm a big board gamer, I thought it would be nice to have a local copy of the BoardGameGeek database. I wrote a program that scrapes data from the BGG site and loads it into CouchDB.

#!/usr/bin/ruby

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'thread'
require 'couchrest'


# Extension to Hpricot

class Hpricot::Elem
  # content at
  # if block given, return value comes from yielding
  # actual content to the block
  def cat(expr)
    result = self.at(expr)

    return nil unless result

    content = result.inner_html.fixup

    return (yield content) if block_given?

    return content
  end

  # content-at converted to int (via to_i)

  def cati(expr)
    self.cat(expr) { |content| content.to_i }
  end

  def search_to_text(expr)
    self.search(expr).map { |e| e.inner_text.fixup}
  end
end

$BGG = "http://boardgamegeek.com"


$catalog_pages = 0
$game_pages = 0
$games_added = 0

# Access to couch db.  From what I can tell, the Databsae is multithreaded.

$DB = CouchRest.database!("http://127.0.0.1:5984/board-game-geek")

# The thread pool.

$POOL_SIZE = 10

$QUEUE = Queue.new

workers = (1..$POOL_SIZE).map do |i|
  Thread.new("worker #{i}") do
    begin
      proc = $QUEUE.deq
      proc.call()
    end until $QUEUE.empty?
  end
end

def enqueue &action
  $QUEUE << action
end


def parse_browser_page(url)
end

class String
  def fixup()
    self.gsub("&#039;", "'").gsub("&amp;", "&")
  end
end

def parse_and_load_game(game_id)

  $game_pages += 1

  page = Hpricot.XML(open("#$BGG/xmlapi/boardgame/#{game_id}?comments=1&stats=1"))

  bg = page.at("//boardgame")

  ratings = bg.at("//ratings")

  comments = (bg/"comment").map do |e|
    {
            "user" => e[:username].fixup,
            "comment" => e.inner_text.fixup
    }
  end

  doc = {
          "_id" => game_id,
          "title" => bg.cat("name[@primary='true']"),
          "description" => bg.cat("description"),
          "designers" => bg.search_to_text("boardgamedesigner"),
          "artists" => bg.search_to_text("boardgameartist"),
          "publishers" => bg.search_to_text("boardgamepublisher"),
          "published" => bg.cati("yearpublished"),
          "categories" => bg.search_to_text("boardgamecategory"),
          "mechanics" => bg.search_to_text("boardgamemechanic"),
          "images" =>{
                  "url", bg.cat("image"),
                  "thumbnailUrl", bg.cat("thumbnail"),
                  },
          "players" => {
                  "min" => bg.cati("minplayers"),
                  "max" => bg.cati("maxplayers"),
                  "age" => bg.cati("age")
          },
          "stats" => {
                  "rank" => ratings.cati("rank"),
                  "averageRating" => ratings.cat("average") { |content| content.to_f },
                  "ownedCount" => ratings.cati("owned")
          },
          "comments" => comments
  }

  enqueue do
    $games_added += 1

    $DB.save_doc(doc)
  end

end


def process_game(game_id)

  begin
    doc = $DB.get(game_id)

    # Found, do nothing

  rescue RestClient::ResourceNotFound

    # Not in the database yet, so fire off a request to parse its page.

    enqueue { parse_and_load_game game_id }

  end
end

def parse_catalog_page(url)

  puts("[%24s] %4d catalog pages, %4d/%4d games parsed/added (%4d actions queued)" % [Time.now.ctime, $catalog_pages,
       $game_pages, $games_added, $QUEUE.length])

  doc = Hpricot(open(url))

  $catalog_pages += 1


  doc.search("//table[@id='collectionitems']/tr/td[3]//a") do |elem|
    href = elem[:href]
    game_id = href.split('/').last()
    enqueue { process_game game_id }
  end

  next_page_link = doc.at("//a[@title='next page']")

  return unless next_page_link

  next_page = next_page_link[:href]

  # Add this last, to give the other actions a chance to operate.
  # A better mechanism would be a priority-based queue, where the catalog
  # page parse action is lower priority than the other actions.

  enqueue { parse_catalog_page($BGG + next_page) }

end


# Kick it off

start_page = $ARGV[0] || ($BGG + "/browse/boardgame")

enqueue { parse_catalog_page start_page }

# Wait for all the workers to complete

workers.each { |th| th.join }

puts "BoardGameGeek loader complete."

puts "Queue not empty!" unless $QUEUE.empty?

Key pieces of this is Hpricot to parse HTML and XML, and CouchRest to get the data into CouchDB.

Did I go overboard? I don't think so ... this runs in about two hours, and created a database of nearly 40,000 documents (about 340MB). Using a thread pool just seemed to make sense, since (outside of the XML parsing), every aspect of this is I/O bound: pulling data from BGG or pushing data to CouchDB.

The code is naive about some threading issues and simply crashes if there's an error. Oh well.

Next up: learning how to build views against this data and deciding how to use it all. I'm thinking Cappuccino.

Wither Derby?

Over lunch last week with co-workers, we were discussing the Oracle buy out of Sun and how that would affect MySQL and what other options there would be. Several people had some experience with PostgreSQL (including myself), but then I brought up Apache Derby ... and no one had used it or event downloaded it, and only a few had even heard of it. I just checked their web site and the last update was in November 2008. When I last played around with it, it seemed to have all the major features of a real database and the advantage of being pure Java (I can't speak to performance).

How many people are using Derby and if not, why not?

Tapestry 5.1 is Beta!

We just had a vote, and Tapestry 5.1 (that is, release 5.1.0.3) is a beta release. At this point, everything is stable except for minor changes to the brandy-new URL rewriting APIs. I've been chasing down bugs and improvements for the 5.1.0.4 release, which will probably be created and voted on later this week. My guess is that 5.1.0.4 will be the final/stable release for 5.1.

I'm already thinking for 5.2. Ditch Maven for Buildr? Strong possibility. Start using Groovy for tests? Why not? Finally get Spring Web Flow integration working ... you betcha! Portlets? Shouldn't be a problem. Release date? I think before the end of the year.

Tapestry @ JavaOne: Tapestry State of the Union

I'll be presenting a Birds of a Feather session at JavaOne about Tapestry this year. Part of the session will be a presentation about important Tapestry 5 features that many people aren't aware of, and a roadmap for Tapestry 5.2 and beyond. The balance will be Q & A and a chance to get your Tapestry opinions heard.

The session is June 4th from 6:30 to 7:20PM in Esplanade 301. See you there!

Is it time to abandon IDEA for Eclipse?

I've been a big fan of IDEA since I made the switch about two years ago ... but as I've run each new preview release of IDEA it seems to be getting slower and slower. My Tapestry integration tests take about 5 to 6 minutes to run from the command line; inside IDEA the time to run has recently spiked up: they are now taking 15 - 20 minutes ... I just had a run take nearly 30! My suspicion is that it has something to do with all the console output from the tests. I see the launched Java process (which runs TestNG, Selenium and Jetty) consuming 3% of CPU and IDEA consuming 97%. IDEA is almost completely non-responsive while the tests run. I run both IDEA and the launched Java process with 600M max heap.

Two years ago, IDEA was stable and Eclipse was not runnable on Mac OS X. That's what prompted my first switch to IDEA. I'm now finding Eclipse to run faster on OS X and on Ubuntu. Lately, I feel pulled in the opposite direction: IDEA is quickly becoming too painful to use!

Tapestry on Google Application Engine

Many people have been working on this exciting subject. Jun Tsai has a Tapestry 5.0.18 application running http://ganshane.appspot.com. It looks like he took the Tapestry 5.1.0.3 quickstart archetype and just shifted the version number down to 5.0.18.

Tapestry 5.1 deployments haven't been working, because of the shift in XML parser from generic SAX to Woodstox STAX. An issue has been added, and the Application Engine folks are looking to add the necessary classes to their "white list", but it's not clear when such support will be added.

Conversational State without Spring Web Flow

I've been trying to find time to write the integration between Tapestry and Spring Web Flow ever since I first met Keith Donald at TheServerSide Symposium in 2005. I really, really, really think it will make it into Tapestry 5.2. In the meantime, a simpler solution to many of those problems is conversational state, and that's now available as tapestry-conversations, from the Trails teams.

Is GZIP compression compatible with XmlHttpRequest?

Does using GZIP on an application/json response stream to an Ajax request cause problems? This question has been coming up a lot in Tapestry, to the point that we had to explicitly turn it off. For some users, on some combination of server and client (not always the ones you'd expect), the response comes out garbled. It's been hard to diagnose, but it almost seems that the Content-Encoding: gzip header is lost, leaving the browser to parse the compressed response as uncompressed (however, this theory is far from confirmed).

None of this occurs with traditional, non-XHR, requests and responses. Is it tied to XHR or to the content type (application/json)? Could this be a bug in Prototype? On Windows only? It's been hard to track down the triggering environment for this, but it appears that my initial guess (a proxy server in the middle) doesn't bear fruit.

Have other app or framework developers confirmed this? If so, what was your approach to addressing this. I know I'd like to see full GZIP compression for all that JSON content.

Using Maven Quickstart Archetype from Eclipse

I'm working on a Tapestry article for JavaWorld, and I wanted to discuss setting up Eclipse with Maven and Jetty support, and creating a new Tapestry project from the latest quickstart archetype.

It ended up being much too large to include in the main article, so I've created a full rundown on the Tapestry360 wiki instead.

This covers where to get the plugins, how do deal with a M2Eclipse bug and the workaround that lets you use the Tapestry quickstart archetype, and some other customizations of the environment.

Reducing the size of a Ubuntu VMWare image

I've been building out my VMWare Ubuntu image, the one I use for Tapestry training, and have found that the size has grown out of control. Although inside the VM, my image is only using about 4G, the size of the VMWare folder was well over 8G, too large to fit on the 8G USB flash drives I use to distribute the image.

The solution is time consuming but not difficult; you need to have VMWare Tools installed in the image. If you run VMWare Tools as root (i.e., sudo vmware-toolbox), you can select the primary partition and shrink it. First, many minutes go by as the partition is prepared ... then you are asked if you want to shrink it. Many more minutes go by. At the end, the image is reduced in size significantly ... in my case, to 5.8G total, well within my limit.

Installing Subversion 1.5 binaries in Ubuntu 8.04

I'm a little left in the lurch currently, because I need Subversion 1.5 client binaries in my Ubuntu 8.04 virtual machine (the one I use for training). This is because Eclipse check projects out in such with 1.5 compatibility (IDEA is kind enough to ask which version to use). In any case, since setting up my Tapestry Workspace involves a bit of command-line SVN, I found myself needing to upgrade SVN to 1.5.

With a little hunting around and some guesswork, I found the solution.

You need to add a new Software Sources location; this is a tool, "Software Sources" inside the Applications > System menu. Click the tab labeled "Third Party Tools" and add a new source. You'll be prompted to enter an "apt-get line": deb http://archive.ubuntu.com/ubuntu/ intrepid main

From there, go into Synaptic Package Manager. Find package "subversion" and you'll see that there's a version of 1.5 now available, thanks to the new software source. You can right click and select "Upgrade", then apply the changes.

Another bad day with Maven

Another bad day with Maven ... first, something changed in the Maven Archetype plugin, so now my archetypes don't work anymore ... and I'm far from the only person with this problem.

Second, I thought I'd use mvn eclipse:eclipse instead of my Ruby script to keep my Eclipse .classpath file up to date: but it doesn't seem to want to download and connect the Tapestry Java sources. So back to the script it is.

1819

Nope, that's not a date. 1819 is the current number of individual tests run every time a change is committed to Tapestry. Over 180 of those are Selenium-based integration tests. That's a lot of testing!

Tapestry 5.1.0.2

I'm in the process of creating the Tapestry 5.1.0.2 release, to be voted on. I think 5.1.0.2 will ultimately be the beta release for 5.1, and possibly the final release. The goal is to have a significant upgrade of Tapestry 5.0 that is backwards compatible^* to 5.0 but offers great improvements in speed, improved components and Ajax capabilities, and improved localization support (especially on the client side).

Release Notes - Tapestry 5 - Version 5.1.0.2

Bug

• [TAP5-39] - Add JSON support for literals (often, inline function definitions) that are used to configure some client-side objects (even if they aren't truly JSON)
• [TAP5-573] - NullPointerException during AJAX form submit
• [TAP5-578] - If a component class is abstract, trying to instantiate it (by including it in a template) yields an inscrutable InstantiationError
• [TAP5-590] - A JavaScript error for non-required fields will force Ajax form submits to be handled as a full-page request instead
• [TAP5-593] - Calling ApplicationStateManager.exists() will throw an exception when the session has been invalidated
• [TAP5-598] - Hard-coded messages inside tapestry.js are not localized
• [TAP5-600] - The new Blackbird console makes Safari JavaScript completely non-functional

Improvement

• [TAP5-265] - Add a Hidden component, used to synchronize a value between the server and the client
• [TAP5-398] - Tapestry should check that the service scope is consistent with the service (throwing an exception if the scope requires a service interface and the service doesn't provide one)
• [TAP5-566] - TextField documentation should explain why the required value parameter is not bound in the example
• [TAP5-581] - Quickstart archetype should include testng.xml and webdefault.xml
• [TAP5-584] - Omit generator meta (from head element) when root element is not html
• [TAP5-586] - The JavaScript waitForPage() handler does not need to be added when rendering a partial Ajax response
• [TAP5-589] - Add a method to DOM Element class to allow the collection of Attributes to be obtained
• [TAP5-591] - FAQ: Creating a page render Link from a service
• [TAP5-605] - There should be a simple way to override automatic JavaScript libraries and Stylesheets

New Feature

• [TAP5-288] - Replace JavaScript client-side logging with Blackbird
• [TAP5-549] - JavaScript libraries should be combined into a single request
• [TAP5-557] - Provide support for URL rewriting
• [TAP5-562] - tapestry-hibernate should provide a built-in status page to show basic Hibernate statistics inlcuding cache hits, etc.
• [TAP5-594] - Add simple PageRenderLinkSource service to allow services to create Links to pages

Task

• [TAP5-510] - Improve code coverage of JSON unit tests
• [TAP5-575] - Add svn:eol-style=native to source files

My plans for 5.2 are to finally get the Spring Web Flow and Portlet support in, along with the "skinning" component I've been thinking about. And to continue to make upgrading from release to release as painless as possible ... and to do it fast, just another couple of months from whenever 5.1 goes out.

Tapestry and Blackbird

Just finished integrating the nifty Blackbird JavaScript console into Tapestry. This will make debugging Tapestry applications much more pleasant, especially under IE (which doesn't have FireBug).

Combining JavaScript Libraries

I've been continuing to do work on speeding up Tapestry performance; I think part of the Tapestry message is not just how easy it is to get started, but how the framework grows with you as your application, and your user base, grows.

One aspect of performance is speeding up server side operations, and Tapestry 5.1 has gone a long way down that path. However, there's only so far that can take you, especially since the lion's share of server-side time is spent waiting for database queries.

Another approach is to minimize both the number of requests and the size of the replies. Tapestry 5.1 will GZIP compress replies for dynamic page content and static asset files. That reduces the size of each reply (when the client supports GZIP compression, which all modern browsers do).

What about reducing the number of requests? As with all web applications, the HTML for the page is just the start: there's also requests for images, stylesheets and JavaScript libraries. Tapestry is already good at giving these resources version-numbered URLs and far future expires headers, so that the client browser can cache them aggressively.

Tapestry 5.1 now adds a feature to combine JavaScript libraries. Whatever JavaScript libraries are requested by components when the page renders are combined into a virtual asset: a URL that represents the combination of the individual JavaScript libraries. This virtual asset has an odd URL (there's some Base64 encoded data as part of the file name), but it also has a far-future expires header and may be served to the client GZIP compressed.

This helps a lot with first-time access to an application; page rendering is interrupted only long enough to download a single JavaScript resource, rather than a series of requests for the individual files. Given that JavaScript must be loaded, parsed and executed serially ... and that rendering of the page is on hold until that occurs, combining JavaScript in this way is a pretty big win, especially given the amount of server-side and client-side caching involved.

The best part is, like many features in Tapestry, there's no configuration to worry about. It just works, it's free, there's no special scripts or tasks to run as part of your build. Better yet, it defaults to off for development mode and defaults on for production mode.

Next release, we may try to minimize the JavaScript as well as compress it; minimizing means stripping out comments and unnecessary whitespace and shortening variable and function names. Again, this would happen on-the-fly.

Watchmen Saturday Morning Cartoon

OK, this is a spoof ... but a wonderful one!

Announcing TapX: Tapestry Extensions

I've set up a new project on Tapestry360 called tapx (Tapestry Extensions). This is where I can put code that can't live at Apache (due to licensing problems).

The first module is a DHTML calendar component, a wrapper around Dynarch Calendar, an LGPL calendar widget. I consider it the "Cadillac" of this space, and it snaps into Tapestry quite nicely.

Just dropping this new JAR into the classpath not only makes the new DateField available, but also enables it as the default date editor when using BeanEditForm.

The library requires Tapestry 5.1.0.1 or better.

I expect to add many small utlilities here.

Clojure at OSCON

Just got the word that my Clojure session has been accepted for OSCON 2009. Covering the language in so little time is going to be quite the challenge!

Like Tapestry? Like GORM? Use 'em together!

A Tapestry user by the name of Otho has posted notes on integrating Tapestry 5.1 with GORM, the dynamic object-relational mapper at the core of Grails. Cool stuff!

1970's (and 80's) Genre TV Panorama

NOW IN COLOR Sci-Fi Heroes by *dusty-abell on deviantART

Wow!

I mean, this has got shows that had a viewership of, like, three. That space station in the upper left corner? That's SID from U.F.O. (and an Interceptor on the right)! Richard Benjamin as Quark (from the short-lived Star Trek parody). The Man From Atlantis. Dr. Who (with Leela, K-9, a Dalek, the Tardis and the Robot (from Tom Baker's first aired episode) and another Robot from "Robots of Death" (a favorite!). Buck Rogers. Battlestar Galactica (including the Cylon Emperor). Both Wonder Women (that is, including the original one-off pilot with Cathy Lee Crosby). Space: 1999. Battle of the Planets (the blue ship on the left). Land of the Giants (Twiki is holding their ship). The short-lived live-action Spider Man and Planet of the Apes series. The one-shot Capt. America (which I'm not certain I've seen). The Six Million Dollar Man, Bionic Woman (with Bionic German Shepherd Max). The Incredible Hulk and David Banner (who ever sees both them at the same time?). Mork from Ork. The Greatest American Hero ... oh, just spotted The Invisible Man (weren't there two different invisible man series in the 70's/80's). Logan's Run (front and center, almost missed it) ... I think that's it.

What else could have been included? How about The Starlost (I've only seen one episode), saturday morning fodder like Ark II, or the original Night Stalker. Or the BBC's Blake's 7. Or the silly but occasionally provoking Land of the Lost. Any other ideas?

Running IDEA 8.1.1

I've switched to IDEA 8.1.1 (i.e. build 9742) and it's working very nicely. It has plugins available for Clojure and GIT, which is nice.

Faster and more stable than 8.0 would be nice; we'll see if that works out in practice. Also, I've configured my environment to use JDK 1.6 (not 1.5) as the default JDK; so that should help with performance as well. I need to decide whether compiling with JDK 1.6 but targetting JDK 1.5 is sufficient for Tapestry releases, or whether I should switch back to 1.5 when creating releases.

Meanwhile, I've been updating my training workshop (in case I end up teaching at Skills Matter in London any time soon), and I'm switching over to using Eclipse, not IDEA, for the workshop. I think people will appreciate the familiarity of Eclipse, even under Ubuntu. There goes my chance to change the world, though!

The Importance of Back Ups

I had seen most of this commercial last year, while on the road, and hazy from jet lag. At the time I wasn't sure I hadn't dreamed it up.

I also like the ending quote from the alternate version of the ad:

Many customers buy one for each side of the bed

Ruby script to rebuild Eclipse .classpath

I'm in the process of rebuilding my Tapestry training workshop to use Eclipse instead of IDEA. One part of this is that I use the Maven Ant tasks to handle dependencies from an Ant build file. The end result are folders of JAR files:

lib/runtime

Runtime JARs

lib/runtime-src

Corresponding source JARs (where available)

lib/test

Test-only JARs

lib/test-src

Corresponding source JARs

With Tapestry + Hibernate, that's like 20 JAR files ... way too many to maintain by hand (especially across five similar projects). I mean, the Eclipse API for this is not designed for frequent changes ... I should be able to point Eclipse at a directory and say "take everything you find" (like I can in IDEA).

I tend to chase the Tapestry release, updating the Workshop project as each new Tapestry release is available. I can't be doing this by hand all the time!

What to do, what to do ... ah, a Ruby script! I've actually gotten very rusty with Ruby, so there's probably easier ways to do this, but the basics are there. I love that Ruby has lots of ways of quoting text that aren't intrusive:

#!/usr/bin/ruby
# Rebuild the .classpath file based on the contents of lib/runtime, etc.

# Probably easier using XML Generator but don't have the docs handy

def process_scope(f, scope)
 # Now find the actual JARs and add an entry for each one.
 
 dir = "lib/#{scope}"

 return unless File.exists?(dir)
 
 Dir.entries(dir).select { |name| name =~ /\.jar$/ }.sort.each do |name|
   f.write %{  <classpathentry kind="lib" path="#{dir}/#{name}"}
   
   srcname = dir + "-src/" + name.gsub(/\.jar$/, "-sources.jar") 

   if File.exist?(srcname)
      f.write %{ sourcepath="#{srcname}"}
   end
   
   f.write %{/>\n}
 end
end

File.open(".classpath", "w") do |f|
  f.write %{<?xml version="1.0" encoding="UTF-8"?>
<classpath>
  <classpathentry kind="src" path="src/main/java"/>
  <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
}

 process_scope(f, "runtime")
 process_scope(f, "test")
 
 f.write %{
  <classpathentry kind="lib" path="src/main/resources"/>
  <classpathentry kind="output" path="target/classes"/>
</classpath>
}
    
end

I integrated this script into my Ant build file, so that it runs after I pull down dependencies.

Once I get to the new workshop session on testing, I may need to update this to take account of testing (including compiling src/test/java to target/test-classes, etc.). Still, 45 minutes of tweaking (which would have been 10 minutes if I wasn't so rusty) will save me lots of time in the future, dividends payable immediately.

Java.Blogs Death Watch

We're now into day three and counting of the Java.Blogs Death Watch. The site has stopped updating (since the 16th) and nobody seems to care ... in fact, I realized that because of aggregation, I wasn't missing anything by removing this site (and TheServerSide) from my bookmarks. Still, it seems sad that the owners of Java.Blogs (and who exactly is that?) are uninterested in keeping the site running.

Speeding up Tapestry 5.1

I've very excited about some big changes coming in Tapestry 5.1: A significant increase in performance.

This may come as a surprise ... that performance changes are even needed! Tapestry's response time is nearly instantaneous for typical applications. I'm really talking about atypical applications ... applications that push the boundaries on performance and scalability.

This has been driven by a couple of clients who are building large applications on Tapestry: both large numbers of concurrent users, and very complex pages, containing hundreds of components that, due to loops, render thousands of times.

One particular client, Lithium, has been moving from a JSP to a Tapestry 5 solution for their network of community sites. They've been a great help in terms of tracking performance problems and solutions.

Make it Work. Make it Right. Make it Fast.

Tapestry 5 is a complete rewrite of the aging Tapestry code base ... though its hard for me to think of it as "new", since I've been working on it since 2006! Still, the first part of Tapestry 5 was to make it work and Tapestry has lead a lot of innovation here, especially in terms of live class reloading, and the new Inversion of Control container. By the time of the final release, Tapestry was "right" and it was fast enough ... but no effort had been made at any point to make it fast. Tapestry 5.0 was coasting along on good basic design, and the raw speed of Java. With Lithium's input, I've been working to make the actual code live up to the performance goals.

Comparing with JavaServer Pages

Comparing Tapestry application performance can be an uphill battle, since Tapestry's memory usage and processing pattern is completely different than something simple, like a servlet or JavaServer Page (JSP). One hurdle that may never be surmounted is the intermediate representation: JSPs have none; the first bit out output will stream to the client, which is brutallly efficient.

Tapestry, by contrast, actually renders templates and components into a light-weight DOM (Document Object Model). Some post-processing of the DOM then occurs ... some new elements are added related to JavaScript and other features. Then the DOM is rendered to a character stream. Tapestry is always going to require more memory per request and take longer to begin rendering content.

Further, Tapestry pages are full, stateful objects, rather than the singleton objects that JSPs compile down to. This too represents more overhead: to create them in the first place, to manage (and reset) their internal state, and to pool them between requests.

Interestingly, for smaller pages, Tapestry is as fast or faster than JSPs. You can see why if you look at the Jasper JSP implementation: it spends a lot of time managing pools of JSP tag instances, constantly checking them out, configuring them for a single use, then cleaning them up and putting them back into the pool. Tapestry uses a much more coarse caching policy (entire pages, not components within pages) and has more opportunities to optimize the flow of data within the page, between components.

Speeding up Page Rendering

My first pass at improving performance was to optimize Tapestry rendering state machine. Tapestry breaks the rendering of each individual component up into a series of states:

A component may provide a render phase method for each state. But most components will supply methods for just one or two of these phases. Some phases only make sense for components with templates (not all components do). Tapestry 5.0.18 grinds through each state for each component regardless ... a bit of thrash in the middle of rendering. By analyzing which render phase methods a component actually implements, Tapestry 5.1 is able to optimize away large chunks of the state machine. This results in considerably fewer render operations: less work to accomplish the exact same output. In my sample test application the number of render operations declined 25 - 30%. More importantly, combined with a few spot optimizations, the Tapestry application came down to spitting distance to the JSP application in terms of performance, even though it had vastly more complex behavior.

I actually experimented with a number of other options which I won't detail here. An important part was to measure performance, using JMeter and YourKit. I spent several days analyzing hot spots and trying out different theories.

Speeding up Page Assembly

This was a much more interesting change for real applications and real development. In Tapestry, pages are objects with internal state (not singletons the way servlets and Struts Actions are); this means we need to create multiple instances of pages when two or more requests reference the same page simulateneously.

Instantiating a page, the function of the PageLoader service, is rather involved; it must coordinate the contents of the page's template with the page's components and their templates; it must match up attributes in templates to parameters of components and hook those up as well. One thing I noticed is that there was a lot of duplicated effort when the same component is used multiple times, either on the same page, or on multiple pages.

My approach here was to split page instantiation into two phases: a one time analysis phase to figure out what needs to be done to actually construct a page, and a second repeatable phase to perform the construction. The goal here was to limit the amount of computation necessary when constructing a page (especially the second time).

The end result is a lot of code like this:

   private void expansion(AssemblerContext context)
    {
        final ExpansionToken token = context.next(ExpansionToken.class);

        context.add(new PageAssemblyAction()
        {
            public void execute(PageAssembly pageAssembly)
            {
                ComponentResources resources = pageAssembly.activeElement.peek().getComponentResources();

                RenderCommand command = elementFactory.newExpansionElement(resources, token);

                pageAssembly.addRenderCommand(command);
            }
        });
    }

This is a big win for development in large projects with many large and complex shared components, cutting down on the refresh time after change to a Java component class.

Speeding Up The Client

Improvements to the server side are one thing, but in many cases, a bigger win is optimizing the client side. Web applications are more than just individual HTML pages: all the JavaScript, stylesheets, images and other assets are just as important to a user's perception of performance.

Tapestry 5.1 addresses this in two ways: versioning and compression.

Versioning

In Tapestry 5.0, classpath assets (stored in JARs) are exposed through the Tapestry filter. These assets end up with a URL that includes a version number, and are provided to the client with a far-future expires header. This means that the client web browser will aggresively cache the file. This doesn't help with the first hit to a web site, but makes a big difference when navigating through the site, or making return visit, since most of what the browser needs to get content up on the screen will already be present without an HTTP request.

Tapestry 5.1 extends this: context assets (files stored in the web application context) can now be exposed with an alternate URL that also incudes an application-defined version number, and also get the far-future expires header.

Compression

Tapestry 5.1 also adds content compression: if the client supports it, then rendered pages and static assets that exceed a configurable size can be sent to the client as a GZIP compressed stream. This does help with the first visit to the application.

There's an advantage to letting Tapestry do the compression; Tapestry will cache the compressed bytestream, so that later requests (from other clients) for the same content will get the compressed version without paying the server-side cost to compress the content. The more traditional approach, using a servlet filter, doesn't have the ability to determine what content is dynamic and what is static, so it has to compress and re-compress the same content blindly.

Backwards Compatibility

The crowning achievement of all these changes is the compatibility issue: the only things that changed were internal implementations and some internal interfaces. Existing Tapestry applications can upgrade and immediately benefit from all the changes with at most a recompile.

Part of my goals for Tapestry is to ensure that your application can start small and grow with you. These performance improvements will not be visible for a small in-house application, or a niche application ... but once your application is successful and grows in scope and popularity, the performance you need is already in there by default.

IDEA 8.1 on Mac

Installed it, launched it. Immediately: Jetty Integration is not compatible and disabled. Strike one. Sure, it has GIT integration and promises to be much faster (which I'm getting desperate for, as I seem to watch the cursor spin quite a bit too much). However, my level of trust dropped through the floor when I saw just how many graphics glitches are all over the screen ... it doesn't inspire confidence that it was tested on a Mac at all. Back to 8.0.1.

A Better Web Framework: Tapestry's Response

Last month, Ibrahim Levent published a detailed posting about the capabilities of a better web framework on his blog.

I'd thought I'd describe my reaction to it, as well as how Tapestry today, and Tapestry in the future, fits in with this vision.

Certainly there's a lot here, and there's a troubling lack of focus: Mr. Levent is demanding very specific features that span a number of domains. In effect, he's asking for an application server vendor to deliver the One True Stack ... to which I say "good luck with that!"

1- Includes all core application layers (MVC):

Web framework should include data access, business logic, controller and presentation layers internally. As frameworks turn out to be an integration hub, it looses value. Every integration among the core layers introduces new complexity, new glue code, new dependency, and conflicting of intersecting features. If data access layer (Model) uses another framework, presentation layer (View) uses another framework, integrating these frameworks adds a very big challenge even if frameworks support each other. Replacing any framework causes many new problems later. For example, JPA is developed for data access independence but at this time you are limited only the features of JPA. IDE is a major development tool, but at this time we need an “Integrated Development Frameworks” environment within IDE. (Similar with ERPs that brought together enterprise applications under the same umbrella)

Choice is a good thing. I can get up and running quickly using Hibernate; others prefer Cayenne or pure JDBC. I wouldn't want to mandate just one, but Mr. Levent is correct that frameworks must adopt the role of an integration hub, and Tapestry (with it's very dynamic, very late binding Inversion of Control container) really fits that bill!

I also chafe at the mention of the IDE: we've been down that path before (.Net, JSF) ... what we really need are tools that work with a minimal amount of support from the IDE.

2- Avoids heavy-componentization:

In web architecture, desktop-like componentization is heavy and inflexible. Components in desktop applications were very successful. They utilized reusability and used in IDEs. In web applications, component model doesn’t work at the same form. Efforts to convert HTML (+JavaScript) into component model will not be successful as desired. This is because HTML is dynamic (DHTML), works on client and declarative (Declarative Programming). With heavy-components, we loose declarative programming to some degree. We loose “Web Graphics Designer” ability to edit web pages because of moving from design-time to run-time and moving structure information from HTML to programs (With losing HTML Editors functions). Web editor’s favorite structure place is CSS files so what about CSS componentization? Another problem is architectural. Web GUI has 2 runtimes; server and client (Browser). At previous years, web frameworks supported only server-side-only functionality. Then today we see client-side-only approach. I think best solution is balanced mixture of both client-side (JS) and server-side code with component templates (not hard components but light partial HTML+JSP+Servlet codes). I’ll not detail further, there are already many discussions about “Component-based” versus “Action-based” frameworks on the web.

I come down strongly PRO componentization; that's been the focus of Tapestry since day one. Mr. Levent is correct that as more of a page's content is encapsulated inside components, the high level templates (page templates, in Tapestry terms) start to loose their ability to be view stand-alone, outside of the running application. However, I'd rather trade productivity and consistency (and testability) across my application for this one "feature". Here's a better question: how hard would it be to set up your application to run for the designer? Tapestry can allow a designer to run the application and see changes in real time.

3- No new tag markup or page template:

Some web framework requires learning a new markup with no added-value. Your form inputs turn out to be strange tags. Finally, developers don’t understand HTML, JavaScript, CSS because no time left for this learning. Who will fix GUI errors? Frameworks should bring minimum or no new tags (instead we may prefer attributes). HTML tags with simple JSP expressions are enough (KISS). Isolating developers from HTML and JavaScript is not possible.

Tapestry really excels here, as the Tapestry Markup Language templates are just XHTML with a namespace for the Tapestry parts; and those can be limited to just a t:id attribute with all other details in Java code. I don't do it that way ... it's more work for little gain, but a purist can appreciate this.

Even at the opposite extreme, a "heavily" instrumented Tapestry template is still pretty light, with no true Java code (though a few proeprty names and expressions will show up).

4- No XML usage:

Heavy XML usage for configurations makes programs hard to develop, hard to understand, hard to test. One example is “Page Flow” information in XML files. Another example is bean configuration. Yes, pulling this information makes it flexible but who needs it? How many times your page flow changed? How many times did we utilize flexible bean configuration? What about source code readability? I don’t like “Dependency” so “Dependency Injection”. I think dependency is not free that you have to manage its subtleties. Here is my anti-pattern “Dependency Rejection”. XML can be used in other useful places like AJAX messages or data import-export.

Here's where I agree; only Tapestry templates use XML. Tapestry 5 did away with all other XML (except for the ubiquitous web.xml, which is only touched once, when first creating a Tapestry project).

I feel that Mr. Levent is really missing the boat here; properly used Dependency Injection is incredibly important. Dependency Injection is what makes a clunky dinosaur of a language like Java useful, scalable in complexity, testable, and extensible (via late binding). To paraphrase: Dependency Injection is like violence; if it isn't working for you, you aren't using enough of it!

Dependency injection is critical to source code readability because it allows you to easily break your code into small, focused bits that each perform a well-defined function. The IoC container's job is to put all those tiny, testable bits back together into a running application. Tapestry IoC and Guice do this with aplomb.

5- Has its own web GUI page elements:

Rich web elements (say light components) are generally found only in JS or AJAX libraries. Web frameworks should provide rich elements like; Calendar, Dialog, Menu, Popup, Progress Bar, List, Grid, Tab (With sub-levels), Master-Detail Windows, Child Windows, Record Navigator etc. Developers can easily extend these elements. We are still turning around simple features like table sorting, filtering etc. We should step ahead. There is still no desktop-like web grid components to use (I see only in JS libraries) that I mentioned in my previous blog post.

Tapestry does well here (and this requirement seems to contradict item #2). In any case, Tapestry has decent support built in, with lots of great 3rd party support.

Fundamentally, Tapestry is page oriented: the Ajax effects can be well integrated, but not the degree of either a Google Web Toolkit solution, or something entirely hand-tooled (on top of Direct Web Remoting, perhaps).

6- Code generation:

Code Generation makes “Rapid Development” possible. Every part of software should be generated (Generative Programming); CRUD data access classes, business code, controller code, and view pages. Code generation takes development one step ahead of “Drag and Drop” WYSIWYG editors. If web framework facilitates code generation, developers could jump to customization details of application instead of building everything from scratch (MDA).

I disagree here: I don't like code generation unless it happens at runtime. If you look at Tapestry's "scaffolding components" (BeanEditForm and Grid particularly), you can see this ethic: the application is dynamically assembled at runtime. Likewise, all of Tapestry's meta-programming happens by class instrumentation at runtime, without a tedious build stage.

7- Has its own GUI JavaScript library:

Another bleeding integration point is JavaScript libraries. JavaScript libraries are not fully-integratable with web frameworks. They try to solve the problem in client side. What we need is close cooperation with client-side and server-side. Most of web frameworks unfortunately have no or little JavaScript in their presentation layer.

Again we return to integration; Tapestry has a set of libraries built on top of Prototype and Scriptaculous. Many applications also bring in jQuery. They all mix together nicely on the client side.

8- AJAX support (Asynchronous Communication):

AJAX eliminates bothering page-refreshes. Web frameworks should properly blend AJAX functionality into their code architecture. AJAX requires server-side coding. As we make client runtime powerful with AJAX, GUI state management code is duplicated. For example, if we update and fill a combo-box with AJAX call then server-side bean that is bound to this element is not aware of this state change. We have to change server-side state as well. AJAX functionality should be implemented without code duplication (Another interesting trend is AJAX MVC).

... and the nature of component encapsulation is to allow Ajax without the fuss and duplication alluded to here.

9- Portable among application or database servers:

Application and database portability is not easy. In Application Server side, class loader policies change, session management changes, deployment model changes etc. In DBMS side, join clauses change, paging, and sequence generating changes. Web frameworks should provide portable packages for different platforms. On the other hand, some web frameworks have their IDE and Application Server (believe me even DBMS). I think we must leave this job to the famous bright products (IDEs and Application Servers in the market).

This can be a sore point; the servlet API doesn't specify a few important behaviors for Tapestry (that mostly show up only in a cluster). I'm not sure what a "bright product" is though? Any clues?

Tapestry does work on popular servers (Jetty, Tomcat, WebLogic) because it's careful to follow the Servlet API rules, especially with respect to careful use of the HttpSession.

10- Input validation:

Data input validation is a very important feature. If validation doesn’t occur in application, database error occurs. Database errors are not user-friendly. Some validation errors may not be related to database. Programmers need automatic validation according to database object metadata. Custom validations should be added if needed.

I agree, and add further, that validation should occur on the client and then be re-executed on the server. Once you escape from the web tier, the errors get uglier.

11- Bug-free:

Because of bugs in frameworks, all average developers become framework expert spending valuable time to figure out the problem. “Focusing business problems” is lost. I read many open source framework hacks and workarounds in many blogs which is not the task of developer.

As if proprietary code is bug free? This one gets my blood pressure up ... I can't tell you how much time I've spent stepping though WebLogic code, guessing at what's gone wrong (where a bit of source code would have helped). The alternative to Open Source is to still become a framework export, but pay through the nose for the privilege, and deliberately let yourself become helpless, in thrall to your vendor.

12- Handles exceptions user-friendly:

If error or exception occurs, user-friendly messages should be returned. Application programmer has some responsibility for this but web frameworks may ease this task.

Tapestry excels here; I strongly maintain that Tapestry's exception reporting is the best of breed, with a detailed exception report and lots of contextual data ... and the ability to easily turn it all off or otherwise customize what happens when things go wrong.

13- Eliminates double-click, double-submission problems:

Double-click may cause double-submission. Double-submission may cause unexpected errors in application (2 threads tries to do same thing). Web frameworks can eliminate this problem even in client-side without going to server.

This is on the wish list; certainly a little JavaScript to disable the form or submit button goes a long way here! But a better solution intercepts the duplicate submission and that requires some coordination across the server cluster, which is why it isn't in Tapestry yet.

14- Authentication and authorization support:

User login (authentication) is still developed by programmers without knowledge of SQL-Injection attacks. Web application authorization is still missing. Who will be granted for CRUD on which application etc.?(User roles, permissions) I am sure that in every enterprise web application, application authentication and authorization is re-invented.

Is it the role of the application framework to define your security constraints? In a very constrained world, such as content management system, these roles and their application is well defined. I the real world of real applications, it's much harder to pin down. I've worked on many apps that had somewhat intricate permission schemes, and the ability for some users to "jump out" of those schemes.

That being said, Tapestry's modularity means that a standard security library can just be "dropped in". That's what we've been doing at Formos; we use a standard permissioning system, based on page and method level annotations.

15- Security controls for web attacks:

Web frameworks should prevent web security attacks like; Cross-Site Scripting (XSS), SQL Injection, URL Manipulation, HTTP Injection, Session Hijacking etc. Web client data is un-trusted and open to tampering so this is why we can’t quit totally server-side validation for the sake of client-side validation.

Tapestry does a great job on these issue; XSS is virtually impossible, as all output generated by Tapestry is "filtered" unless you specifically ask Tapestry not to. SQL Injection can't occur in a world where you are using Hibernate or another layer to generated prepared statements (this isn't PHP!). URL Manipulation is also somewhat of a non-starter because URLs are linked to components and components are configured on the server side to perform specific functions. It's not like Struts or Rails where you can hack a form submission to turn your admin flag on!

As I mentioned earlier, Tapestry re-performs input validation on the server side.

There is a concern in Tapestry in that Forms store serialized object data on the client side. This is both insecure and inefficient. A future release of Tapestry will address this by either encrypting or signing the data, or by storing the data server-side and just sending a "token" to the client.

16- Reporting integration and barcode support:

Reporting integration is important. We need reporting products/frameworks integration. Would you use your data access objects in your reports? Would your reporting engine use the same JVM runtime? Barcode is not a general requirement but in ERP applications it is very useful (AI/DC Automatic Identification/Data Capture). Barcode printing, barcode reading and matching may be provided by your web framework.(What about RFID?) Would your reporting product support your application barcode?

This is one of those entries in the original blog that simply makes me wonder; Mr. Levent clearly works on a specific category of applications, but I certainly have never written an application that needs to know about barcodes. Barcode reading? What does that even mean in terms of a web framework?

17- Messaging and workflow integration:

Web frameworks may support easy integration with messaging (JMS) and workflow products. Workflow is one of major element of BPM (Business Process Management). In some middleware stacks, this is included (i.e. JBoss Seam jBPM). Web application frameworks may support business events and workflow activities. These events can also be used to feed messaging backbone (ESB).

Even the example here is odd, and reinforces my earlier points: JBoss Seam doesn't have built-in workflow, its the Seam jBPM module that integrates into Seam. So as long as you are good at integration, we're in the success zone. And Tapestry is great at integration.

18- Application to application integration (i.e. Web Services):

In Java, there is external system (EIS, legacy) integration API, which is JCA, but inter-application communication within same JVM is not standardized. Let’s say we have 2 applications and one should use some call other application code. There is no standard for this. Basic solution is just adding other application’s path into its class-path and then using other application objects. We developed an Adapter API for standardization of this. In one-application environment, this is not a problem but if many applications are required to communicate, it gets more important. You can even convert your APIs into web services when necessary (integration with remote or non-Java systems). Web frameworks may provide tools for web services code generation, deployment and monitoring.

Mr. Levent has moved, about here, from some strong goals and guidelines for a web application framework to a kind of development environment wishlist.

19- Admin application for run-time process and user session monitoring:

This is very important in point of user and system management view. What are my users executing at the moment? Which applications take longer to finish? Which users are on-site? Which pages are they surfing? In each session, which objects are they created? What are the URLs that a user requested? Which SQL statement did a user execute?

This is an interesting concept and one that could perhaps be implemented using Tapestry's various meta-programming facilities. I've definitely been thinking long-term about a Dashboard facility.

20- System resource management:

If your application runs big queries that require a lot of system resource (CPU, RAM, DISK I/O), we are faced the reality that resources are limited. If applications don’t restrict user processes, then system will consume its all resources and will not respond to even small processes. For the sake of system availability some user may be rejected by system. Web framework may have such limitation API’s.

This concept is a tricky thing to bootstrap; if your machine is truly strapped, it may have trouble just getting to the point where it can determine how strapped for resources it really is! I know of no general purpose web frameworks that have this kind of feature.

21- Cluster support:

When server load is high and performance is a major concern, load-balancing is required. Application server clustering will not suffice, web frameworks must support cluster architecture. One simple example is framework’s id generators. They will collide in clustered Application Server environment.

I'm not sure what framework id's he's getting at here. Clustering a servlet application is generally quite sufficient, and clustering Tapestry is even easier, as it is very careful about what data is stored into the session. Tapestry is also good at keeping mutable objects stored in the session "fresh" when they are updated, but mostly it stores many small immutable objects where other frameworks store large mutable objects.

In terms of IDs; session ids can have, for example, DCE ids that can be cheaply generated anywhere with a guarantee of uniqueness (they just tend to be quite large). Database ids are generated, efficiently, by the shared database.

22- Multi-database, multi-company, multi-window, multi-session support:

Application user may need to work on multiple database instances. One user may have to work with multiple companies. User may want to use multiple GUI windows. Web framework should handle or prevent state corruption among windows. User may need to work on the system with many sessions.

To me, this indicates a single application deployed, and perhaps "skinned", multiple times ... or represents a single application that is capable of connecting to multiple databases at the same time.

Multiple windows can be something of a challenge; a single server-side session is shared across windows. Tapestry can encode state into URLs, which is handy but ultimately limiting. I think in the future Tapestry has the best chance of dealing with this cleanly because there's the gulf between persistent page fields and the session, which allows Tapestry to arbitrate ... literally, store different values in the session for different windows, but the same user. Not something implemented today, but quite possible.

23- Internationalization:

If there are global users, then i18n support is important. One key aspect here is Application Server and DBMS should also support your localization.

Tapestry has greate L10N support; applications can have localized message catalogs as can individual pages and components. Templates and assets (images, stylesheets, etc.) can be localized as well. Tapestry uses your browser's reported locale, but this can be overridden programatically. In Tapestry 5.0, you'll receive a cookie with your "true" locale. In 5.1, the "true" locale will appear in the URL (which is more search engine compatible).

24- SSL support:

If web application is wanted to be secure in insecure networks, SSL-support is important. SSL deployment in HTTP Server would not be enough. Even if SSL is not used, frameworks must encrypt sensitive data between client and server, like user passwords.

You may mark Tapestry pages as secure, using an annotation. Tapestry will automatically use HTTPS when building links to secure pages, and will reject any attempt to access a secure page using insecure HTTP.

25- Document attachment:

In every enterprise application, document attachment is important. Users may want images, Excel documents attached to their application records. Every programmer first search for an upload utility then tries to understand server document folders. Instead, built-in functionality saves valuable time.

Back to a wish list and not a real framework goal.

26- Mobile device support (i.e. Internet Explorer Mobile):

If we want to plan mobile access to our applications, how can we do this with web technologies? Many mobile devices have built-in web browsers and we may run our applications in these browsers. Web framework mobile support would be very beneficial at such cases. Otherwise, you should explore mobile web browser limitations by yourself.

I have long maintained that an application for a mobile device and an application for a desktop browser are not the same application. Creating a useful version of an application for a size and bandwidth limited client is more than just choosing new fonts and omitting a few options ... to do it succesfully is a completely different flow, and therefore, a different application (or at least, a seperate corner of the application).

You often hear about a magic XST transformation (Coccoon, anybody?) where a single service layer could be vended out in multiple formats. But I've never seen one in practice that worked, scaled and was maintainable, never mind acceptable to end users.

27- Portal features:

Partial web components should be supported to use in Portals or external sites. In portal terminology, its name is portlet. There are many synonyms; Widget, Mashup etc.

Tapestry 4.0 was a great platform for Portlet development, that will return in Tapestry 5.2.

28- Scheduling:

Application task may be batched and scheduled. After task completion, users may see results.

I have long thought of a layer for Tapestry to leverage Quartz for this purpose. Again, Tapestry's current goal of being a comprehensive user interface layer (rather than a total vertical application framework, which is what Mr. Levent is looking for) has made some of these non-goals for Tapestry.

29- Keyboard hot-keys:

Users, especially old TUI (Text UI) users want keyboard hot-keys. Buttons, command icons should be bound to hot-keys. Web frameworks elements can support this instead of developing in every application.

This is largely a function of HTML and JavaScript, things well encapsulated by Tapestry.

30- Alerts between users:

Users may want to send messages to each other or system admin may want to send messages to users like notifying a shutdown or an application restart. This feature will be very handy.

Again, a wishlist item that could easily be implemented for a specific application.

Summary

Mr. Levent has brought up a number of interesting concepts, and a number of real oddities, in his quest for the "improved web framework" (in fact, he's looking for a vertical application framework with some very specific niche capabilities).

I can't say that Tapestry fits his bill perfectly ... but I can say that Tapestry would be my first choice to anchor the stack that would meet his needs. The most important features of his "better web framework" are already present in Tapestry today.

Clojure at TheServerSide Symposium

I'll be presenting a lightning fast overview of Clojure at this year's TheServerSide symposium. I'm speaking at 3:50pm on Wednesday, March 18th.

Obama Icon Me

Tapestry 5 Refcard

DZone has just published a Tapestry 5 Refcard, authored by yours truly. It was a great challenge fitting even a representative sample of all the great concepts and features of Tapestry 5 into less than six pages. Download it and tell me what you think!

Did I mention that RealPlayer is evil?

So for the last few months, my wife has been having constant trouble with her MacBook, specifically, wireless connectivity. Our wireless network has always worked well for me, but hers was constantly un-responsive. Intenet connections would work for a minute or two, then hang. It was constantly trying to re-connect to the router. DNS names would not look up properly. Pings would never return. I tried upgrading from a DLink router to an Apple Internet Extreme ... same problem.

Was it a hardware issue? Suzy took her MacBook to the Apple Store. They noticed some oddities and tried re-installing the operating system (without wiping the drive).

Was it a Comcast issue? I mean, they are rat bastards and totally clueless ... and as it turns out, our cable connection was wired backwards and it was lucky we got any signal at all. But that never affected me, and even once Comcast came out and rewired it, the change didn't seem to make a difference for Suzy's MacBook.

Meanwhile, Samantha's Dell worked great on the wireless. So did Merlyn's MacBook Pro. And my MacBook Pro. And the Wii. It got so that every time Suzy would bring it up, I'd start getting angry because I couldn't fix it! And fixing things is what I do!

Last night, staring in frustration at her MacBook, I did something I should have done a ways back: launch Console.app and see if there was anything interesting in there. And there was ... endless error messages from "RealPlayer Downloader" (which was running in the dock) with messages specifically mentioning the network interfaces.

It wasn't clear what it was doing there. Calling home? Opening up ports? Suzy had installed RealPlayer to watch a work-related video for a client months past and had forgotten about it. Guess what? I removed the application (using AppZapper) and Suzy's internet connection has been smooth sailing ever since.

Moral of the story: Run, Run, Run away from RealPlayer!

Using Maven to create a new Tapestry 5.1 project

I've been digging deep into the (revised) Maven archetype plugin: the tool used to generate new projects from a template. The existing quickstart archetype had some glaring omissions, partially related to limitations of the old plugin. The new plugin makes many more things possible.

In the past, you needed to specify the new project's groupId, artifactId and other data on the command line. This process was so tedious and error prone that I advised wrapping it up in a Ruby script.

Now it's a bit easier, as Maven will ask you for any necessary properties:

$ mvn archetype:generate -DarchetypeCatalog=http://tapestry.formos.com/maven-snapshot-repository
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'archetype'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO]    task-segment: [archetype:generate] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] Preparing archetype:generate
[INFO] No goals needed for project - skipping
[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [archetype:generate]
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
1: http://tapestry.formos.com/maven-snapshot-repository -> quickstart (Tapestry 5.1.0.0-SNAPSHOT Quickstart Project)
Choose a number:  (1): 1
[INFO] snapshot org.apache.tapestry:quickstart:5.1.0.0-SNAPSHOT: checking for updates from quickstart-repo
Define value for groupId: : com.formos 
Define value for artifactId: : demo1
Define value for version:  1.0-SNAPSHOT: : 
Define value for package:  com.formos: : com.formos.demo1.web
Confirm properties configuration:
groupId: com.formos
artifactId: demo1
version: 1.0-SNAPSHOT
package: com.formos.demo1.web
 Y: : 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24 seconds
[INFO] Finished at: Sat Jan 31 13:12:31 PST 2009
[INFO] Final Memory: 8M/14M
[INFO] ------------------------------------------------------------------------
~/work
$ 

The key part is the -DarchetypeCatalog=http://tapestry.formos.com/maven-snapshot-repository, which points Maven at Tapestry's own nightly snapshot repository. It picks up a catalog file from there which points to the quickstart archetype stored inside the snapshot repository. No muss, no fuss.

Of course, the first time you run this, Maven has to download about 34MB of modules!

In any case, it works nicely. I put some extra effort in so that the archetype description clearly identifies the exact version of Tapestry.

The new quickstart archetype is much improved; it now includes a Layout component that wraps the page content using a free CSS layout, "concrete" (which has a Creative Commons license). In other words, your application will look good from day 1, though (of course) I'd expect you to throw away the layout subsequently. Anyway, this will answer some questions such as: "how do I use a Layout component?" and "how do I access a message catalog?"

I can see replacing the "concrete" layout with something more general, less blog-oriented (but that's beyond my CSS/HTML design skills!).

All of this is all usable right now. Have fun!

A tip for TestNG: Keep the Constructor Simple

TestNG does not do a great job of reporting exceptions when instantiating your test classes. Keep the content simple. If you are going to do anything that can throw an exception, defer it by creating a setup method annotated with @BeforeClass.

In my situation, I had an exception, and add my normal TestNG verbosity level, 2, there was no obvious output explaining why my tests didn't run, just a message about "unable to instantiate" my test case class. Upping the verbosity to 10 helped me identify the problem ... at which point I moved a bunch of setup logic out of my constructor and into a setup configuration method.

Using a Samsung ML-1450 with Airport Extreme

I just upgraded my router to an Airport Extreme, partially just to allow Suzy to use the printer even when I don't have my laptop hooked up. However, the printer in question is a Samsung ML-1450 which doesn't have great compatibility under Mac OS X.

After a little searching I have it working quite nicely. An Apple Support Thread plus some poorly written release notes almost tell the whole story.

Remove your printer (from the Printer Preferences) if it currently exists.

Delete the folder /Library/Printers/SAMSUNG. That's all the old set of Samsung drivers.

Download the updated Samsung Drivers.

Run the install; it will do a full restart.

Open Printer Preferences and add the printer back in. It should auto-discover the new drivers (which will display as "Samsung SPL 2.5").

The Tapestry Roadshow at The Great Indian Developer Summit (April 22 - 25, 2009)

I'll be presenting three sessions at the Great Indian Developer Summit in Bangalore, India this coming April. The sessions are on Clojure, Tapestry IoC, and a long tutorial session on Tapestry 5.

I'm not sure what days I'm speaking yet.

Comparing Prototype and jQuery

Glenn Vanderburg has written a careful treatise on why he prefers Prototype to jQuery. The thing about Glenn is that he doesn't write (or talk, or think) in terms of "feelings" or "impressions"; his thoughts are clear, and organized, and fact based, and detailed.

I looked at jQuery about two years ago; to me it seemed a little iffy, a little skewed towards web developers doing bespoke web page development, rather than the dynamic generation Tapestry is all about. That impression may or may not be accurate, but Glenn has nailed some significant differences in the design and implementation of the two frameworks, which is why Prototype comes out on top.

Upcoming features for Tapestry 5.1

I've been working on some new features for Tapestry 5.1. Importantly, I've been trying to get my head around Spring Web Flow and how that would integrate with Tapestry without sacrificing the flavor of either framework ... fortunately, Keith Donald (SWF Head Honcho) has been helping out.

I'm also very interested in performance ... or really, throughput. I made a number of changes in December that dramatically improved Tapestry performance for very large and complex pages (with thousands of component renders). It can be an uphill battle, given that Tapestry creates an in-memory DOM of the page and streams that out at the very end (unlike a JSP that can spit out bits and pieces of content immediately at the start of its render). There's certain overhead in creating all those DOM objects and then navigating them while rendering. So where we take away on one side, we need to add back on another ...

The goal is to minimize the amount of traffic over the wire; specifically, the amount of static content: assets such as images and stylesheets that don't change during the life of the application.

Tapestry 5.1 now has built-in GZIP compression of responses, both for dynamic page renders (traditional and Ajax) and for those static assets.

I'm also automatically building versioned URLs for context assets (not just classpath assets stored inside JARs); they'll get the benefit of automatic compression and far-future expires headers too.

This falls into my vision for Tapestry in particular, and frameworks in general: they should help you employ best practices, even if you don't even know such practices exist. Just using Tapestry becomes a best practice all by itself^*.

Another way to embrace this view is to see Tapestry as the embodiment of the combined experiences of the Tapestry developers and community over the last many years. Good lessions, hard won, realized in code.

^*At this point, steam should be coming out Ted's ears!