Tapestry Training -- From The Source

Let me help you get your team up to speed in Tapestry ... fast. Visit howardlewisship.com for details on training, mentoring and support!

Monday, December 22, 2003

Presenting ... my card

In case anyone whose reading this blog isn't aware, I left the safe confines of my ordinary, salaried job for the unpredictable consulting space. I haven't had much of a chance to market myself, but I have made up some business cards: I've blurred out my home address (just out of fear of identity theft).

So, consider this the first salvo in marketing Howard Lewis Ship; more is coming, building on some   good   notices. You want to build good Tapestry applications ... and I want to help.

Spindle / Tapestry Sweatshirt

Geoff may be a Real Smart Guy for creating Spindle, but he's a Total Genius for creating the Spindle Sweatshirt!

Friday, December 19, 2003


I'm done at NLG ... at least until 2004. I have no day job to go to! I can finish the revisions and copy editting of the book. I can go skiing. I can sleep in. I can visit friends. I can see Return of the King. I can write the rest of the Tapestry reference docs. I can update the unit test suite. I can push Tapestry to 3.0 final. I am free --- from everything, but my inner taskmaster. I can live with that. Freedom!

Thursday, December 18, 2003

Large scale Tapestry approaches

In a previous post I mentioned some ideas that came out of my talks with the folks at Intelligent Works.

Their problem, which is not too uncommon, is that they are adding additional lines of business to their existing app. They already have about 100 pages, most of which are CRUD (Create Read Update Delete) for their existing object model.

Now they are adding new lines of business that extend their object model with new subclasses. In many cases, the high level looks very similar, and they are only seeing differences fairly low in the navigation tree. That is, a Work Order looks much like a Purchase Order (with a couple of fields swapped out), and the real differences don't crop up until you drill down to the line item level or below.

So, I came up with some solutions off the top of my head ...

Smart Components

Use the same pages, but have smart components that are responsible for editting the details of the object. These smart components, built around Block and RenderBlock, can mix-and-match a user interface on the fly, drawing appropiate components from Blocks on secondary pages.

This kind of approach means that WYSIWYG preview goes out the window, the page becomes a kind of shell whose job is to just set the stage: what object is being editted and how. At multiple layers below, the smart components are complex conditionals to provide the right UI for the particular type of object being editted.

For IW, there might be an EditAnyOrder page used for editting both Work Orders and Purchase Orders. Again, within this single page, and even within components within other components within the page, there will be dynamic logic to choose the right UI (both static and components).

This is a very powerful approach that leverages the dynamic nature of Tapestry. I'm concerned, however, that finding and fixing bugs will be too complex ... that there will be too much detective work to find where a particular text field comes from ... and too many unwanted side-effects of changes because components and code are being shared in an obscure way.

Dynamic Page Assembly

Tapestry has a bunch of hooks that allow a careful developer to extend or even replace the way Tapestry locates page and components specifications and templates. In theory (and I haven't tried this yet), you could magically "brew up" an EditPurchaseOrder page on the fly. The framework would not care that the specifications and even templates were constructed on the fly, rather than read from static files.

There could be a bit of magic involved in this ... parsing the page name ("EditPurchaseOrder") to figure out how to create the necessary page (some kind of template for "Edit", plus the PurchaseOrder type, plus specific extensions for editting Purchase Orders). What this kind of mapping would look like is more than I can come up with on my feet.

Template Driven Development

The question is: do you need to really assemble it all on the fly? If you know you are going to need an Edit page for Work Orders and Purchase Orders, can you use a templating system to do it. I've been using XDoclet at work lately, and it seems reasonable to use XDoclet to create page and component specifications from Java classes ... if you can figure out XDoclet and come up with the necessary rules, you could have a single Java class (public abstract class EditAnyOrder ....) and from that, generate more Java classes and page specifications (EditWorkOrder, EditPurchaseOrder).

Perhaps Velocity, in the guise of VPP (Velocity Pre-Processor) is more appropriate for generating page and component templates.

I tend to like this solution, because it moves the difficult decision making into the build phase. There are still lots of unknowns.

Other ideas

Long time Tapestry user Jiri Lundak has built a framework on top of Tapestry to address these issues: MetaworkX. I haven't looked at the code or design (yet -- so much to do!) but he's been at this for a while and probably has something clever up his sleeve.

Tuesday, December 16, 2003

My visit to Intelligent Works

Over the weekend, I visited snowy Ottawa, Canada to meet with Geoff Longman (creator of Spindle).

This wasn't a social trip (though it could have been as Geoff is a cool guy) ... it was a chance to meet with another team of people who are aggresively using Tapestry, and who may be needing Tapestry training in the future (or, potentially, involved in training yet more folks). There's a lot of opportunities brewing north of the border for Tapestry, for myself, and for Intelligent Works, Geoff's company.

There were several people in attendance, here's a writeup from Chris about my talk:

From: Geoff Longman <glongman <at> intelligentworks.com>
Subject: Fw: Howard's talk...
Newsgroups: gmane.comp.java.tapestry.user
Date: Tue, 16 Dec 2003 11:36:12 +0000
We were lucky enough to have Howard pop up to visit Intelligent Works here in Ottawa last weekend and give us a very informative (~4 hour) session focused on Tapestry and Hivemind. I asked Chris Justus, a consultant working with Tapestry here, to jot down his thoughts.. Geoff Geoffrey Longman Intelligent Works Inc. P.S. I agree totally, bit I must add that the Hivemind portion of the session was equally as informative! ----- Original Message ----- From: "Chris Justus" <cjustus <at> alceatech.com> To: <glongman <at> intelligentworks.com> Sent: Tuesday, December 16, 2003 11:26 AM Subject: Howard's talk... > I've been working with Tapestry for 6 months, and we have been building > an application that currently consists of about 100 tapestry pages and 50 tapestry > components... Howard's talk was good in terms of explaining some ideas > and concepts behind Tapestry. Having worked with other frameworks in > the past, I consider the idea of using the jwcid attribute a real > solution to a very real problem of having designers and developers work > together on an application... > > In addition: We are also now at the cusp of some major work in which I > was considering using a body of 30 pages, 30 templates, and 30 class > files as a base for 2 other areas of the application - this would result > in the creation of 200 additional files within the project. I spent 3 > minutes explaining our application to Howard, the internal object model, > and the layout of our tapestry pages... In just a minute or two, Howard > had suggested several alternatives to the plan that was going to be > implemented, which will save us tens of hours up front, and likely > hundreds of hours in maintenance down the road... This provides a very > quantifiable savings on the order of tens of thousands of dollars in > development and maintenance. > > Overall Howard's visit was very beneficial... Anyone doing serious > Tapestry work would be wise to meet with Howard and get feedback on how > they are implementing various aspects of their application... > > Chris >

As usual, I enjoy talking about Tapestry (and HiveMind), especially to an audience that really gets it. I also enjoy hearing about user's business models and making suggestions about how to best use Tapestry. In fact, thinking on my feet there, I came up with some very nifty approaches that I'll summarize soon.

Sunday, December 14, 2003

Monday, December 08, 2003

Love to see that support!

In the way that all discussions on TheServerSide tend, a book review of The Art of Java Web Development has drifted into a chest-thumping "my framework is better than your framework" slog-fest. A couple of years back, I was accused of doing this kind of thing, so now I'm extra careful to hang back and let others fight my battles for me :-) ... or, at least, let others bring up Tapestry first.

Jason Cox, a familiar name on the Tapestry mailing lists, has offered a particularily nice description of the advantages of Tapestry, joining some similar notes by Erik Hatcher and MindBridge. Always nice to see new evangalists come on board!

Friday, December 05, 2003

The Art of Java Web Development

New on TheServerSide is a review of The Art of Java Web Development by Neil Ford.

This book covers a number of important open-source frameworks, including Struts, WebWork (1.0) and Tapestry (2.2).

I reviewed the Tapestry chapters over the summer; I was dissapointed that he used such an old release of Tapestry though it probably wasn't reasonable for him to write a book about 3.0 while it is still changing.

I think he hit the major points, but sometimes the focus was off; his examples spent too much code fretting about setting up Log4J (that's what log4.properties is for) which undermines the "hey, where did my code go?" aspect of Tapestry development. If I remember correctly, there was also too much duplicated code concerning database connection pools in the example page classes (that could easily have been moved to the engine or visit -- or in 3.0, to the global object).

I'll need to get a copy of this book to see what's changed since I reviewed it. I'm also hoping he followed my suggestion to reference the substantial improvements in Tapestry 3.0 over Tapestry 2.2.

Still, it's nice that Tapestry is treated to the same level of coverage as Struts, WebWork, Velocity and so forth. And it's great to see anything Tapestry in print.

NHL.com - Powered by Sun, Tapestry and Cayenne

The NHL.com - The National Hockey League Web Site is powered by Tapestry (version 2.3) plus Cayenne. It receives more than 5 million hits a day and uses a mixture of live Tapestry and pre-rendered pages (Perl scripts "scrape" Tapestry generated pages at scheduled intervals). Hats off to Eric Schneider at Central Park Software (hey look -- their home page is a Tapestry app, too).

Resting on my laurels (more from my visit to Darden)

Jamie Orchard-Hays was another person attending my two-day fun-filled Tapestry training session at Darden, and had more favorable comments to go along with Erik Hatcher's.

Thursday, December 04, 2003

The name: Tapestry

Some amusing thoughts by Gavin King about how to name software projects (and about my sense of theater).

Wednesday, December 03, 2003

Typesetting is happening ... HiveMind still in transition

The typesetting for the Tapestry in Action book has started; I've now seen exactly what chapter one will look like (in PDF form). I understand that chapter 2 is nearly ready as well.

How odd, I just looked at the Tapestry in Action page, and the picture has changed ...

Nobody told me! Looks like they're matching the color scheme of this blog :-) I don't know, I've worked pretty hard on this book, I think I deserve a better cover illustration ... maybe a charging Samurai warrior with a bloody sword. So far, we've gotten a chick with a goatee (ok, just a bit of background detail near her chin, but that's what it looks like) and now some kind of prancing skirt-wearing type.

Also, just talked to Prashant, my buddy at WebCT: the HiveMind IP transfer (a matter of getting WebCT to sign over the rights to the HiveMind code over to the ASF) is still going on, they've just had office moves, code freezes, and user conferences get in the way -- it's not a high priority item for them. It may take some time (frustrating!) but still sounds like things are happening.

I've been feeling a bit isolated at my day job (temporarily here at NLG). Seems like the other employees here kind of consider the work their day job, they just don't share my (extreme) passion for finding the right solution, for really getting into the meat of things. I haven't found anyone to replace Prashant at WebCT (or Greg Burd at Primix, or Kenny Anderson at Stratus) ... people who really live and breath this stuff, people who enjoy thinking beyond the next line of code.

Wednesday, November 26, 2003

HiveMind still in transition

The IP issue for HiveMind is still unresolved which is really irking me. I know its just a matter or priorities and schedules inside WebCT, but I don't want to miss the target of opportunity. A number of people are getting very interested in HiveMind ... including the Scott Weaver (whose working on Jetspeed and Pluto). And others within the Jakarta community (who I'll let speak for themselves).

Based on some other posts going around, I think it's likely that HiveMind can become a Jakarta project, once the IP is fixed. The basic sentiment is that mandating that it move over to Avalon is effectively choosing sides, something Jakarta doesn't do. I'm still analyzing my own feelings about this ... intellectually, it may make sense to move the code into Avalon, assuming it will fit. But at another level, I'm very nervous about dilluting the code, and the focus, of HiveMind into Avalon, given that there are many fundamental differences in coding and design style. For me, joining Avalon and introducing HiveMind features into it represents a much higher level of effort. I'd prefer to present HiveMind as-is, and let the Avalon guys pull out what makes sense for Avalon (and the Spring guys, and the Pico guys).

It's kind of like fantasy sports cars. HiveMind is my own personal Ferrari. Avalon is, perhaps, a Diablo. At a high enough level, a Ferrari is interchangeble with a Diablo: four wheels, lots of power, no trunk space. Yet the subtle cues provided by a Ferrari (the sound of that engine, the shape of the hood, the color of the paint) excite me in a way that a Diablo does not ... and a Diablo fan may not be engaged by a Ferrari.

As I've said elsewhere, HiveMind represents my personal "hot points", the issues that are most important to me, reflecting my personal approach to coding and design style. Most of what's in HiveMind exists, in a different mix, inside JMX or Avalon or Keel, but in a different mix, that doesn't give off those same subtle cues that mean so much to me. For others developers, Avalon or Keel or Pico may provide the right set of cues, may be the most natural approach to getting work done. For myself, or those who have very similar sets of hot button issues, the others are just not quite enough, and HiveMind is the way to go.

Tuesday, November 18, 2003

Who will take charge of these people?

Herein being a factual account of the events of Monday, November 17th, 2003 involving myself, Howard M. Lewis Ship, my wife Suzanne, and Mr. Craig McClanahan in and about the "Star Trek: The Experience" attraction at the Hilton Hotel in Las Vegas, Nevada.

So here at ApacheCon, part of the point is to meet people you otherwise only know by an e-mail address or a bio on a web page. One of those people being my <sarchasm>bitter blood-rival and mortal enemy</sarchasm> Craig McClanahan, creator of Struts and lead developer of JSF. Craig was manning Sun's booth and passing out CD's with NetBeans on it and we chatted for a few minutes.

Flash forward to the reception at the end of the day, and Suzy and I found ourselves in line for the appetizer buffer with Craig again; later we ran into him again while talking with Justyna Horwat (another member of the JSF team) about mutual friend Greg Burd. Craig and I chatted a bit about HiveMind and such and then the three of us (including Suzanne) decided to head over to Star Trek: The Experience.

Perhaps we were inspired by the entertainers hired for the reception: a couple of Klingons, a Ferengi and a Borg. They stayed well in character, though I almost got a reaction out of the Borg when I asked him "what's it like to work for Microsoft?"

If you haven't heard of "The Experience", it's a hoot. It's a portion of the casino dressed up to look like a Star Trek set ... lattice-works of aluminum and chrome, lasers and neon. This leads back to sizable museum of Star Trek props. Overhead are large models (about the size of a good SUV) of Star Trek ships: Voyager, Enterprise-A and Enterprise-D and a Bird of Prey. Keeping in character, the props are labeled as historical artifacts, often attributed to entities such as the Klingon National Archives, Vulcan Science Academy and so forth.

The props, especially those from the classic series, provoke a strange reaction: a wave of nostalgia, and simultaneous incredulity at how cheap and cheezey the props actually are (even those props from the more recent series). Craig is a secret (or not so secret) fan, and was right in there with the references and trivia (his favorite race: Cardassians).

However, the heart of the attraction is the The Experience itself. A group of about thirty are led into an antechamber and lined up in front of doors leading to the motion ride ... except that's not what happens. Instead, the whole lot of us are beamed across time onto the Enterprise (a very fun effect involving wind, twinkling lights, darkness and an elevator). This is where this blog's title comes in: a crew member asks the question of the ride operators; from there, we're escorted down a a realistic Enterprise corridor to the bridge, find out about the ride's "plot", ride an out-of-control turbolift and, eventually, end up in a shuttle craft (the motion ride itself).

The ride is typical motion-ride fare, again enhanced by the theme and environment. It is also one of the more violent (in terms of large, jarring motion) rides of this type I've been on, especially if you sit towards the back.

All-in-all, quite a bit of fun, and quite lovingly put together. It uses seven or eight live actors (and some interactions with pre-recorded segments featuring Riker, LaForge and Picard), so it is somewhat involved. I kind of wished more small children had been in the mix, because all of us adults could only enjoy it somewhat sheepishly. At the end of the ride, after they try to shill Star Trek credit cards, we're let out into a gift shop (shades of Disney) and "Quark's Lounge". We bought a couple of drinks and deserts ... all with Star Trek punny names or references. I had the Tranya (if you visit The Experience, I hope you will relish it as much as I) but could as easily have ordered a "James Tea Kirk" or a "Wrap of Khan".

Entry to The Experience allows you to visit the museum and ride The Experience as many times as you like. We decided to finish checking out the prop displays (we had been interrupted earlier when we chose to ride The Experience) and it was nearly closing time for the museum. We ended up right at the entrance to the ride, just as they were ready to run it for the last time. And the three of us were the only ones in line.

I lobbied that we should just take off: it would just be the three of us, the ride is a bit hokey, and I felt guilty that they should have to run the whole thing just for us, instead of going home early. All three of us hemmed and hawed a bit and, suddenly, we realized that we did want to ride it again ... and then the chucking started. Which turned into giggling. Which led to uncontrolled bouts of laughter. And it was infectous ... several of the "crew members" couldn't keep a straight face when they saw the three of us. The line "Who will take charge of these people?" was just hilarious when the actors outnumbered the audience (and the audience was so incapcitated with laughter that a sleepy kitten could have taken charge of us). We were making jokes about the Klingons using nitrous oxide on us ... I also made a lame joke about hebephrenic particles that no one got (I had been drinking, in my defense).

The actors rotate through different roles, which is nice for them; I suspect working at The Experience would be totally unbearable otherwise. They tried to stick to their lines; for example, asking us to move forward into the bridge to make room (even though there was just the three of us). The second trip let us see a lot of detail you can miss on a single pass: for example, the comm panels at the back of the bridge are live ... initially they show scanner data about the three Klingon ships, then switch to a bio of the Klingon captain when he hails the Enterprise. They're also helpful to lean on when you can stand any more from laughing too hard (just don't let the bridge crew see you).

By the end of our second pass through The Experience, I was having trouble breathing from laughing too hard for too long. My throat was raw and I felt physically beat up. I can't promise you that you'll enjoy it nearly as much as we did (some experiences simply can't be recreated) but it might just be worth a visit, if you are a Trek fan (and can stomach the $30 entrance fee).

Monday, November 17, 2003

ApacheCon: XMLBeans

I'm sitting in the middle of a very interesting session on XMLBeans, a very powerful ASF project (currently in incubation). Very interesting; they really grok the XML Schema thing, but are very pragmatic. This session left me with a really good feeling about this framework, as opposed to everything I've seen about JAXB, which only seemed good in theory.

It keep both a (cursor-based) DOM model, and a interface-based Java model of the same document, which is kept synchronized. Don't quite know how to explain this, but a document looks like a standard DOM, but can also be treated as a tree of strongly-typed Java objects (represented as interfaces); the Java interfaces are created from an XML schema document (using a command line tool).

There seems to be a lot of magic in there; when you create new nodes on the Java tree it usually can figure out, automatically, where to put the nodes on the DOM tree ... based on the XML schema.

They seem to follow a good, pragmatic split in their approach to the API; convienience methods for doing things the way you usually want to, and formal methods to give you total control.

This has definately bumped up on my list of technologies to check out. I can't imagine that it's as efficient as SAX and Digester, and it's based on XML schema (for good or ill).

It appears to behave reasonably well, even when the input document is damaged. Validation is optional. They've also done some difficult, interesting things to maximize speed ... for instance, the XML schema is "compiled" into a binary metadata format (based on the Java class format). So you get the benefits of Schema without the cost of parsing those giant schema documents. It's 100% compatible with 100% of W3C schema. They also claim performance about the same as Xerces and much faster than JAXB R1.

Howard Lewis Ship 1.1: Now with ASF Membership!

A side note ... just found out that as part of a mass election here at ApacheCon, the ASF membership voted in 13 (?) new members, including myself. Talking with some involved parties, I learned that there was a good deal of concern ove the current HiveMind IP fiasco (see lots of notes in this blog). Apparently, in this one area, I was branded "incompetent but non-hostile" (or something to that effect).

How does ASF membership affect me? I don't know ... I'm now privy to more ASF discussions (how to keep up?) and, theoretically, have more of a say in setting ASF direction. What direction is that? I have to start figuring out.

ApacheCon presentation post-mortem

Just thought I'd say that the feedback on the presentation was very, very good. I had some time troubles (I've never done any kind of presentation with a fixed time limit before), and I had minor slide issues (the HTML wasn't very readable for the back half of the room) ... but people were involved, interested, taking notes and very receptive. And I forgot to distribute business cards! Why am I here again?

Of course, I can talk about Tapestry for days at a time, not just the minutes I had. And my rates are very reasonable.

Next step is to do something I should have done even before arriving in Vegas ... setting up a Tapestry BOF and/or "guru" sessions. That'll give me a chance to show of Spindle ... which due to a mixup on my part, barely got mentioned during the session (somehow I put the Spindle slide after Q&A, not before, so I never displayed it).

At this moment, I'm sitting on the floor outside Apollo 1 (the venue for the "State of Geronimo" session). It's packed and we arrived late. Andrew Oliver is next to me, IRCing about Tapestry. No, Tapestry is not Cocoon, thank you very much. Andrew was worred that if we walked in together, I would be "guilty" by association (he considers himself the "black sheep" of the ASF community) ... instead, we're not even inside, so I guess I'm in the clear.

There's probably more computing power walking around here in laptop and PDA form than existed twenty years ago.

Sunday, November 16, 2003

On-site at ApacheCon

We'll, I'm sitting in the lounge at ApacheCon working on The Book.

I did a run through of my presentation, and it comes in at about 40 minutes --- that's as long as I can allow for, since it leaves only ten minutes for Q&A. I think I'll perhaps trim just a little, and maybe have time to at least show Spindle in action. I snuck a Spindle slide into the presentation.

Which brings up lies. My bio says that I'm a member of the Jakarta PMC but it turns out that I'm not ... my "election" was not run properly or recorded properly. I believe its being redone.

My session synopsis claims I'll be creating a web application live on stage using Spindle, which isn't accurate either ... at best, I'll have a moment to show off a couple of Spindle features.

Updates tomorrow ... do I knock their socks off, or score a yawn?

Don't tell anyone, but they comp'ed Suzy (my wife) a pass. Apparently, they want a few more women in the audience. Not that she's technical (except perhaps by osmosis). Her badge says "Tapestry Tart" (off of someone's blog entry I can't seem to find). She was excited that she ran into some random ApacheCon attendee who knew about Tapestry, using words like "next big thing". Then she lost a chunk of cash at the blackjack tables.

I'm not sure who'se supposed to be here that I should know. Drew Davidson had mentioned possibly getting together, and I forget if Erik Hatcher was coming or not. Andrew Oliver is around here somewhere I suspect; I'll be attending his Poi session.

Suzy and I are going to catch Zumanity (the newest Circe Du Soleil show) Tuesday night. Advertising works ... hadn't even heard about this show until we were waiting for our luggage at the airport. Expensive, but probably worth it; certainly more so than any of the other pandering t-and-a shows in this bizarre town. Vegas is just plain wierd.

Thursday, November 13, 2003

The Occasional Blogger chimes in

Glen Stampoultzis has posted a few comments about Tapestry in his Blog, The Occasional Blogger.

I do feel everyone's pain with respect to the documentation not being ready yet. I'm just about over the hump with the work for Manning (not to mention the trip to ApacheCon), and light is visible at the end of the tunnel.

This was a trap I dilligently worked to avoid with HiveMind, where I've worked very hard to have useful documentation kept up-to date with the actual code.

Temporary Location for HiveMind Docs

I've put up a temporary, limited version of the HiveMind documentation at:

This doesn't include JavaDoc, the clover code coverage report (> 90%, btw) or source xref ... but it should be enough to whet peoples curiosity until the full site can go live again (once WebCT makes the software grant).

Wednesday, November 12, 2003

ServiceSide picks up the HiveMind Proposal

This was pretty unexpected. The ServerSide's Dion posted this article: HiveMind Proposal to Join Apache Jakarta Project

Seriously; I'm perplexed by the level of interest in HiveMind. Don't get me wrong ... I like to have my ego stroked and all, and I think HiveMind will become a huge success, wherever it is that it ends up (Avalon, Jakarta, or whatever).

Still, people must be hungry for something, they are looking for the way. We want our objects back, we're sick of feeding our little chunks of code into the great J2EE appserver machine.

Once everything gets straightened out with HiveMind I will start to market it ... it's just that it is somehow marketting itself already. The only publicizing I've done is in this blog, and in that ServerSide discussion of Picocontainer (and, to be sure, I did not bring it up myself!).

Well, we'll see where we are when we get back from ApacheCon!

Tuesday, November 11, 2003

Proposal for HiveMind Project

(0) Rationale

HiveMind is a simple framework for creating pluggable, configurable, reusable services.

Simple: HiveMind is a way to create a network of services in terms of Java interfaces and classes; it cherry picks the most useful ideas from Service Oriented Architectures such as J2EE, JMX and SOAP, but removes the aspects that are typically overkill for most applications, such as service remoteability and language neutrality. HiveMind creates a natural network of related services and configuration data, all operating within a single JVM.

Pluggable: HiveMind enforces a complete separation of service definition and implementation. This is manifested by a division of services into an interface definition and a service implementation as well as a split between defining a service (as part of a HiveMind module) and providing the implementation of that service (potentially, in a different module).

Configurable: HiveMind integrates a service oriented architecture to a sophisticated configuration architecture; the configuration architecture is adapted from the Eclipse plug-in model, wherein modules may define configuration extension points and multiple modules may provide contributions to those extension points.

Reusable: HiveMind is a framework and container, but not an application. The HiveMind framework and the services it provides may be easily combined with application-specific services and configurations for use in disparate applications.

The API for HiveMind allows thread-safe, easy access to services and configurations with a minimal amount of code. The value-add for HiveMind is not just runtime flexibility: it is overall developer productivity. HiveMind systems will entail less code; key functionality that is frequently an after-thought, such as parsing of XML configuration files, logging of method invocations, and lazy creation of services, is handled by the HiveMind framework in a consistent, robust, and well-documented manner.

HiveMind fits into an area that partially overlaps the Apache Avalon project, with significant differences. HiveMind's concept of a distributed configuration is unique among the available service microkernels (Avalon, Keel, Spring, Picocontainer, etc.). Avalon is firmly rooted in a type-1 inversion of control pattern (whereby services must explicitly, in code, resolve dependencies between each other using a lookup pattern similar to JNDI). HiveMind uses a mix of type-2 and type-3 IoC, whereby the framework (acting as container) creates connections between services by setting properties of the services (type-2) or making use of particular constructors for the services (type-3).

HiveMind represents a generous donation of code to the ASF by WebCT (http://www.webct.com). HiveMind originated from internal requirements for a flexible, loosely-coupled configuration management and services framework for WebCT's industry-leading flagship enterprise e-learning product, Vista. Several individuals in WebCT's research and development team in addition to Mr. Howard Lewis Ship contributed to the requirements and concepts behind HiveMind's current set of functionality including Martin Bayly, Diane Bennett, Bill Bilic, Michael Kerr, Prashant Nayak, Bill Richard and Ajay Sharda. HiveMind is already in use as a significant part of Vista.

(1) Scope of the package

The package shall entail a core framework JAR (containing essential classes and services), a standard library JAR (containing generically useful services), along with ancillary artifacts such as Maven plug-ins and, of course, documentation, all distributed under the Apache Software License.

(1.1) Interaction with other packages

HiveMind has dependencies on several standard commons packages, including: commons-lang, commons-beanutils, commons-collections and commons-logging.

HiveMind makes use of the Javassist bytecode generation library, which is available under the MPL (Mozilla public license).

(2) Identify the initial source for the package

The initial code base has been developed by Howard M. Lewis Ship within the Jakarta Commons incubator.


(2.1) Identify the base name for the package


Note: the current code base reflects an alternate package name, org.apache.commons.hivemind. Subsequent research has shown that HiveMind is not a suitable candidate for the Jakarta Commons. The existing code base will be migrated to the new package during the transition out of the sandbox.

(2.2) Identify the coding conventions for this package

The code follows a modified version of Sun's standard coding conventions, with the following stylistic changes:

  • instance variables are prefixed with an underscore
  • a newline is inserted before all braces

(3) Identify any Jakarta resources to be created

(3.1) mailing lists

hivemind-user@jakarta.apache.org -- User discussions
hivemind-dev@jakarta.apache.org -- Developer discussions and CVS update notifications

(3.2) CVS repositories

The package will use a root branch of the hivemind CVS repository (to be created).

(3.3) Bugzilla

The package should be listed as top level component, "HiveMind".

(4) Identify the initial set of committers to be listed in the Status File.

Howard M. Lewis Ship <hlship@apache.org>
Prashant Nayak <prashant.nayak@webct.com>
Martin Bayly <martin.bayly@webct.com>
Christian Essl <christianessl@yahoo.de>
Harish Krishnaswamy <hkrishnaswamy@comcast.net>
Knut Wannheden <knut.wannheden@paranor.ch>

This list represents the most active HiveMind participants within WebCT and on the Jakarta Commons Developer mailing list. Notably, Mr.s Essl, Krishnaswamy and Wannheden, among others, have already been actively mentoring other interested users on the mailing list in how to use HiveMind as well as contributing design ideas and patches to the framework itself.

Get this in your toolbox

I mostly try to stick to open-source other otherwise free tools ... I'm just cheap perhaps. Or I want to be able to look under the covers. But every once and a whilie, there's a reasonably priced tool so useful you have to get it.

For about $20, Blazing Tools Software's Instance Source is completly worth it.

It's a plugin for Internet Explorer that provides a real view-source window that is so chock-full of features I already can't live without it. Here's a few:

  • Colorized source as sent from the server
  • Colorized source as currently in browser (i.e., accounting for document.write(), etc.)
  • Frame aware -- different frames / framesets in different tabs
  • Element under cursor -- point the cursor at some text and see the HTML for it (yow!)
  • Selected text -- select some text, see HTML for selection

Basically, this is what I've always needed in my browser to help me debug HTML ... no more having to save HTML to the desktop to disect them, no more guessing what the HTML really looks like, no more frame pain. This is just too cool.

Monday, November 10, 2003

HiveMind Proposal on Tuesday

The proposal for HiveMind should be public tomorrow, Tuesday Nov 11th 2003. It'll be coming out of WebCT, from good buddy Prashant. This is a relief; I was starting to think in terms of total rewrite ... and even that wouldn't necessarily eliminate the intellectual property issues. This will. It's highly likely that HiveMind will find a home somewhere in Jakarta; the worst possible case is that WebCT will sign the grant and HiveMind will end up as a Tapestry sub-project. I'd actually prefer HiveMind to end up on its own, seperate from Tapestry.

Friday, November 07, 2003

Another great Tapestry quote

This one is from Jason Cox and showed up in the Tapestry user mailing list with the title Nothing short of beautiful :
At 11am this morning, the first version of our Web Portal prototype was completed. I'm sad to say that this is an Intranet project and so won't be usable for marketing Tapestry and for confidentiality reasons I cannot even name the company at this time. However, it is a very large company with an international presence, and the portal I am developing is going to be used mostly by managers and up to senior level executives.

I just want to give a big thanks to the members of this list, which I have been reading constantly.

On overview, though I doubt they will be examples of Tapestry best practices, the templates, specifications, and java code has all been very elegant. Not because I am such a stellar developer, but because Tapestry lends itself so well to MVC design. Once I got it in my head to fully implement all aspects of the portal in Tapestry, it took me about 3 days. There was far less effort than I've had using Struts or the custom framework we had at my last job.

The only low point is that I am not convinced that I particularly like how Tapestry handles Javascript. It just slams it all into the page. No cached references to .js files like you typically have in standard HTML and even JSP's.

On the other hand, I was able to perform some neat little tricks that would have been slightly more cumbersome using traditonal JSP development, including dynamic comments that only appear when certain conditions occur. Realistically, they're just there for reference if someone messes with the HTML template, but I didn't want them to actually appear on the page, so I nested them inside the error blocks.

Implementing my own Border web component was incredibly elegant, and if I would just take the time to properly 'HTMLize' it I can actually hand it off to a web designer in the future. Though I am still looking at dynamic menu generation and some other features that will make it less static. Not a high priority right now. What's the most amazing is that it actually works seamlessly with the ridiculously obtuse javascript files that most of our Intranet sites use to build their header and footer.

Now the next challenge is implementing a content management system that will also use Tapestry as its outward facing framework.

Tapestry momentum building

It's pretty clear that the momentum behind Tapestry is building; I just can't keep up with the flow of mail on the user list. Fortunately, the other heavy hitters (MindBridge, David, Erik and Harish, primarily) are keeping up. The first chapter in the book is being proofed right now; I suspect it will actually be available via MEAP next week-ish and the book will be in stores January or February; not so bad, considering the amount of last-minute rewrites. I suspect Tapestry will explode once that's available.

I know the pain, currently, of doing Struts development. I did some at WebCT, but I'm primarily doing UI work for NLG. It Struts, 1.1, Tiles, Tomcat, JBoss. Session facade layer to entity EJBs. Heavy use of XDoclet for the EJBs, for the Strut's Actions and servlet tags, etc.

However, any time I go to do anything, it feels like I'm having to juggle through so many different files. Some obvious things, such as having one Action forward to another Action (the second Action exists to get data from the facade and push into request attributes for use by the JSP) were just painful; I ended up creating a global forward for that since I had several different Actions that had to reference the data-loading Action.

Then there's the problem of figuring out how to reference actions from other actions, or from JSPs. Leading slash or no leading slash? ".do" suffix or no suffix? Global forward or local forward? Seems like this should be easy to do, but I had to do lots of fiddling around.

I don't like Dynaforms but the team standardized on them. You are very limited in terms of what data can be stored in a dynamform. Seems like it should automatically be able to convert to any types that takes a single string constructor parameter, but no ... very much hard coded (I peeked at the code).

Feedback: One of the four "pillars" of Tapestry (with Simplicity, Efficiency and Consistency). Boy do I miss Tapestry exception reporting! When things go wrong, I have no clue ... I just always have to run JBoss with the debugger and always have to set lots of break points. I ended up adding a custom JSP tag to our pages to display servlet request parameters and attributes and session attributes (a limited form of what you see in the Workbench) and that helps, a little.

Erik Hatcher couldn't believe I'd even consider a job developing Struts ... but, hey, I'm building up a war chest here while I start lining up the serious Tapestry opportunities.

HiveMind in transition

HiveMind is causing a bit of commotion in ASF-land ... the whole issue of who owns the code is causing grief. In the short term, the Jakarta PMC has taken down the HiveMind CVS repository and the HiveMind site is probably next. Can't blame them (and I is them ... I'm a Jakarta PMC member myself); the code shouldn't be there, in light of there being any ambiguity about code ownership.

So, HiveMind will be on a short hiatus until such time as WebCT grants the code and submits the proposal.

The great thing about being an independent consultant is that these issues go away; especially if I'm contracted to teach Tapestry courses, the moment I'm off the clock all ambiguity about IP ownership goes away. In the future, I'll make sure that any work I do has special exceptions for spinning off code into open-source land.

Saturday, November 01, 2003

HiveMind proposal almost there

A proposal to promote HiveMind up to a top-level Jakarta project is just about ready. A draft of the proposal has been bouncing back and forth between me and WebCT.

I did some reading and thinking on the plane to and from Virginia; when I get a chance, I have some good ideas about how to integrate HiveMind and JMX, to support things like:

  • Representing the MBeanServer instance as a HiveMind service
  • Representing standard MBeans as HiveMind services (using the MBean interface as the service interface)
  • Creating a performance monitoring interceptor that records invocation counts and times in an MBean
We'll see what other ideas come forth from this.

I've also been thinking of extending some of the event notifications inside HiveMind (thread cleanup and registry shutdown, specifically) to provide "levels" used to provide a coarse level of invocation order. This would be an intermediate step towards real dependency tracking. For instance, during a registry shutdown, I'd like the following order to apply:

  • Disable all configuration item list proxies and discard configuration items
  • Disable all service proxies
  • Cleanup all services
Since objects are registered in the ShutdownCoordinatory using just the RegistryShutdownListener interface, there's no way to order the notifications without additional information. I see the interface for ShutdownCoordinator changing to: public void addRegistryShutdownListener(int level, RegistryShutdownListener l);

Something similar would be good for thread cleanup; I'd like to temporarily disable the threaded and pooled service proxies, invoke the cleanup method on the service implementations, then re-enable the proxies.

Shameless self-promotion

Now that I'm in the world of independent consulting, I can't pass up any chance for good, shameless self promotion, such as this recent posting to the Tapestry user mailing list:
First, apologies are in order to the tapestry-user list community.... we kidnapped Howard Lewis-Ship for a couple of days for onsite training taking him offline from answering questions on the list - please forgive us :) We, in the Darden Solutions Group within Darden Graduate Business School at the University of Virginia, decided to switch from Struts to Tapestry a few months ago for our next project. Since then we've been an early previewer of Howard's wonderful Tapestry in Action book, and working our way through our first passes at implementing a Tapestry application. We had built enough to know what we didn't know and where we wanted a quick boost of insight.

Our mission is to develop a suite of applications in a short period of time. We brought Howard in for two days of onsite training to give us a kick start in the right direction and ask him detailed questions about implementation issues we've encountered. It started as a semi-formal presentation to introduce us to Tapestry and quickly took a custom tangent into the areas we were most interested in, like creating custom components, how parameters / properties / bindings work, how do to deal with form validation, and much more. Howard was adept at fielding our random questions. At no time did we stump him. He even very quickly picked up on our business use cases and even offered suggestions in that realm, unrelated to the gory Tapestry details we brought him in for.

Our team greatly appreciated having Howard's time and expertise. We have easily saved weeks of time learning this stuff the hard way, and given everyone a very quick boost into how Tapestry works and how to use it effectively and properly.

Everyone here knows that Tapestry can have a steeper learning curve than other less powerful frameworks. If your project has the same delivery timeframe as ours ("yesterday"), yet you want to do it right, I suggest you consider doing what we did and bring Mr. Lewis-Ship in for training.


That's Erik Hatcher, "the Ant guy", frequent speaker at No Fluff Just Stuff symposiums and all around great, well-connected, finger-on-the-pulse kind of guy.

I had a terrific time down in Virginia, it's a real rush to spend two whole days thinking on your feet. Everyone on the team was very complimentary about the presentation ... which just goes to show you how different things look from opposite sides of the podium. I was concerned that I was mumbling incoherently like an escaped mental patient, but that's not how I came off :-). It was very rewarding to see the connections get made inside people's heads. Of course, the first major questions hit all of Tapestry's dirty laundry: the awkwardness of component parameter directions, for instance. But part of open-source is that your dirty laundry it out for all to see, and that no project is ever finished.

We had a really fun time, with discussions of Tapestry and other technologies rolling out of the class room and out over lunch and dinner. It was a quick trip, but we made it out around Charlottesville thursday night, which has a very European-styled "mall" ... an open, wide, cobblestone walkway that extends several blocks and is filled with book stores, coffee bars and the like. Refreshingly, it doesn't seem gap-ified (didn't see a Starbucks or any other national chain). We also swung by the Gravity Lounge, a cool mix of Internet cafe, book store, and live performance stage. It also has an Art-O-Mat, from which I purchased some art for my wife--a wooden block with a kind of beat-style/kitch black cat. A good draw from the Art-O-Mat, since you never know what you'll get ... my wife just loves cats!

So far, that's two Tapestry consulting gigs this year ... one was way back in January for a company out in Los Angeles. Not so bad since I haven't been seeking these out, and have had to work around full-time jobs to accept the gigs. I haven't really started marketing myself since I pretty much have a full-time gig going for the next few months at NLG, but as that looks to wind down (probably late January) I can start trying to schedule out engagements for 2004.

Friday, October 24, 2003

Composition Trumps Inheritance

Just a random thought ... composition is more powerful than inheritance because, using composition, you always work in terms of a single, outward facing, public interface.

With inheritance, you have the public interface and the "protected" interface: the constructors and protected members inherited from a super-class. This creates a greater "surface area" that you have to adapt to.

With composition, you can extend an existing service (yes, this is about HiveMind because Tapestry is so inheritance based) by wrapping around it. You don't subclass to the other service, but you do delegate to it.

In some ways, inheritance is just a awkward and limited kind of composition. If class A inherits from class B, then class A implements the same interface as class B and delegates all methods to class B that are not overridden by class A. super is just explicitly delegating to B, and the rest is syntatic sugar for automatically delegating to B. Well, pretty much (this concept breaks down once you enter the idea of a super class invoking methods overriden by a subclass, not to mention reflection).

I think UML has recognized this for a long time; I've always had a hard time creating sequence diagrams for classes that make use of inheritance (and recursion) because there isn't a really good way of representing those kinds of invocations. With no inhertiance buts lots of delegation, such sequence diagrams are completely reasonable.

Working for the competition?

Just noticed a new open-source MVC web framework, Jucas. Interestingly, its built on top of ... HiveMind.

Last day at WebCT

I'm finishing up my last day at WebCT. I start, as a sub-contractor, at NLG on monday(!). No rest for the weary. It's awkward and a bit sad to be moving on, but I'm looking forward to a few months out, when (hopefully) I largely support myself with Tapestry and HiveMind. I've spent a lot of time in my career "batting cleanup", coming in late to projects and products to bring order out of chaos. I was refactoring (in PL/1) before I even knew such a term existed (our group coined the word "Howardizing") ... not to mention creating unit tests and all that ... everything old is new again.

I'm also looking forward to concentrating on fewer things; I've felt very distracted at WebCT ever since I got back from sabbatical, because I came back too early, and have had the book (and shopping for a house, and moving, and now the job switch) eating at me.

Definition of Service Oriented Architecture?

Looking around online, most of the definitions of SOA that I find are inextricably tied to language neutrality and remoteability. Under those guidelines, SOA is restricted to solutions like SOAP.

I find that too restricting. The goals of SOA (as I define it) are to enhance reuse and manage complexity. A key way to achieve both of these is to remove statefulness. When objects don't have individual state, they are easier to combine together ... because state implies that there is an order of operations that gets in the way of easily combining services together to form more powerful services.

In HiveMind, services may have state ... but there's a cost on any application that makes use of internal state.

Remoteability is another holy grail that costs more than it gives back. EJB's local object support is an admission that remoteability of services is the exception, rather than the rule, and that a container is useful even if objects are shared within a single JVM.

When a service needs to be remote ... go for it! XML over HTTP or even full-blown SOAP is appropriate. But once the gears start turning inside your application, keep it all in the same JVM. A common maxim I've heard and used is "XML at the edges"--meaning that XML is an inefficient format (especially in the flexible, dynamic Java environment) to represent data in within your application, but is a good choice when presenting an external face to your application. I'd extend that to "Remote access at the edges" as well.

Thursday, October 23, 2003

Services vs. Configurations

One stumbling block with HiveMind which keeps coming up in various conversations is a pattern like this:
  • I want to define my foos in a HiveMind configuration
  • Some of the foos will be defined at runtime (from the database)
  • Some attributes of the foos will be settable by users at runtime
  • How do I update my HiveMind decriptors with these changes?

You don't. HiveMind is used to describe the static configuration of your application. This is equivalent to saying "How do I change the default value of a final static field in one of my classes (and store it back into a JAR (and distribute the change around the cluster))."

What your HiveMind configuration will store is the default configuration of foos until a) they are updated by a user and/or b) new foos are discovered at runtime.

What you need is a service that can answer the question: "what is the configuration of my foos?" This service will have access to the configuration ... but also to the database for updates and overrides. It will integrate that information drawn from the HiveMind configuration with overrides from the database and provide a consistent view. If you are smart, you'll break it apart into a master service with several additional services (for the discovery piece, for the database override piece).

I can't think of a scenario where external code (code that is not a HiveMind service) should directly access a configuration extension point. It is allowed, the API is there ... but in all reasonable cases, putting a service between client code and the configuration data is the best bet ... especially for applications that expect to grow in the direction of greater dynamicism.

Wednesday, October 22, 2003

HiveMind Presentation report

So, last night was the HiveMind presentation. It went rather well on the whole; people seemed interested and stayed for the whole (long) Q&A. I'd forgotten how exhausting doing a live talk can be ... plus it was hot in the room.

Major stumbling blocks:

  • The benefits of a service-oriented architecture are intuitively obvious to me but I had trouble explaining them off the cuff. SOAs promote testability and reuse, by allowing easy decomposition of complex problems into simpler building blocks. SOAs make you think about state ... where you need it, where you don't.
  • XML and XML rules. This prompted a distracting side track about the usefulness of XML and Dan Jacobs was really questioning the concept of having a stack-oriented scripting language (regardless of how focused it is) as part of HiveMind. People were also looking for alternates to XML.
  • There were some concepts that needed to be fleshed out a bit more; especially the idea of using a factory to construct new services (which came up again in Q&A).
  • A few key ideas that were out of scope for the presentation (such as the threaded service model) should have not been mentioned at all.
  • Needed to set the expectations better ... people kept coming up with ideas (remote services!) that are, basically, just J2EE in disguise.

Still, the Q&A was lively and people seemed interested. It revved me up for my upcoming ApacheCon appearance (which I need to practice for!). Life is just too busy.

Friday, October 10, 2003

HiveMind presentation at WebTech Users Group

I'll be giving a presentation about HiveMind at the Boston-area WebTech user's group on Tuesday, Oct. 21st. Meeting starts a 7pm and will be a bit over an hour, plus time for Q&A.

The most common question at these events is: "Is this presentation online?". The answer is: yes (follow this link).

Monday, October 06, 2003

Thinking about coding

Do you think about coding? A lot? I do.

My own personal style has shifted a bit recently; I've adopted a few minor tricks, like naming instance variables with a leading underscore and I've stopped naming interfaces with a leading 'I'.

More than that, I've really been trying to adopt an every more spare style in my interfaces. I'm looking for the Perfection and Simplicity that Ken Arnold talks about. I often visualize my object interfaces in terms of a suit of armor, which should have the minimal number of openings. Ken talks about the surface area of an interface, that can expand when there's a lot of inheritance involved ... or my own personal issue: convienience methods.

When the Collections framework first came out, I was aghast that useful things (like sorting) were in a static class. Ugh! Surely that's no OO!

I've since come to understand the usefulness of that. My early OO was in Objective-C. Between categories (an early forerunner of aspects) and a lack of garbage collection, you could end up twisting yourself up. Better to put a method onto an existing class than to have to create a whole new class!

But nowadays, in terms of keeping my surface area small, I seek out places where I can replace convienience methods (often protected methods in some base class) with a static utility method.

I look at methods now (I'm doing this somewhat in Tapestry) and say ... hey, change a few variable references to accessor methods and you could pull this entire method right out of this class. For example:

public class AbstractLineItem  implements LineItem
   private int _quantity;
   private double _cost;
   protected double computeTotal()
    return _quantity * _cost;
could easily be replaced by:
public class LineItemUtils
  public static double computeTotal(LineItem item)
    return item.getQuantity() * item.getCost();

Note the big advantage: its a public method expressed in terms of the interface not the class. Suddenly, the logic here is so much simpler and more testable. In HiveMind terms, it may not be a static class, but instead of service used to calcuate the total ... same difference.

This does an end run around problems such as multiple-inheritance. Too often, I want MI because there are convienience methods in base class A and base class B that I want access to. Moving those convienience methods out into a static class or service removes the need. Suddenly, those base classes aren't so useful either; I'm more likely to start with java.lang.Object and just implement the right interfaces.

Nowadays, I look at the Tapestry code base and think ... woops; a bit too much of Tapestry is inheritance-oriented rather than interface-oriented. A page or component in Tapestry fills different concerns (model, view and controller) at different times and perhaps the mythical Tapestry 4.0 will address that (Tapestry 4.0 is my personal code word for the total , non-backwards compatible rewrite of Tapestry that I would only think about under the guise of a JSR, so don't hold your breath). I'd love to seperate out the IComponent interface into seperate objects and interfaces to represent property storage, component hierarchy, rendering and request handling.

In the meantime, I'm finishing up at WebCT. HiveMind is going gangbusters (just added a pooled service model). I spent the weekend doing major revisions on chapter 1 of The Book and feel I've nailed it. ApacheCon is coming up, as is my new and changed life as a happy-go-lucky independent consultant. Now if I could just get the Vista application deploying into WebLogic 8.1 properly, I'd be a fully happy camper.

Friday, October 03, 2003

ThreadLocal and thread pooling

Anthony Eden's post, ThreadLocal and thread pooling, illustrates one of my HiveMind goals. His post was about the importance of clearing out ThreadLocal objects at the end of a (servlet) request.

HiveMind includes a service for storing thread local information in a generic fashion ... effectively, a ThreadLocal Map of data. It all hooks into the common ThreadEventNotifier service, so that the thread local data can be discarded at the end of the request in a simple, uniform manner.

I constantly say that a good framework should make the easiest approach the correct approach. This is another example ... in this scenario, the correct approach is to discard thread local data at the end of the request and it is easier to use the ThreadLocalStorage service than to roll your own ThreadLocal object ... and certainly easier to let HiveMind handle the end-of-request cleanup than to do so in your own code.

Thursday, October 02, 2003

Tapestry in Action

Manning has started to market my book on Tapestry, "Tapestry in Action". This is great stuff.

A glorious new world of opportunity!

Today. Is. The. Day.

After months and months of thinking about it, even agonizing about it, today is the day that I resigned from WebCT.

This is good; this means I will be available for Tapestry (and HiveMind) consulting! That's the whole point, today is the day I launch a consulting career.

Even better, in this questionable consulting market, I have pretty much full time work as a sub-contractor through my old boss, Tim Dion (now CTO of Riverton). How could anyone turn this down ... more freedom, a chance to pursue the dream (full time Tapestry!) and, at the same time, steady work. Still remains to be seen exactly how I'll balance Tapestry consulting with my responsibilites to Riverton's clients' ... but I'm sure we'll work something rational out.

Shortly (once all these details are worked out) I can start actually drumming up business.

Wednesday, October 01, 2003

Adding some lifecycle support to HiveMind

Just checked in some changes to support some basic lifecycle on a HiveMind registry and its services and configurations.

There is now a Registry.shutdown() method that shuts down the registry. All proxies will stop working (service methods throw ApplicationRuntimeException), same with all configurations. Core service implementations can now implement RegistryShutdownListener to be alerted when the registry shuts down.

I like the idea that proxies stop working when the registry shuts down so much that I changed the deferred service to always return the proxy. That, with C. Essl's inner-proxy solution (to avoid synchronization once the service is created) is very cool. No guesswork, no dangling anything ... shutdown the Registry and kill all the services (at least, all the non-singleton services).

It's all good baby ... going to add more involved lifecycle for threaded services (so they can tell when they are discarded) and, perhaps, add a pooled service model (with notifications when the service is activated and passivated).

More super-secret news coming soon!

Friday, September 19, 2003

Excellent discourse on WebObjects and Tapestry

Drew Davidson, (co-?)creator of OGNL, recently posted this message about WebObjects to the Tapestry user mailing list:
I went to WWDC 2001 full of excitement about WO, since at that time they were just getting the pure Java 5.0 version out the door. I had talked with them previously about how they were going to transition from ObjC to Java and was concerned about the API.

Their main focus was to make it API compatible with ObjC - something I thought was ridiculous. From their point of view, I guess that their customers were clamoring for this because it would make transitioning their applications from ObjC to Java easier; most of the WO work we did at Running Start (where I worked with Erik Hatcher) during 2000 involved using the ObjC-Java bridge and the Java API to WO that was provided. So there was an existing Java API for WO.

However, this decision had long-term consequences to users. The API as presented by the ObjC-Java bridge was an Objective-C API - no two ways about it. It was not Java-like in the least. The API used internal collections (NSArray, NSDictionary, etc.) instead of the soon-to-be-standard Java Collections in java.util. The method names were ObjC-like and not JavaBeans patterns where that made sense. ObjC concepts like categories and inherited class methods that Java does not support were rough translations and sometimes the disconnect was obvious.

Short term this is a benefit as developers can more easily port their code.

Long term, however, this means that you must live in the Apple WO world instead of the bigger Java world. Adaptors must be written when interaction is required between APIs and standard tools for doing JavaBeans stuff just won't fit without making custom BeanInfo objects for the necessary objects.

The WO API itself was written very long ago (it started back at NeXT back in 1994, for crying out loud!) and it was written with the perspective that you build a WO *application* that acts as an application server. It does not take into account the notion of Application Servers and therefore solves the entire Application Server role from soup to nuts. At the time it was written this was sensible.

But the world has moved on. Servlets are now the standard way to deploy Java web applications. Apple has made efforts to integrate WO with the Servlets, and even integrate with JSP, but IMNSHO they have not really made it feasible to develop and deploy this way. The API is just too elderly and they needed to have ripped a new page off the notebook, taken the concepts and started over. The concepts are still very valid. Nothing else (up until Tapestry) had attempted to really componentize the web, and that is very much to NeXT's (then Apple's) credit. When Howard started over with the basic assumptions of Application Server, Servlets, etc. and applied the excellent concepts from WO he created Tapestry. Tapestry is, to my mind, what WO 5.0 should have been. The failure was due to Apple's doing what the customer *wanted* and not what they *needed*. I'm sure that Apple's rationale for doing what they did was to support their customer base and not lose those customers. But look at their market now - they are niche. But the 3 WO customers left probably had a very easy time porting their 4.5 applications to 5.0.

BTW I mentioned the collections problem at WWDC (in one of the sessions, in front of everyone) and was brushed off by Ernie Prabakhar (the manager in charge of WO at the time) by him claiming that their collections were not functionaly compatible (mainly due to null handling); I told him that I had integrated the two without conflict with some facades that wrapped the NS stuff to Java Collection stuff. It worked, but was wasteful because of all of the adaptors hanging around and the constant awareness you had to have at integration points. They really had no interest in solving this problem, and it was clear they just wanted me to shut up and go away because I was expressing concern and critique instead of praise and adulation.

You, Jamie, are correct in your assesment that WO has stopped evolving. It has become a niche solution wherein you have to buy into the entire Apple web development package, rather than being able to use the tools you want and exclude those you don't. The only company that can get away with this kind of behaviour and still keep market share is Microsoft.

- Drew

Wednesday, September 17, 2003

Components Vs. MDA? (vs. J2EE)

Been reading some of the blog by Rickard on Model Driven Architecture and the resulting maelstrom at TheServerSide.

I have a pretty deeply ingrained abhorance of code generation. Note bytecode enhancement at runtime; that I'm really into. There's no baggage with that, no chance for the build process to screw it up, no need for a developer to go modify the output, no chance for bad merges or lost changes.

I've never like code generation; my early exposure to NeXT and InterfaceBuilder really showed off why it wasn't necessary for GUIs and I still don't think its needed for enterprise apps.

Code generation is a last resort that should only be used when real techniques fail. For me, real techniques are combinations of components and service-oriented architectures .. the kind of stuff you see in Tapestry and HiveMind.

In MDA, you define your system in terms of objects and relationships ... but then you spew out code representing that system that isn't isomorphic to the system you defined. If it was truly isomophic, you would be able to reconsititute your original model from it, but that's simply not reasonable.

In my vision of a proper MDA, the objects and relationships in the model would be directly expressed by matching objects and relationships in the runtime image. In Tapestry terms ... application extensions and helper beans; in HiveMind terms, services and implementations. In both cases, I would expect that the MDA would provide a library of the components (and service implementations) referenced in the model and the runtime image.

Tapestry has room for meta properties in all the key elements; it is quite reasonable for a tool to plug in meta data into pages, components and helper beans to support seamless round-trip engineering ... reading the output to reconstitute the original model. Wouldn't be, couldn't be, perfect ... but it would be better than spewing out code.

Still, if the underlying problem is the complexity of J2EE in terms of getting reasonably sized projects working ... then J2EE is overused and should be bypassed. EJBs don't lend themselves to real, lightweight, agile development models (XP or not) ... they are too reliant on the container and on the runtime environment to be easily tested outside of that container. Any reasonable development model involves running frequent tests ... for example, Tapestry has around 400 tests that run in 40 seconds (and I want to speed that up significantly). HiveMind has 160 tests that run in three seconds. Did I break something? Click the test button.

At work, for me to test an EJB change requires a build (to package the EJB and deploy it) which is currenlty taking about ten minutes. You read that right ... we have some issues with our build environment (they're being worked worked on, but ... "House built on a bad foundation ...")

So, with every change requiring a minimum of ten minutes to validate (usually, much, much longer) agile is dead. Refactoring to use HiveMind (or other microkernels) could support a fast, robust, agile methodology ... and bring into question whether MDA makes sense.

Meanwhile, my other peeve with MDA is that it assumes there will be one uber-developer (the architect) who can create the model, and create the code generation scripts (or modify the supplied ones). How many people can wear that many hats? Doesn't that create a choke point? Isn't that another block to agility ... you don't fix your code, you have to find and fix the code that generates your code, regenerate all, retest, hope it didn't break something outside your fiefdom.

MDA. Offshort outsourcing. Mission statements. ISO 9000. What's the next fad that's going to distract me from getting actual work done?

Monday, September 15, 2003

Presenting at ApacheCon

I'll be presenting an all-too-short Tapestry session at ApacheCon on Nov 17th at 11am. I have 60 minutes to talk Tapestry and handle Q&A ... glad I already have some materials ready, but I don't know how much I can cover in that amount of time.

Friday, September 12, 2003

HiveMind call for participation

HiveMind is, I believe, ready to go! The last few days have been spent refining the framework in a number of small ways. Essl Christian has really gotten very much involved, even contributing a useful patch that addresses the complex class loader environment within Tomcat. He's also provided some great input that led to further shaving away of unwanted code. Sometimes coding is like sculpting ... you just remove the stuff you don't absolutely need.

I'm now interested in putting together a real community to bring HiveMind forward. This will mean putting together a real proposal to move HiveMind out of the sandbox and into Jakarta commons proper. A community around HiveMind would be a good start.

My vision for HiveMind going forward:

  • Select a starting contributors community. Me, Essl, Bill Lear?, Harish?, David Solis?
  • Move to Jakarta commons w/ a charter, voting and all the dressing. Yea! More administration!
  • Move the source code over from sandbox to proper. Reorganize it into a Maven multiproject
  • Progress to a 1.0 release
  • (In parallel) Begin to add additional sub-projects to wrap common code as resuable HiveMind modules. Examples: object pools, database pools, Lucene?, Betwixt?, Hibernate, etc.
  • Conquer the world, and all that

So, if interested, drop me an e-mail or post to the Jakarta Commons Developers list.

Sometimes testing numbers just don't make sense

Based on what I did yesterday, I thought I'd see if I could improve the efficiency of the LoggingInterceptor by directly invoking methods on the commons-logging wrapper class, rather the the Log interface. Fortunately, I decided to do some tests to see about relative costs. I don't fully understand the results (Windows XP, Sun JDK 1.4, Hotspot server) ... obviously, Hotspot is doing something interesting to the code.

Run Log4J Direct Jakarta (Interface) Jakarta (Impl)
Run #1 741 711 701
Run #2 1622 671 551
Run #3 661 521 541
Run #4 641 540 571
Run #5 621 541 551
Run #6 610 521 551
Run #7 611 531 560
Run #8 611 531 551
Run #9 621 521 550
Run #10 611 521 551
Run #11 611 530 551
Run #12 611 521 551
Run #13 611 520 561
Run #14 611 531 550
Run #15 611 511 531
Run #16 581 510 581
Run #17 621 531 601
Run #18 631 571 570
Run #19 621 541 571
Run #20 621 541 570

The tests invoked isDebugEnabled() 50,000,000 times. I did what I could to help ensure that things didn't get unrealistically inlined.

What I don't understand is why going through the Log interface is sometimes faster than either going through the Log4J Logger class, or through the commons-logging Log4JCategoryLog class. I checked the commons-logging code ... Log4JCategoryLog is a very thin wrapper that delegates to a Logger.

Anyway, until I find better tests for figuring out performance, there's no reason to change the LoggingInterceptor code.

Thursday, September 11, 2003

Improved HiveMind efficiency

I was thinking about interceptors in the shower this morning (best place to do serious thinking) and was also thiking about the performance results I got a little ways back. It seems like the big hit as the constant calling of interface methods instead of instance methods. It's well known that calling a method on an interface is much slower than calling a method on an object instance ... the JVM has to do a more expensive dynamic lookup and there are fewer options for optimizing the call.

One idea I had was to rework how interceptors work, so that all the interceptors would work together to make a single interceptor class. Didn't seem like the right approach, since it would be a lot of work, and I think it will be rare for a service to have more than two interceptors. Hard to test, hard to implement, no assurance of a return.

Next, I considered having each interceptor subclass from the previous; this would allow calls to use super.foo() rather than _inner.foo(), which would be more efficient (a super-class invocation rather than an interface invocation).

Then I realized that I always have the actual type and instance of each interceptor (or the core implementation) ... which means I can build interceptors in terms of concrete classes rather than interfaces.

To see if there was any benefit, I needed to test. I extended my test harness to add to new Adder services; the first is like the Singleton service, but has one NullInterceptor (this interceptor simply calls through to the next object). The second has two Interceptors. First run (JDK 1.4, Hotspot server) showed the cost of those interceptors:

Run Bean Interface SingletonDeferred ThreadedOne Interceptor Two Interceptors
Run #1 211 390 2183 2824 3185 2393 9174
Run #2 250 2343 2324 2864 3014 2434 9203
Run #3 240 2354 2323 2844 3054 2394 9253
Run #4 241 2333 2353 2824 3045 2403 9183
Run #5 231 2353 2333 2825 3064 2383 9194

Compare column "Singleton" to "One Interceptor" and "Two Interceptors". I don't exactly have a theory for why the difference between one and two interceptors is so large. Also, the value is red is not a mistake, we've seen it before ... that appears to be Hotspot making an early optimization when there is only a single implementation of the interface; later it must go back and rewrite things when new implementations show up and it can no longer be certain that interface test.Adder is always class test.impl.AdderImpl.

I then reworked the interceptor code to use the actual class to call the next inner-most (interceptor or implementation). This is possible because of the order of contstruction: the core implementation is constructed first, then the lowest order interceptor is wrapped around it, the the next lowest order interceptor, and so forth. This time there was a dramatic change:

Run Bean Interface SingletonDeferred ThreadedOne Interceptor Two Interceptors
Run #1 210 361 1993 2593 2925 2203 2233
Run #2 210 2153 2144 2613 2804 2194 2213
Run #3 220 2143 2153 2594 2784 2203 2203
Run #4 221 2143 2153 2583 2804 2194 2193
Run #5 220 2143 2153 2574 2814 2193 2213

And so we see the desired outcome; adding interceptors is now a small, incremental cost.

Why is this important? We often hear about the 80/20 rule, that 80% of the performance problems are in 20% of the code. My concern is that all the extra layers of interface invocations will become a visible factor, cumulatively ... but because all those calls are so completely, widely distributed, they will become impossible to track down. Sort of grit in the machinery ... no single, large point of failure, but a kind of overall friction in the works.

Given just how many iterations of my code loop I need to get measurable results (50,000,000), that's probably not a large concern ... still it's nice to claim I've optimized HiveMind for performance.

Tuesday, September 09, 2003

More substantial docs on HiveMind

I've added several new documentation items to the HiveMind site. One describes how to "bootstrap" registry ... exactly what code you execute to locate and parse all the HiveMind module deployment descriptors and form a Registry (it's just a couple of lines). A second page describes how to override a service with a new service. Finally, a big page is a case study of how HiveMind is used in Vista to manage startup and shutdown logic.

Meanwhile, HiveMind is really taking root at WebCT. Several new projects within Vista will leverage HiveMind in one way or another; all of these fit the HiveMind pattern: service oriented, with distributed configuration needs. For example, a new data access layer is in the works, and different tools and services will contribute DAOs (Data Access Objects) for different types of databases. Rob Lorring has just put together a JMX-based monitoring package using HiveMind ... different tools contribute different monitors. The new caching service is configured using HiveMind as well.

The recent call for contributors to form a HiveMind community is also coming along, with a few people popping up to see what it's all about. Once I have a free moment (a laughable concept) I'll put together a proposal to move HiveMind from the sandbox into Jakarta commons proper.

Friday, September 05, 2003

HiveMind --- now with downloads

I hacked together some stuff to allow downloads (binary and source) of HiveMind. It's out of my home page at jakarta.apache.org. There's some definite interest building in HiveMind ... pretty soon, it'll be time to form an actual community and propose moving it out of the commons sandbox.

Wednesday, September 03, 2003

"Tapestry in Action" cover

I got goosebumps! Manning has decided on the cover illustration and here's what the book will look like:

With luck, the first chapters will show up on MEAP in about two weeks.

"Tapestry in Action" will be MEAP

Got some confirmation that the book will be available as incremental, online PDFs prior to dead-tree format. Not sure at this time what the schedule is ... there's still a lot of copy editting, indexing and production to be done.

In other news ... I'm trying to set up the necessary definitions to allow Javassist to build inside Gump. I'm awaiting S. Chiba's permissions, even though I probably don't need it -- can't see why he'd object. This will be good for other Gump projects that have a Javassist dependency, such as Tapestry and HiveMind.

Saturday, August 30, 2003

HiveMind EJBProxy is back

Just got the latest Javassist JAR from Shigeru Chiba and he's fixed the problem related to try ... catch that was breaking the EJBProxyFactory service. He's on Tokyo time, so communications can take a while; I believe he wants to do a release of Javassist 2.6 in the next few days.

Pull the fork back out?

So now the "Tapestry in Action" goes into production; a complicated process of copy editting, typesetting and marketing. One part of this is that I must go back through the manuscript and create the index, a new and mind-numbing procedure. In addition, it's looking good for Manning to public the book under MEAP (early access / online versions of the chapters).

I got some odd feedback from Maryan Bace (publisher at Manning) and Ted Neward about a previous blog. Ted felt that:

The last set of reviews were pretty positive, except for Ryan Cox who was, like, 95% positive, but really didn't like a few aspects of the book ... didn't like the Hangman example app, didn't like the sidebars and warnings, didn't like the deep investigation of the Virtual Library application. This is part of the process ... about fifteen different people have reviewed some or all of the book, and they disagree wildly. Mind Bridge, Geoff Longman, Greg Burd and Dave Buck think the book is aces, and they are my real target audience so I'm not exactly worried.

... was a slam on Ryan. If it could be interpreted that way, I apologize profusely. Both Ryan Cox and Bill Lear were the most negative about the book, and therefore the most useful. Both took considerable amount of time to really work through the manuscript and provide reams of questions, notes and challenges. That's how the process works ... it's suprisingly democratic; as the author, I'm required to respond to each and every reviewer comment. Responding to Bill's comments is what, in my opinion, kicked the book up from a pile of facts into a real, flowing, useful tome. Ryan's comments came later, and so couldn't have the same impact, but he definately zeroed in on particular areas that needed expansion and elaboration. The point of my original blog was that even someone who disagreed on some fundamental aspects of the book was still very positive about the book in general.

Tuesday, August 26, 2003

Performance of different HiveMind service models

Thought I'd do a little comparison of how the different HiveMind service models perform (relative to each other). First of all, I found a huge difference between Hotspot client and Hotspot server; HiveMind seems to really benefit from Hotspot. Second, the big difference is between simple virtual method invocation and invoking interface methods ... an order of magnitude difference with HotSpot server. I think the big difference is that the very simple adder bean gets completely inlined by HotSpot. I'm going to try and find a way to convince HotSpot not to inline the bean code.

The method being invoked is:

public long add(long arg0, long arg1);

Each run invokes the method 50,000,000 times. The elapsed time is the time in milliseconds. Note that the Java clock (System.currentTimeMillis()) is very, very innaccurate.

JDK 1.4, Windows XP, Hotspot Server

Run Bean Interface Singleton Deferred Threaded
Run #1 200 320 2003 2544 2874
Run #2 220 2324 2223 2614 2844
Run #3 210 2454 2894 2654 2894
Run #4 210 2223 2083 2604 2734
Run #5 230 2083 2073 2594 2794

Bean - invoking the method directly on the service implementation class (that is, instantiate the class without using HiveMind)
Interface - Invoking the method against an interface, not object, reference
Singleton - Invoking the method against the service using the singleton model (as expected, very close performance to interface)
Deferred - Invoking the method against the service using the deferred model (slight increase in time reflects the extra proxy and synchronized block)
Threaded - Invoking the method against the service using the threaded model (another increase for the additional proxy and the ThreadLocal stuff)

JDK 1.4, Windows XP, Hotspot Client

Run Bean Interface Singleton Deferred Threaded
Run #1 381 2183 2193 5157 8573
Run #2 380 2203 2213 4116 7701
Run #3 371 2203 2243 4186 7681
Run #4 371 2203 2253 4126 7701
Run #5 371 2203 2213 4126 7701

Big difference switching from Hotspot server to Hotspot client; hard to say why in particular ... less inlining but also less of a difference between bean and interface, but the additional cost of the extra proxies (for deferred and threaded) and for the ThreadLocal stuff (for the threaded model).

Something useful to do next would be to create a more realistic bean, something less likely to be completely inlined by HotSpot, so we can really see the difference between the Bean and Interface invocation costs. I think the take away is that the big cost is the difference between Bean and Interface, the cost of adding Deferred or Threaded service model is costly, but incremental.

In addition, the little bit of performance testing I've done using the HiveMind test suite shows that the one-time cost of parsing the HiveMind module deployment descriptors is signifcant; all else is dwarfed by this cost. I'm concerned, or perhaps merely curious, about performance when you build a full application from a wide mix of singleton, deferred and threaded services. Services calling services calling services, all as interfaces, all with proxies and perhaps interceptors ... what will be the cost of all that wrapper code? On the other hand, what are the costs of executing a managed object as an EJB? I'm fairly confident that refactoring stateless session EJBs into HiveMind services (deferred or threaded) would be a net win performance-wise.

Monday, August 25, 2003

Stick a fork in it!

Tapestry in Action is complete, uploaded and ready to go. Well, not quite, tomorrow when I'm not quite so bleary eyed, I'm going to make a final pass over the manuscript to find any lingering errors. Still, this has been eight months of fairly continuous effort, overshadowing everything else going on in my life, stealing my summer, distracting me from extending Tapestry and HiveMind.

Next up, fill in the Tapestry documentation gaps and make sure Tapestry is 3.0 (final) before the book is on the shelves. Clock is ticking, we have about two months!

Hopefully, many hands will make light work; looks like we may have three (!) new Tapestry members shortly, including Erik Hatcher (aka, Mr. Ant), Tsvetelin Saykov (who works with Mind Bridge) and Harish Krishnaswamy.

Meanwhile, a few folks are not too happy that I responsed to some flame-batewith some industrial strength venom. Was it unprofessional? Perhaps; I do take Tapestry very, very personally (which is paradoxical with how pleased I am at how the community and developer pool is growing). Still, leaving a challenge like that unanswered is not in my nature ... and if you read what's in my reply, I think you'll see that, even within the flame, I'm still on-message and marketing! marketing! marketing!

And now to bed ... tomorrow I buy my new house (!). Yes, it's everything, all the time, 24/7 and full after-burners at Chez Howie.

If you like retro arcade-action-puzzle games, check out Rocket Mania! by Popcap Games. This is the other thing keeping me from finishing the Tapestry docs :-)

Saturday, August 23, 2003

Home Stretch on the Book

I'm on the home stretch now for "Tapestry in Action". Manning wants the final version by the end of this month (Aug 2003) ... I'm reconciling all the references to listings, tables and figures, and exporting my ugly diagrams from Visio into TIFF, which is what the production people need. Pretty much, a gruesome, crushing process.

The last set of reviews were pretty positive, except for Ryan Cox who was, like, 95% positive, but really didn't like a few aspects of the book ... didn't like the Hangman example app, didn't like the sidebars and warnings, didn't like the deep investigation of the Virtual Library application. This is part of the process ... about fifteen different people have reviewed some or all of the book, and they disagree wildly. Mind Bridge, Geoff Longman, Greg Burd and Dave Buck think the book is aces, and they are my real target audience so I'm not exactly worried.

Attack Struts! Don't mention Struts! Compare it to Servlets! Don't mention Servlets! More sequence diagrams! Avoid UML! Keep it short! Go in-depth! ... it's maddening. Tapestry is just so big ... and yet, so simple at the core of it. The same little trick repeatedly endlessly, and the only way I know to demonstrate that is to show that same trick again and again and again.

I just had to take a break, so I did some more work on HiveMind. I've added the concept of different "service models" for services; the existing ones are singleton and deferred. Singleton is a singleton service, created at first reference. Deferred is based on some work I did last week; the service creation is deferred; a proxy is created to stand in for the actual service until a service method is invoked.

The new service model is threaded. This is something new and neat; it bypasses the multi-threading problem by allowing each thread to have its own dedicated instance of the service. That is, the proxy will construct an instance of the service that is bound to the current thread (ah, the magic of ThreadLocal). From that point on, any time that thread invokes a service method, it is directed to that thread's service instance. Same proxy passed to a different thread will get a different service instance ... I actually test this as part of the unit test suite.

There's now a service, ThreadEventNotifier, that is used to inform objects when the thread is "cleaned up" (i.e., the end of the request cycle in a servlet or Tapestry app).

I suspect in the future, I'll add additional service models; for instance, one like threaded, but using a pool of service instances ... and maybe one like threaded, but permanently attached to the thread once created.