Reworking Tapestry Docs and Demos

A slight shift of focus this week, especially today. I've been reorganizing the Tapestry project home page; from now on, the menu down the left side will be consistent from page to page, not contextual. The tabs across the top are now more like book marks. I think it will make navigating the site easier. Further, the first thing on the home page is another set of quick links to the most common pages, including the Users Guide, Quick Start Tutorial, JavaDoc, etc.

Now, the Quick Start tutorials represent another way to ease newbies into Tapestry. It's been something I delegated to others in the past, and it never happened, so I've taken it back on myself. I'd rather be writing new code or even fixing bugs (or working on my own labs), but I've been getting feedback that the QuickStart is making a big difference among prospective Tapestry users. In fact, there have been over 2200 downloads of the tutorials already!

In the same light is my new approach to the Tapestry demos: the Workbench and the Virtual Library. In the past, you had to build them yourself (because they use non-ASF code, and can't be distributed from the ASF). But that's asking a lot from people, especially those just casually investigating Tapestry.

The new approach is a seperate distribution for the demos, that comes from a non-ASF server (http://howardlewisship.com/downloads/quick-start/).

This bypasses the concerns of the ASF ... though it does eat into my personal web site's bandwidth! But what it means is people can get up and running with Tapestry really easily:

• Install JBoss 4.0.2
• Install Ant
• Unpack the examples
• Run the build script using Ant
• (Re)start JBoss
• Run the applications in a browser

A step up from this would be an installer, but I was short on time to do that today. It would be nice, and there are reasonable, free installers out there. That would eliminate the need for Ant ... but I assumed that nearly everyone interested in Tapestry would already have Ant.

It's a shame that living inside Apache causes these convulsions; it will rise up again in 4.1 when we start wanting to do deeper integration of Tapestry and existing AJAX client-side JavaScript libraries ... that work will have to take place largely off-Apache.

Switch?

So, the other day, I had some time to kill at the mall, and I wandered into the Apple store. Walked up to a big G5 with the killer 30" display and downloaded Eclipse 3.1 ... well, within a couple of minutes I was pulling down HiveMind code from subversion and compiling and executing the tests.

Now, on the one hand, I had severe techno-lust. All that eye candy! Vast tracts of land on that giant screen. Candy shaped buttons. Going home in a way (I used to develop on NeXTSTEP). Realistically, I don't think I can afford the 30" display, but maybe the medium sized one and I probably need a pretty maxxed out G5 for the kind of stuff I want to be doing. I mean, right now I do all my development on a Pentium 4M 2ghz with 1 gig of ram and a 40GB drive, but I'm increasingly impatiant with my development environment ... but what I want is something that really screams for all the development I do! And I do want to do some video editting as well.

So, convince me ... I can build out a monster 'doze box and buy a big LCD, or I can go the Apple route. How much is the tax on all those pretty icons? What's the tarrif on the sleek design? And what's the future with the move to Intel looming?

Lots of small objects ... or too many?

I was very interested in the post Object Count Impact on Garbage Collection Performance. As someone who tends to work using large numbers of small objects, I do have a concern in the back of my head about crossing over the line ... especially in light of the slower startup speed of Tapestry 4 (on top of HiveMind) vs. Tapestry 3.

In fact, the article jives well with my thinking; Slobodan's minimum object count map (which he promises to discuss in later blog entries) may likely look like the class I have yet to write ... a very, very simple Map implementation tuned for a small number of keys and a very, very, very high ratio of reads to updates ... the kind of thing that, say, stores attribute values for an XML entity (something used all over HiveMind). I was thinking about using an Object[] array at the core, with alternating keys and values.

Meanwhile, I still think the general approach I take is the right one ... and the reason I can't ever go back to a non-garbage collected language. I like breaking problems into tiny pieces and getting emergent behavior by combining the tiny pieces into more complicated wholes. From an efficiency standpoint, Tapestry is doing the right thing by caching entire pages (not components within pages, which would be more of a JSP tag model). But Tapestry does use an awful lot of HashMaps.

Resurrecting the Virtual Library

One of my goals for the final release of Tapestry 4.0 is to have the Virtual Library working again. The goal is to work through the issues, fixing things that are broken by the upgrade from 3.0 to 4.0, and taking advantage of the new features of Tapestry 4.0 as well (including annotations). I think this is important; the Virtual Library is a small but real application, and seeing the upgrade path for it will allow other 3.0 users to extrapolate what they are facing in upgrading to 4.0.

Getting the Virtual Library working has been my focus all day today ... mostly struggling with JBoss. Eventually I gave up, and the turnkey demo will overwrite the defaultDB instance rather than create a vlib database instance ... with that compromise in place, I'm again making rapid progress.

I've also considered porting the code to Hibernate and/or JDO (away from entity EJBs). This would allow the entire application to be delivered as a simple WAR, not an EAR. Getting the shared libraries working in JBoss was a pain ... I had to set the Manifest Class-Path of my EJB jar, to pick up the shared libraries (shared by the EJBs and vlib.war). However, changing the ORM will partially confuse the issue of the changes in the presentation tier, so I doubt I'll even attempt that until 4.1 ... and by then, I might have a bigger, better Tapestry example.

Looks like I'll be spending some time manually re-entering the data as well; QuantumDB had problems exporting and importing the data for me.

Still, I should have most of it done this week .. assuming I continue to play hookey from my paying gig (don't count on it!) So we probably won't see this until beta-10.

Bad JBoss feedback

I harp on feedback all the time, here's an example of why it's so important.

I've been trying to deploy a new Hypersonic database into JBoss 4.0.2, to contain the data for the Tapestry virtual library demo.

When I deploy my application, here's the exception I get:

org.jboss.util.NestedSQLException: No matching credentials in Subject!
       at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:107)
       at org.jboss.ejb.plugins.cmp.jdbc.SQLUtil.fixTableName(SQLUtil.java:159)
       at org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCEntityBridge.init(JDBCEntityBridge.java:143)
       at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.initStoreManager(JDBCStoreManager.java:420)
       at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.start(JDBCStoreManager.java:353)
       at org.jboss.ejb.plugins.CMPPersistenceManager.start(CMPPersistenceManager.java:157)
       at org.jboss.ejb.EntityContainer.startService(EntityContainer.java:340)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
       at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:418)
       at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
       at $Proxy49.start(Unknown Source)
       at org.jboss.ejb.EjbModule.startService(EjbModule.java:395)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
       at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:418)
       at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
       at $Proxy22.start(Unknown Source)
       at org.jboss.ejb.EJBDeployer.start(EJBDeployer.java:605)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:964)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:956)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:775)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:738)
       at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:121)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
       at $Proxy8.deploy(Unknown Source)
       at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:325)
       at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:501)
       at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:204)
       at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:277)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
       at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:418)
       at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)

I'm still digging around; since I can connect to the database over TCP/IP, it means the problem is in the connection pool. Still, wouldn't it be nice if JBoss told me that? How about if it told me what was broken - where it was broken - how to fix it? That's the kind of support built into Tapestry and HiveMind: line precise exception reporting plus lots of attention to detail in error messages and error reporting in general.

Well, I've been hacking away for a couple of hours, and will probably be at it for a while yet. Not the way I like to spend my day. Sigh.

Code Coverage Metrics and Testing

For the last few months, I've been tracking some metrics about Tapestry. On each release, and occasionally in between, I've been tracking the NCLOC (non-comment lines of code), the number of Java source files, the number of tests, and the percentage of code coverage (according to Clover).

I have an almost religous belief that you must have at least 85% code coverage for a project; I think the bugs "hide like cockroaches" and are only flushed out when you hit that magic number. Thus, I spend most of my coding time writing tests, not production code. The trick to staying sane is to remember that production code and test code are all just code, of the same cloth, each useless without the other.

You can see the amount of code growing with occasional downward spikes as dead code is located and eliminated (the large spike around 8/10 represents a change in how I was counting code, counting the code for each sub-project seperately -- these charts are just for the framework itself).

And here you can see the stuggle to keep the code coverage about 85%. Over this same time range, the number of unit tests has risen from 906 to 1,257.

In fact, the news is actually pretty good; in Tapestry most new code, code introduced for release 4.0, has close to 100% code coverage. These charts are just for the framework proper, but the annotations library, for example, has true 100% code coverage (every line of code, every branch).

The majority of new tests are written as true unit tests, on top of EasyMock ... all but about 40 of the tests, which execute mini-applications inside a simulated servlet container. Those 40 tests take about 220 seconds to execute; all the remaining tests combined take about 25 seconds. Thus the power of unit testing.

All told, the Tapestry project includes over 1400 tests. And Tapestry is built upon HiveMind, which has about 740 tests itself.

Personally, I'd like to see the code coverage numbers jump up into the 95% range. I think Tapestry 4.1 will start with a purge of dead and deprecated code, and then a lot of filling in the gaps in the unit tests, before we start working on new ideas. And I have a dream of simplifying the integration tests and productizing them, so that they can be used by Tapestry developers to test their applications. What would be great would be to link together something like jWebUnit with a mock-up of the servlet API, for true unit testing (without having to run the application in a seperate process).

TheServerSide Software Problems

It's discouraging to see news stories on TheServerSide.com about their ongoing software problems. TSS has been a banner site for Tapestry, and chatter about problems makes for bad Tapestry press.

The basic issue is that I signed on with The Middleware Company to do a number of phases of development for TheServerSide.com. The first phase was the basic translation of the site to a component object model, leaving all the functionality unchanged.

At the same time this was occuring, a seperate team was converting the backend access from entity EJBs to Solarmetric Kodo.

In the end, I had less than a week to integrate the two before going live. And yet, for the most part, the result was quite succesful -- measured partly by the amount of time that passed before anyone realized that the site had been almost completely rewritten.

However, with the acquisition of The Middleware Company by Tech Target, my involvement with TSS came to an end; the later, more interesting phases, where we simplified the stack and built considerable UI improvements, has not come to pass. All I've seen is the introduction of more and more ads on the site.

I can't talk to the root problem today; it is incredible frustrating that many posts get accepted and lost. That speaks to some horrible issue with transaction management and the database. Interestingly, TechTarget has not approached me for help ... this also speaks volumes as to where the problems lie.

From observation, and from discussions with Joe Ottinger, I do know that Tapestry is doing exactly what its supposed to be doing, that the functionality problems (missing posts and such) are a problem at the application layer (the stateless session bean used to manage transactions) and the interaction between that layer, Kodo, Coherence, JBoss and the PostgreSQL database. In fact, given the simplicity of the database schema (just six or eight tables) I suspect the problem really is in the configuration and integration of these elements.

Based on what I've read, and some high level discussions I had with them last winter, I believe TechTarget is building a single enterprise wide solution for all their many web sites, migrating away from the Tcl-based Vignette solution used by the majority of their sites, as well as the Tapestry-based solution for TSS.com and TSS.net. All I know about the solution is that it will be based on JEE (assuming that hasn't changed since our discussions).

Enjoying Web Development with Tapestry

Kent Tong has updated his e-book on Tapestry 3, Enjoying Web Development with Tapestry, in two ways:

• It now covers Tapestry 4.0
• You can now buy it as hard copy instead of PDF

I think it's great that books on Tapestry 4 are coming into print ... and that I'm not the one writing them. Not only will other people have a perspective closer to the reader, but the very fact that people outside the central Tapestry developer community, people like Kent Tong and Warner Onstine, are expending the effort to write these books, underscores the kind of high quality community Tapestry has developed over the last few years.

Minor Eclipse Gripe

I'm a big fan of Eclipse; I think its full of great ideas, and is a big gift to the larger community. It gets the job done quite well. And it's free.

However, there are quite a few rough spots.

One thing I hit quite often is mysteriously locked files. I see this most often when updating or replacing my directories from SVN (and sometimes CVS). Often, an update will fail with some kind of locking error on the local file.

Sometimes, I have the file open in another window. I close the window and the process works. Less than ideal, but I can live with it.

Sometimes, I don't have the file open and I still get the error. My working theory is that Eclipse has it locked, perhaps due to indexing. Often I have to restart Eclipse to get the lock to clear!

Am I the only one who experiences this?

A clean slate ...

After well over two years, countless installs and uninstalls, a creeping sluggishness, and one horrific event, I finally bit the bullet and rebuilt my laptop computer. Yep, I've formatted the hard drive and reinstalled everything.

Along the way I learned a few things:

• You have to install Outlook just to transfer mail messages into Thunderbird. Hopefully it will uninstall OK.
• JDK 1.5 will only install into C:/Program Files/Java/, which gives Ant fits (the space in the file name is a problem). After trying things with shortcuts, I eventually copied C:/Program Files/Java/jdk1.5.0_04/ to C:/.
• I no longer can remember how I got svnserve (the Subversion server) to start automatically! Woops! Perhaps its a difference between the cygwin version and the Win32 native version?
• I'm having trouble getting my SSH and PGP keys back. This could cause some grief.

I'm still picking up the pieces but I'm very close now. A key tool for me in this endeavor was Acronis True Image, which can take a snapshot of an entire partition and put it on backup media (or an external drive, in my case). Even better, you can "mount" the image as a new read-only drive, making it easy to locate and retrieve individual files you need.

Will my Dell Inspiron 8200 return to its former glory? Hard to say ... I think Windows is getting slower with each release and security patch. But so far, things are much quicker. We'll see once we get to real development.

Update: I have Subversion for Cygwin working great. The missing piece was the --foreground argument. My repository is at c:/svnrepo, so my command line is: cygrunsrv --install Subversion --path /usr/bin/svnserve --chdir /cygdrive/c/ --args "--foreground -d -r svnrepo"

After that, I can start Subversion:

cygrunsrv -S Subversion

Once started, it appears to restart automatically on each reboot.

You can verify a correct start:

bash-3.00$ cygrunsrv -Q Subversion
Service             : Subversion
Current State       : Running
Controls Accepted   : Stop
Command             : /usr/bin/svnserve --foreground -d -r svnrepo