Load-balancing with the TIBCO Portal

The TIBCO Portal (and perhaps other TIBCO web-based products) does not support load-balancing within the product itself. It is the responsibility of the servlet container to implement load balancing.

A brief explanation of the TIBCO system is necessary. TIBCO uses a product called the TIBCO Administrator (I'll call it the Administrator for short) to maintain an "administrative domain". This administrative domain consists of machines, servlet containers and software. Machines and servlet containers are registered with the Administrator. The TIBCO portal is packaged as an EAR which is deployed by the Administrator on the domain. The EAR contains several services (WAR files) which are deployed by the Administrator to one or more servlet containers registered with the domain.

You would think that you'd be able to deploy the portal services to more than one host and then place those hosts in the load-balanced cluster.

Unfortunately, you can't deploy most portal services to more than one servlet container in a domain. So, you can't do load-balancing this way.

As I said earlier, it is the responsibility of the servlet container to implement load balancing. The Administrator expects that the cluster behaves like a single servlet container, and that you register the cluster with the Administrator, not the constituent servlet container instances.

I wanted to do a load-balanced install of TIBCO Portal Builder under SunONE 6.1 SP4. This version of SunONE doesn't support clustering natively, and as I mentioned above, the Portal won't allow us to deploy services across multiple servlet containers. To work around this, TIBCO suggested that we install the portal on one machine, then copy (manually) the configured and deployed web applications to a servlet container on the other machine.

There is a snag with this approach, however. Some Portal services cannot be installed in this way. The BPS_PB, and CPS services must not be deployed onto more than one servlet container. So, to work around this, you have to place these services in yet another servlet container.

There is one final point with this approach. The Administrator appears to store the file system path of each registered servlet container in the Administration database. Each Portal service uses this information to find the file name of the servlet container configuration file (e.g. server.xml in SunONE) and parses this file on servlet initialisation. Precisely why this is necessary, I don't know, but the upshot is that:

1. You have to install the servlet container in exactly the same location on both load-balanced machines, and


2. You have to set the server name of servlet container on both the load-balanced machines to the same name.

If you don't, you won't be able to start the second server instance successfully.

My and my "Canguro"

Well, here is a picture of me :-)

It was taken at a church function in Spain with my girlfriends' family. I was presented with a "Canguro" since I'm Australian!

Code Formatting for Blogs

It has been annoying me for some time that I didn't have properly formatted code in this blog. I could get it to either indent correctly but expand and ruin the page formatting (using <pre>), or line-wrap inappropriately (using <tt>).

Today I did a bit of web searching and found this excellent blog article on how to format ruby code for blogs. While the whole solution is a bit more than I need, inserting the following snippet of CSS into my blog style was just the ticket.

pre {
    background-color: #f1f1f3;
    color: #112;
    padding: 10px;
    font-size: 1.1em;
    overflow: auto;
    margin: 4px 0px;
          width: 95%;
}

There's still a little bit that's not prefect (small <pre>...</pre> sections still look a bit funny), it's almost exactly what I was looking for!

Isn't the web wonderful :-)

Proxy me baby

We have pretty much decided to use the mod_proxy_html module for apache with SunONE as our platform. This means that the apache module will parse and rewrite the HTML generated by the portal, fixing up the URLs as appropriate.

Why? Well the production environment is using Novell iChain, and it does basically the same thing. Thus, developing and testing with this parsing overhead is a reasonable way to proceed.

Hopefully this will work properly!

Hard to find a place in the SunONE

As the TIBCO Portal doesn't support Tomcat in a production installation, we are using SunONE 6.1 SP4 as the web container.

Getting SunONE to install with the portal has been frought with difficulty. What we had wanted to do was to install SunONE on port 80. We want port 80 since the infrastructure already allows access to this port in the demo rooms.

After much work, we discovered that it is basically not possible to install the web server in this manner without having setuid-root shell scripts. This is because the deployment process in TIBCO Administrator relies on being able to programmatically restart the web server. Needless to say, we're not precisely keen on this.

As a side item, if you want to install SunONE for use with the portal, you actually have to install TRA, Administrator, SunONE and the Portal as the same non-root user. The documentation suggests that having these products share the same group is sufficient, but that isn't true.

Eventually we have got to the stage of having a reverse proxy in front of our web server. This works great with Tomcat, but not so well with SunONE.

The basic problem with the Portal and the reverse proxy is that the portal generates some absolute links in it's HTML markup. These links must have the host:port of the reverse proxy and not that of the portal web server if they are going to work properly.

Under Tomcat, this is simple - set the proxyHost and proxyPort properties of the <Connector> in the server.xml and voila! it works. Unfortunately SunONE provides no such mechanisim that we can determine. From what I can see, there is no way to configure SunONE to operate transparently behind a proxy!

Deployment and the TIBCO Administrator

If you have used the TIBCO PortalBuilder product, you will know that the 5.x release is now integrated with the TIBCO Administrator. Deployment of components of the portal, or new JSR 168 portlets is done by creating EAR files and deploying them under the Administrator.

The way deployment works is like this:

1. The Administrator tells the Hawk agent (not the hma) to deploy the EAR


2. The EAR is unpacked into a scratch area by the Hawk agent's deploy plugin


3. The unpacked EAR is examined by the deploy plugin and XML files modified to set up the configuration as was specified in the Administrator console


4. The unpacked EAR is copied to the nominated web server by the deploy plugin


5. The nominated web server is restarted


6. The deploy plugin waits for the server to start

So, to watch the deployment proceed on Unix you need to:

1. Use nohup to start the Hawk agent so a log file is kept (nohup.out)


2. Use prstat to monitor the running processes (will show you the start/stop scripts running)


3. Use tail -f nohup.out on the Hawk agent nohup output to watch the reconfiguration happening


4. Use tail -f on the server log file (catalina.out for Tomcat, errors for SunONE)

Another key item that you need to do is reset the timeouts for the Hawk agent and the Administrator.

The Hawk agent timeout (i.e. the amount of time the Hawk agent will wait for one the activities it does to complete) is specified in hawkagent.cfg (located in the TRA domain directory).

The Administrator timeout (i.e. the amount of time the Administrator will wait for the Hawk agent to complete) is specified in tibcoadmin_DOMAINNAME.tra (located in the Administrator domain directory).

These two values are in seconds and should be set to something between 600 and 720000. Values higher than 720000 are ignored (and 600 will be used instead.) I use a value of at least 3600.

The other item that is crucial is to know when the hawk agent should be restarted. As the hawk agent (through the deployment plugin) scans files and looks up classes, it must be able to find any classes referenced in the configuration. For example, if you set an authentication wrapper class or other plugin class, this class must be known to the hawk agent, or the deployment will fail. Now, as the hawk agent establishes it's class path and loader essentially on startup, JAR files added to the path won't be found after startup, and modified JAR files won't be re-read.

Basically, you should restart the hawk agent whenever you add a new JAR file to the system.

Finally, the plugin directory PORTAL_HOME/5.1/lib/extensions is not present (by default) in the class path used by Hawk, so any JAR files placed here won't be found. You have to add this directory manually to the hawk configuration:

• Open the TRA_HOME\domain\DOMAINNAME\hawkagent_DOMAINNAME.tra file for editing.


• In the java properties section, add the following to the tibco.class.path.extended path:
  PORTAL_HOME/5.1/lib/extensions.


• Restart the Hawk agent.

If you get all of the above right, then deployment should just work for you!

A long time...

Yes, it's a long time and no posts! This has been in part due to my wanting to spend more time on other things (i.e. my girlfriend) and on my reluctance to post to this blog while at work.

I don't know about you, but I just don't feel like doing "computing" things when I get home anymore. Perhaps it's a sign of old age, or perhaps I'm becoming more work-life balanced.

I am still fighting with the TIBCO Portal suite, which I will post upon shortly.

In other news, Laura (my girlfriend) and I are now engaged and due to be married in August 2007!

Hasta luego!

JDK1.5 and Pluto

Today I was installing Pluto on my home PC. I built and deployed the sample RSS Portlet from the POST site. It didn't work :(

So, I looked into the logs and found:

14-Jan-2006 19:01:18 org.apache.pluto.portalImpl.services.log.LoggerImpl error
SEVERE: PortletInvokerImpl.render() - Error while dispatching portlet.
javax.xml.transform.TransformerFactoryConfigurationError: Provider org.apache.xalan.processor.TransformerFactoryImpl not found
 at javax.xml.transform.TransformerFactory.newInstance(Unknown Source)
 at com.plumtree.portlet.portlets.RssPortlet.init(RssPortlet.java:98)
 at javax.portlet.GenericPortlet.init(GenericPortlet.java:100)

A little googling showed this was another instance of the JDK 1.5 XSL parser change. To fix, you just need to download and install Xalan into TOMCAT-HOME/common/endorsed and everything works fine.

Pluto is the bomb!

Yesterday I was stuck with a not-working TIBCO Portal, by virtue of the database settings (see my previous post!). While the portal wasn't working, I thought there's gotta be a better way to test this stuff - the install time into the TIBCO Portal is just tooooo long and cumbersome.

So, I downloaded Pluto and tried out deploying into the Portal Driver.

In summary - it's magic. Startup for the bundled Tomcat server is a few seconds, and you can run the shebang under Eclipse which makes debugging a breeze! I wish I'd discovered it sooner!

One thing that did cause me a bit of grief is the portlet.xml format. Make sure that you get the namespace and schema declarations correct (see the Troubleshooting part of the help view in Pluto's Admin Portlet for the correct one) and that you validate your XML before deploying. Some errors are not detected until portlet run time, so if your portlet is not working, check the portlet.xml first!

...and back working

Got it fixed, after a call to tech support. It turns out that I'd run out of database connections! That fixed, and all is working again.

Boom! Ground to a halt

Today my productivity ground to a halt... why? Because deployment of my portlet EAR files has ground to a halt :-) The TIBCO Administrator now refuses to deploy any portlet EAR file into the domain!

CreatePASEAR weirdness

Yesterday I encountered a weird problem. My creation of an EAR file using the CreatePASEAR tool stopped working, generating an ANT error 'Unable to delete file':

Unable to delete file C:\Temp\cps\cps_source.ear
        at org.apache.tools.ant.taskdefs.Delete.execute(Delete.java:449)
        at com.tibco.ps.anttasks.utils.AntTaskUtils.deleteFile(AntTaskUtils.java
:129)
        at com.tibco.ps.anttasks.deployment.GenericPSBasedEARTask.execute(Unknow
n Source)
        at org.apache.tools.ant.Task.perform(Task.java:341)
        at org.apache.tools.ant.Target.execute(Target.java:309)
        at org.apache.tools.ant.Target.performTasks(Target.java:336)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1339)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1255)
        at com.tibco.ps.anttasks.TIBCOMain.a(TIBCOMain.java:618)
        at com.tibco.ps.anttasks.TIBCOMain.start(TIBCOMain.java:186)
        at com.tibco.ps.anttasks.TIBCOMain.main(TIBCOMain.java:225)

After much investigation, it turns out the this error is bogus - it is generated by the tool during clean-up operations after the real error has happened. The actual error was malformed XML in my web.xml file (the CreatePASEAR file rewrites the web.xml during EAR generation).

The real error can be found in the TIBCO_HOME/portal/5.1/logs/utils/PortalUtils.log, for example:

[createPASEAR] Caused by: Failed to parse descriptor: Next character must be ">" terminating comment .
[createPASEAR]  at org.apache.cactus.integration.ant.WebXmlMergeTask.execute(WebXmlMergeTask.java:142)
[createPASEAR]  at com.tibco.ps.anttasks.deployment.TIBCOWebXmlMergeTask.execute(Unknown Source)
[createPASEAR]  ... 17 more
[createPASEAR] Caused by: org.xml.sax.SAXParseException: Next character must be ">" terminating comment .
[createPASEAR]  at org.apache.crimson.parser.Parser2.fatal(Parser2.java:3376)
[createPASEAR]  at org.apache.crimson.parser.Parser2.fatal(Parser2.java:3370)

I hope this helps someone from enduring the hours of frustration I had trying to find a bogus file permission/locking problem!

TIBCO Spring portlet issues solved

Well, I have succesfully deployed the Spring sample portlets into TIBCO PortalBuilder! The final problem was a my mistake - somehow the incorrect configuration file was getting packaged into the WAR, so my custom bean for handling the charset problem was not getting instantiated! With this fixed, everything is fine.

Here's the summary of my findings with using Spring under TIBCO PortalBuilder 5.1.

1. DOCTYPE is removed from all XML files under WEB-INF.


2. As a result, any Spring class which validates XML files when reading won't work (that's all of them that I have seen so far)


3. You need to extend at the XmlWebApplicationContext class to disable validation on XML file load. This is done by overriding the initBeanDefinitionReader() method and calling setValidating(false).


4. You need to add a new context-param definition to the context in web.xml to force your non-validating context implementation to be used. The name of this parameter is contextClass.


5. You need to extend at the XmlPortletApplicationContext class to disable validation on XML file load. This is done by overriding the initBeanDefinitionReader() method and calling setValidating(false).


6. You need to add a new init-param to each Spring portlet defined in portlet.xml to force your non-validating context implementation to be used. The name of this parameter is contextClass.


7. You need to ensure that the content type for your views does not include a charset. In other words, text/html is OK, but text/html;charset=UTF-8 is not. The TIBCO implementation does not ignore the charset.

Even with the above changes, I've been unable to deploy the Spring Portlet Sample sample application into TIBCO Portal Builder 5.1.

The final issue appears to be due to the view resolver implementation in Spring - my customised view class isn't being instantiated, so the removal of the charset definition isn't happening. I'm going to post on the spring forums for help on this one - I'm stumped.

Spring and TIBCO

I've decided to use Spring as a framework for writing Portlets within TIBCO Portal Builder 5.1. The reasons are:

• Has support for Portlets as of 2.0M1


• Doesn't require a bridge layer (unlike Struts)


• Can be used with JSPs


• Has a MVC implementation that is supported in 2.0M1

So, I build and attempt to deploy the Spring Portlet Sample sample application. Naturally, it fails, and the debugging begins.

The logging output from Tomcat shows that the applicationContext.xml file is not valid. Examination of the deployed EAR in Tomcat shows that the DOCTYPE definition has been removed from all XML files in the EAR (except for web.xml). By default, the bean configuration reader for Spring has validation enabled, so having the DOCTYPE missing causes the parse of the bean definition XML file to fail.

Initially, I had thought that TIBCO's EAR creation tool was the culprit (used to build the deployable EAR from the portlet WAR), but it turns out the TIBCO Administrator modifies the EAR during deployment to the application servers in the administrative domain.

The next step is to use a custom XmlWebApplicationContext class which disables validation on the XmlBeanDefinitionReader before loading the bean definition files.

Hopefully with this work-around a Spring portlet can be deployed in TIBCO PortalBuilder 5.1.