How we use XPages

Cameron, Jesse and Paul have been putting a lot of effort into explaining how they use XPages and it has prompted me to post this on how we at FoCul  use XPages, something I have been meaning to do for a while. I will hopefully post more about why we like XPages so much but this is a long enough read for now.

How it started

18 years ago I started a small Notes based development house supplying bespoke Notes applications to Engineers in the chemical sector. I had been an industrial Engineer in this sector and was frustrated by a lack of collaboration and knowledge management solutions available. This went really well ( paid the bills and I enjoyed it immensely ) until 2008 when a combination of the recession and the dwindling Notes install base caused 50% of our work to disappear very quickly.

We took the enforced free time and took the best of our bespoke Notes solutions and offered them as XPage based SaaS solutions. Being web based they were much more expensive to develop and less reliable but it transformed the business into one where 85% + of our revenue is recurring and we have a much more natural relationship with our clients where there is a strong ongoing relationship that is entirely based upon the success of the products in everyday use.

In the previous bespoke world I had always struggled to let go of the projects knowing that just 10% more effort on application tuning during the first 6 months of use would make a 25% improvement in the usefulness of the application. In the SaaS world continuing to invest in the success of the application is now our decision and one that typically pays off.

We are still 100% on Domino / XPages and having settled on a good application architecture it works very well for us.


Each customer has one or more Domino servers, typically licenced via XWorks. The servers are virtual servers running on VMWare or increasingly on MS Azure. The Operating system of choice was CentOS Linux but we are mid migration to Windows Server. I personally would not have predicted this migration and I will write about it another time. The crux was support for CentOS on Azure and who owns an issue when IBM ( as it was ) blames the OS for the problems and MS owns the ”hardware” and some MS utilities that are required to make it work.

Domino Version

Our applications all run on Domino 9.x and we have not yet had the confidence to move to 10.x. The issue is the perception of a loss of reliability and developer efficiency but to be fair to HCL we have not yet invested much ( @ 3 man days ) in trying 10.x and that was mostly trying DQL.

We were a fairly early adopter of XPages and although there was good IBM sponsored developer evangelism we still found it very tough. As a direct consequence we have a policy of staying away from the bleeding edge where there is an alternative. Having said that the architecture that we settled on for XPages is pretty advanced in some respects.

Our applications

Our applications are the good old fashioned bread and butter collaboration and workflow applications that naturally found their home on Lotus Notes. Things like Shift Logs or Management of Change workflow applications for the chemical and utility sectors. We have SaaS applications with over 1M records but most have less than 50 documents created per day and less than 50 concurrent users. We also have the luxury of very few applications per server compared to most Domino servers.

How are applications are architected

We are on our 5th iteration of an XPages architecture having started with SSJS and a drag & drop approach to XPages. The current architecture has the following attributes :

  • We use Java for 90% of the code, all edited through Domino Designer.
  • Objects on the screen are bound to Managed Java Bean based representations of “documents” rather than directly to documents.
  • “Views” or lists of records are constructed from Full Text Searches rather than views – we have very few notes views relating to what users see.
  • Almost all user “views” have faceted search capability to help users filter down to the records that they want to work with.
  • The “forms” in the application are almost all generated via a “form builder” process in the application admin area where new tabs and fields can be added. This configuration is typically carried out by ourselves as opposed to customers.
  • We use an auto save feature on documents
  • We use either DocX4J to export data to Word and PDF.
  • Other processes can access the data via configuration driven REST APIs or XML feeds.

Taking these one by one :

Why Java ?

In the early days we needed to have quite sophisticated word / PDF export capability and we brought in Ben Poole to help us deliver this using DocX4J. Ben used Java for this and from working with Ben we came to understand the power of Java over SSJS. The availability of 3rd party libraries like DocX4J was an interesting part of this.

Now almost all of our code is now Java and from a manager’s perspective it has been lovely to see how the developers have evolved from the typical “Line of Business” Notes developers into much richer developers. This evolution has given me confidence that we can adapt to whatever the next platform is.

Why represent documents with Java Beans ?

We found that our “documents” were increasingly composite documents from multiple data sources and that we needed more control over how documents were saved. Another important factor was that because forms were based on a configuration rather than being hard coded ( via the “form builder” ) it was important to have quick cached access to the data representing the form structure.

Having become comfortable with Java it was a natural move to use Managed Java Beans and we were initially inspired by Karsten Lehmann. This made it much easier for us to work with the data and control how users edited it. The representation of “view” data is explained below.

Why did we move to Full Text Search to display “view data”

The biggest reason that we did this was to allow users to have “Faceted Search” capabilities when displaying “views”. This is a search where the search terms are progressively reduced as the user add more search criteria.
A strong secondary reason was that because the forms are configured using a form builder a field which has data key to a business process could have any field name and does not appear in a view.

Displaying a list of users, as an example, would go like this :

  • Skinny Bean :
    The records that are going to be displayed are collected by two searches. The first is a full text search and the 2nd is a formula search of the un-indexed documents. The 2nd search has to be relatively small for this method to work and we force an index update @5 minutes.Having collected the documents the “skinny bean” is constructed to represent the documents in such a way that they can be sorted for a view – only the minimum “columns” are added to the bean. To do this we go to each document and extract items of data – this is the time consuming part.
  • View Bean :
    Having a representation of the ordered document list we can now create a “view bean” which represents a page of the view – so between 25 and 100 documents. To do this data about each view column is constructed but this is only done for a page at a time as part of the view rendering process. This will typically include icons and hyperlinks.
  • Page Bean :
    When a user clicks into a row and as the document is displayed a Page Bean is constructed. This represents the document and is only constructed upon demand. The data may be pulled from multiple back end documents in a process that is a bit like SQL joins. If the “document” is edited then the edits get pushed back into the back end documents at the end of the editing process. The source documents will typically get locked to other edits during this process.

While this approach may seem complicated the same approach is used in most of our SaaS products so it is one code stream that is maintained in one place across all of these applications. It allows us to be able to produce some very powerful applications very quickly – mostly through configuration rather than coding.

Why Faceted Searches ?

Customers expect modern web applications. As our customer base is mostly Engineering types they do tend to push the boundaries in terms of wanting to cut and dice data. An interface without faceted search would have been less successful.

The trade off is obviously speed and this is further exacerbated by the use of a form builder to configure the forms – there is very limited opportunity for tailored views to make the display of data more efficient. Again commonality across most of our applications is key here as the same code is used very widely but maintained centrally

Why use a form builder approach ?

Although we have multiple products they almost all share the same code base – as in they are all based on the same NSF. To be fair we have to work quite hard to make sure that it does not become bloated but the advantages are obvious.

A key enabler of this approach is to configure applications rather than hard code them. To do this we developed a form configuration module within the web facing application admin area where new forms can be built tab by tab and field by field. We did initially try to store copies of this configuration on the actual documents but we quickly exceeded the available data volumes and that started us down the Java Bean approach that we now use.

The customers ( those Engineer types again ) love this approach because we can quickly add or remove fields from forms on the fly. A configuration re-visioning process means that the historical documents will still use the old configuration while new documents will use the modified version.
This approach also has the advantage that we can construct simplified export version of the forms ( word or PDF ) based on their configuration rather than having to hard code the exports.

We do fairly often need to resort to custom elements within the configured forms. We reference these bespoke custom controls in the form builder and they get inserted into the form when rendered just as a configured field would. Increasingly we are looking to have a suite of configurable custom controls that we can use.

We also need custom workflow code in some deployments. To do this we have a series of defined points within our code where we will go off and check a coded routine. If a particular customer needs something special then the routine will have custom code otherwise it is empty. We use a value within the faces-config.xml to define which “plug-in” to use. I am not an expert in Java but I think of it as a crude equivalent to overloading. All of the different customer code is in the same GIT repository / NSF and we don’t use extension libraries but I do wonder if we should ?

Auto Save

The nature of many of our applications are that, despite training, documents are often left open in edit mode for long periods of time. We developed an auto save feature that saves the “documents” after a configured period, typically 3 minutes. This works well.


We use the opensource DocX4J package to produce high fidelity Word / PDF exports. We also use a HTML rendering process for simpler exports. DocX4J is very powerful but has a lot of moving parts. Rather than converting the Word documents to PDF, which can be problematical, we sometimes leave them in word format but with a random password.

Other stuff

We backup applications to S3 with a simple script that drops Domino briefly and copying the files. DAOS really helps with this.

Things we don’t use and maybe we should :

We do not use the OpenNTF Domino API ( alternative Java ). It is very appealing but has fallen on the wrong side of our wish to keep Domino Vanilla. This is probably the one area that I would be keenest to look at.

We do not use any extension libraries other than the IBM issued ones.

Since you got this far – why do we like XPages

I need to write a longer piece about this but the main reason why we like XPages is that the UI almost comes for free. Developers are almost automatically full stack developers whereas with many other platforms there are Front End and Back End developers. As a small development team ( 4 ) the idea of full stack developers is very appealing.

Other secondary reasons include the versatility and relative simplicity of the Domino server, the security and  the robustness.

But that is all for another day 🙂

Show the product !

I always used to criticise IBM for not “showing the product” so here is a quick screen shot of one of ours 🙂

ICONUK and the State of Domino Web Development

Caution Long Post, I’m stuck on a train.

The best User Group yet.

I am just back from a very impressive ICONUK. I can honestly say that it was the best LUG that I have been to. The best part for me was the informal discussions within the community. It is a time of change and the community discussions were real, interesting and honest.

I came away with more questions than answers but my new questions are better informed and more useful than the set I arrived with.

Why I went

In all honesty I went to ICONUK hoping to understand what platform we should move to ( from the experiences of other community members rather than IBM ). I didn’t come away with an answer but I did came away with a much better understanding of my options and even a new option in the form of ASP.NET MVC ( never thought I would say that ).

XPages does a lot

More importantly I came away with an understanding of just how good Domino / XPages actually is compared to other stacks.

I know that it has some ( fairly serious ) issues but it does do a lot more for us than perhaps we give it credit for, particularly for those of us that need / want to minimise the amount of front end work that we need to do.

I think that the lack of a clear alternative is an indication of how good XPages has been, particularly with RAD ( Rapid Application Development ).

To move to a separate back end and front end requires two different skill sets and synchronisation of changes on both ends whereas with XPages one developer can do it all. Security is much easier (although authentication is often not) and the database , app server and web server are all rolled into one.

So why change ?

So…. lets go back to basics for a minute :
Why do I want / need to find a new stack ? Just to give some context I have no interest in email and run a niche SaaS provider whose solutions are all currently built on XPages.

For me there are 4 drivers :

1 . I cannot take the risk of running my business on a dead platform.

A real concern that the future of XPages is linked to the future of Notes and Notes is rapidly disappearing from the UK.As Notes disappears the IBM resources allocated to XPages are clearly diminishing – I mean the development team, the evangelists, the product managers.

The community resources are also diminishing too.Modern web application platforms need constant maintenance – its not like letting Lotus 123 linger on in the comparative safety of a walled garden until 2014.

IBM sometimes make the point that “XPages is complete” – I very much disagree, try doing source control for real.

2 . XPages is no longer fit for our purposes ( yes we are moving the goal posts ) and there is no demonstrated intent to make it so.

The XPages platform is not moving forwards.IBM have said that XPages is complete so it is only natural that there should be less people working on it ( as there are with lotus script and formula ) – lotuscript and formula have effectively been deprecated by XPages so that is OK – XPages has not been deprecated.

A modern web application server cannot be left at a point in time and called complete.Java 6 and the constraints of the NSF ( compared to Cloudant as an example ) are the biggest demonstration of this. As we develop our competencies with Java ( which many XPage developers have ) we increasingly want to use external modules and improved Java techniques. Having to use a version of Java from 2011 is a constraint and as much as I admire the OpenNTF Java API project it needs to be in the core product.

Likewise we are having to do all sorts of whacky work arounds to deliver faceted searches, activity streams and other rich data streams that other platforms can deliver much better by using modern databases.

It was great to talk to Frank van der Linden who has been working with Cloudant as part of his winning entry to the developer competition. He is full of enthusiasm for what he has discovered in Cloudant – this is great but why is he the only one ? Why wasn’t there a session at ICON from IBM about Cloudant and why is Frank having to write the boiler plate code to integrate Cloudant with Domino and lastly why isn’t Cloudant part of Domino ?

3 . Dojo 2 and passive maintenance

I have a concern that Dojo might become the thing that kills XPages. Did you know that Dojo 2 is due in Fall 2016 – here is the github progress – what plans does IBM have for Dojo 2 ?

I am not a front end developer but it is obvious that a) XPage is hugely dependent on Dojo and b) Dojo is much less loved than it was – what happens when Dojo stops working ? This could be a totally bogus notion so please correct me if I am wrong.

4. Technical Debt

This is about the amount of time that lapses between now and when we move platform.

Every new feature that my team adds to our products creates new technical debt because we will have to re-develop those features on a new platform – we operate a SaaS model and this debt has to be addressed at our cost.

The reality of this is it has taken us 5 iterations of XPage applications to settle on a really good approach and it is inevitable that our first few attempts on a new platform will also need to be re-written several times as we become experts on that platform.

So….. where does this leave me ( and maybe you ).

The first thing to say is that our ( ICS ) situation is in some ways no more severe than those who will need to move from Angular 1 to Angular 2 or those whose wordpress sites no longer run after PHP 5 ( I have 2 such sites ). The pace of change in web application development means that step changes happen and platforms get deprecated – in a way we were spoilt with an IBM platform that could run a version 3 application on a version 9 server with no problems.

Having said that people on Angular 1 can see that there is Angular 2 and this takes us to the crux of my problem.

The crux

The IBM people that I spoke to over the last few days genuinely want to believe that there is a significant announcement going to be made about “Application Development” in Q4. They are asking for our ( the communities ) patience and speak genuinely of understanding that the current situation is unacceptable.

They also speak in a way that acknowledges that the App Dev side of Domino is something in itself with it’s own user base independent of mail accounts. In informal discussions they speak of good reasons why the XPages resources have not been available and how this was a business necessity to deliver other much needed work ( not Toscanna although I have heard rumours otherwise ).

What is clear is that none of the IBMers are “in the know” and that is where I get worried. How can IBM pull a rabbit out of the hat if there has been limited consultation with users or their own business team ? In conversation everyone just defers to Ed Brill as the person who will decide what needs to be delivered.

Now to be fair to IBM none of us saw XPages coming or contributed to that concept ( which in itself came out of Workplace ) and yet it was absolutely the saviour of Notes development in 2011. This is the article I wrote at the time about XWorks ( a very positive thing for me ) but it is interesting to reflect on where we are now as a result of XPages => IBM’s XPages App Server Dilemma – a faster horse, a cash cow or a compelling new concept ?

On the flip side the experiment that was XPages on Bluemix was in my view a miss step because it has soaked up valuable resources and community energy and delivered nothing that is useful today. To be fair to IBM they insist  that it was necessary to show that XPages was still being taken seriously, I disagree but I can see some logic.

What would it take to win me back ?

IBM are asking for our patience while they develop a plan for App Dev.

They claim that they have changed and that our trust will be well placed. What would it take for me to stay with XPages ……… ?

It would take a significant step change in XPages capability in a timely manner and with IBM evangalist support to help the community to exploit the new capabilities ( and stop the haemorrhage of users )

I would want need to see :

  1. A Clouldant / CouchDB style database in the XPages Core product ( on premises ) that leverages the existing Java APIs ( @dblookup etc ) and has new ones so that we could develop performant applications with larger data sets, faceted filtering and dynamic charts ( instead of cached ones ).
  2. Java 8
  3. Source Control fixed ( elimination of time stamps etc ) in the core product – not a hook for an OpenNTF project ( as good as that may be ) – if nothing else to demonstrate that IBM does have ownership of its unfinished sub standard issues.
  4. A technical evangelist who can help us exploit this new functionality.
  5. All of the above by the end of Q3 2017 with demonstrable proof of progress in the mean time so that IBM can earn the trust of their customers.

The Fix Pack V’s Version thing and 2021

I really can’t get excited as to whether it is fix packs or versions as long as it is done well. There are clearly a lot of things that have not been thought through ( what do we call interim fixes ? how is certification handled ? ) but… if it were a robust continuous release process I would have no issues.

IBM claim that is is saving many many hours so lets see them put those man hours to good use. As for 2021 5 years is a long time in IT and in any case if the product stops evolving it is a moot point anyway because it won’t stay fit for purpose as the world moves on.

In Summary

Thanks to Tim Clark and his team – typically they don’t publish a list anywhere – they just deliver :-), the sponsors and the speakers and IBM for hosting the event.

Domino does a hell of a lot. If a new startup took it as a concept to a Silicon Valley venture capitalist I think that they would be impressed. Replacing it is turning out to be more difficult than we thought but as the need to replace it grows more people will rise to the challenge.

IBM is asking for a chance to make things right and say that they understand that it needs to be a significant step change. I have described above what it would take to convince me and am willing to put my scepticism aside and stay positive- but it is a “one shot chance” as an IBMer said today.

What would it take for IBM to convince you ?

URGENT : Chrome Bug 570622 affecting our users – is it affecting yours ?

The bug reported by David Leedy appears to be affecting our users significantly.

Are your users reporting screen freezes in XPage applications in Chrome ?

If so please “star” this bug ( top left corner ) to raise its profile. Even if you are worried that it might affect your users then please “star” this bug.


It has been fixed but it is unclear when it will hit production without enough priority.

There are no symptoms exhibited on the console or in domlog.nsf.

Also note that chrome auto upgrades ( the 27th Jan in this case ) , our enterprise customers started to see this issue this morning.

Thanks, Sean

Update 1 – Google are hoping to provide a fix on Thursday

Update 2 – I have created some instructions for users that you can adapt to suit your own needs.

Update 3 – Wednesday – Google have confirmed that the patch is now being deployed to users 🙂

XPages Document Save Error may be a document locking error

Moral of the story – read the logs

This one had us stumped for a while so I though I would post it.

We had an inconsistent error on an XPage application where it threw a save error even though the Author fields  were correct.

The error was

Exception occurred calling method

After much head scratching followed the advice in log.nsf

please consult error-log-0.xml located in /local/notesdata/domino/workspace/logs

whereupon we discovered the message

Could not save the document 5386 NotesException: Notes error: The document is already locked by someone else.

So document locking was turned on which is not really much use in an XPage application ( we rolled our own application scope based scheme )

So I know it is a pain to connect to the server to get the log documents but it is worthwhile. I wonder could we write an XPage app to recover them ? 🙂