Cameron, Jesse and Paul have been putting a lot of effort into explaining how they use XPages and it has prompted me to post this on how we at FoCul use XPages, something I have been meaning to do for a while. I will hopefully post more about why we like XPages so much but this is a long enough read for now.
How it started
18 years ago I started a small Notes based development house supplying bespoke Notes applications to Engineers in the chemical sector. I had been an industrial Engineer in this sector and was frustrated by a lack of collaboration and knowledge management solutions available. This went really well ( paid the bills and I enjoyed it immensely ) until 2008 when a combination of the recession and the dwindling Notes install base caused 50% of our work to disappear very quickly.
We took the enforced free time and took the best of our bespoke Notes solutions and offered them as XPage based SaaS solutions. Being web based they were much more expensive to develop and less reliable but it transformed the business into one where 85% + of our revenue is recurring and we have a much more natural relationship with our clients where there is a strong ongoing relationship that is entirely based upon the success of the products in everyday use.
In the previous bespoke world I had always struggled to let go of the projects knowing that just 10% more effort on application tuning during the first 6 months of use would make a 25% improvement in the usefulness of the application. In the SaaS world continuing to invest in the success of the application is now our decision and one that typically pays off.
We are still 100% on Domino / XPages and having settled on a good application architecture it works very well for us.
Infrastructure
Each customer has one or more Domino servers, typically licenced via XWorks. The servers are virtual servers running on VMWare or increasingly on MS Azure. The Operating system of choice was CentOS Linux but we are mid migration to Windows Server. I personally would not have predicted this migration and I will write about it another time. The crux was support for CentOS on Azure and who owns an issue when IBM ( as it was ) blames the OS for the problems and MS owns the ”hardware” and some MS utilities that are required to make it work.
Domino Version
Our applications all run on Domino 9.x and we have not yet had the confidence to move to 10.x. The issue is the perception of a loss of reliability and developer efficiency but to be fair to HCL we have not yet invested much ( @ 3 man days ) in trying 10.x and that was mostly trying DQL.
We were a fairly early adopter of XPages and although there was good IBM sponsored developer evangelism we still found it very tough. As a direct consequence we have a policy of staying away from the bleeding edge where there is an alternative. Having said that the architecture that we settled on for XPages is pretty advanced in some respects.
Our applications
Our applications are the good old fashioned bread and butter collaboration and workflow applications that naturally found their home on Lotus Notes. Things like Shift Logs or Management of Change workflow applications for the chemical and utility sectors. We have SaaS applications with over 1M records but most have less than 50 documents created per day and less than 50 concurrent users. We also have the luxury of very few applications per server compared to most Domino servers.
How are applications are architected
We are on our 5th iteration of an XPages architecture having started with SSJS and a drag & drop approach to XPages. The current architecture has the following attributes :
- We use Java for 90% of the code, all edited through Domino Designer.
- Objects on the screen are bound to Managed Java Bean based representations of “documents” rather than directly to documents.
- “Views” or lists of records are constructed from Full Text Searches rather than views – we have very few notes views relating to what users see.
- Almost all user “views” have faceted search capability to help users filter down to the records that they want to work with.
- The “forms” in the application are almost all generated via a “form builder” process in the application admin area where new tabs and fields can be added. This configuration is typically carried out by ourselves as opposed to customers.
- We use an auto save feature on documents
- We use either DocX4J to export data to Word and PDF.
- Other processes can access the data via configuration driven REST APIs or XML feeds.
Taking these one by one :
Why Java ?
In the early days we needed to have quite sophisticated word / PDF export capability and we brought in Ben Poole to help us deliver this using DocX4J. Ben used Java for this and from working with Ben we came to understand the power of Java over SSJS. The availability of 3rd party libraries like DocX4J was an interesting part of this.
Now almost all of our code is now Java and from a manager’s perspective it has been lovely to see how the developers have evolved from the typical “Line of Business” Notes developers into much richer developers. This evolution has given me confidence that we can adapt to whatever the next platform is.
Why represent documents with Java Beans ?
We found that our “documents” were increasingly composite documents from multiple data sources and that we needed more control over how documents were saved. Another important factor was that because forms were based on a configuration rather than being hard coded ( via the “form builder” ) it was important to have quick cached access to the data representing the form structure.
Having become comfortable with Java it was a natural move to use Managed Java Beans and we were initially inspired by Karsten Lehmann. This made it much easier for us to work with the data and control how users edited it. The representation of “view” data is explained below.
[Edit] We also found this series of articles by Gary Glickman very useful in setting out our Java architecture.
Why did we move to Full Text Search to display “view data”
The biggest reason that we did this was to allow users to have “Faceted Search” capabilities when displaying “views”. This is a search where the search terms are progressively reduced as the user add more search criteria.
A strong secondary reason was that because the forms are configured using a form builder a field which has data key to a business process could have any field name and does not appear in a view.
Displaying a list of users, as an example, would go like this :
- Skinny Bean :
The records that are going to be displayed are collected by two searches. The first is a full text search and the 2nd is a formula search of the un-indexed documents. The 2nd search has to be relatively small for this method to work and we force an index update @5 minutes.Having collected the documents the “skinny bean” is constructed to represent the documents in such a way that they can be sorted for a view – only the minimum “columns” are added to the bean. To do this we go to each document and extract items of data – this is the time consuming part. - View Bean :
Having a representation of the ordered document list we can now create a “view bean” which represents a page of the view – so between 25 and 100 documents. To do this data about each view column is constructed but this is only done for a page at a time as part of the view rendering process. This will typically include icons and hyperlinks. - Page Bean :
When a user clicks into a row and as the document is displayed a Page Bean is constructed. This represents the document and is only constructed upon demand. The data may be pulled from multiple back end documents in a process that is a bit like SQL joins. If the “document” is edited then the edits get pushed back into the back end documents at the end of the editing process. The source documents will typically get locked to other edits during this process.
While this approach may seem complicated the same approach is used in most of our SaaS products so it is one code stream that is maintained in one place across all of these applications. It allows us to be able to produce some very powerful applications very quickly – mostly through configuration rather than coding.
Why Faceted Searches ?
Customers expect modern web applications. As our customer base is mostly Engineering types they do tend to push the boundaries in terms of wanting to cut and dice data. An interface without faceted search would have been less successful.
The trade off is obviously speed and this is further exacerbated by the use of a form builder to configure the forms – there is very limited opportunity for tailored views to make the display of data more efficient. Again commonality across most of our applications is key here as the same code is used very widely but maintained centrally
Why use a form builder approach ?
Although we have multiple products they almost all share the same code base – as in they are all based on the same NSF. To be fair we have to work quite hard to make sure that it does not become bloated but the advantages are obvious.
A key enabler of this approach is to configure applications rather than hard code them. To do this we developed a form configuration module within the web facing application admin area where new forms can be built tab by tab and field by field. We did initially try to store copies of this configuration on the actual documents but we quickly exceeded the available data volumes and that started us down the Java Bean approach that we now use.
The customers ( those Engineer types again ) love this approach because we can quickly add or remove fields from forms on the fly. A configuration re-visioning process means that the historical documents will still use the old configuration while new documents will use the modified version.
This approach also has the advantage that we can construct simplified export version of the forms ( word or PDF ) based on their configuration rather than having to hard code the exports.
We do fairly often need to resort to custom elements within the configured forms. We reference these bespoke custom controls in the form builder and they get inserted into the form when rendered just as a configured field would. Increasingly we are looking to have a suite of configurable custom controls that we can use.
We also need custom workflow code in some deployments. To do this we have a series of defined points within our code where we will go off and check a coded routine. If a particular customer needs something special then the routine will have custom code otherwise it is empty. We use a value within the faces-config.xml to define which “plug-in” to use. I am not an expert in Java but I think of it as a crude equivalent to overloading. All of the different customer code is in the same GIT repository / NSF and we don’t use extension libraries but I do wonder if we should ?
Auto Save
The nature of many of our applications are that, despite training, documents are often left open in edit mode for long periods of time. We developed an auto save feature that saves the “documents” after a configured period, typically 3 minutes. This works well.
DocX4J
We use the opensource DocX4J package to produce high fidelity Word / PDF exports. We also use a HTML rendering process for simpler exports. DocX4J is very powerful but has a lot of moving parts. Rather than converting the Word documents to PDF, which can be problematical, we sometimes leave them in word format but with a random password.
Other stuff
We backup applications to S3 with a simple script that drops Domino briefly and copying the files. DAOS really helps with this.
Things we don’t use and maybe we should :
We do not use the OpenNTF Domino API ( alternative Java ). It is very appealing but has fallen on the wrong side of our wish to keep Domino Vanilla. This is probably the one area that I would be keenest to look at.
We do not use any extension libraries other than the IBM issued ones.
Since you got this far – why do we like XPages
I need to write a longer piece about this but the main reason why we like XPages is that the UI almost comes for free. Developers are almost automatically full stack developers whereas with many other platforms there are Front End and Back End developers. As a small development team ( 4 ) the idea of full stack developers is very appealing.
Other secondary reasons include the versatility and relative simplicity of the Domino server, the security and the robustness.
But that is all for another day 🙂
Show the product !
I always used to criticise IBM for not “showing the product” so here is a quick screen shot of one of ours 🙂
