How we use XPages

Cameron, Jesse and Paul have been putting a lot of effort into explaining how they use XPages and it has prompted me to post this on how we at FoCul  use XPages, something I have been meaning to do for a while. I will hopefully post more about why we like XPages so much but this is a long enough read for now.

How it started

18 years ago I started a small Notes based development house supplying bespoke Notes applications to Engineers in the chemical sector. I had been an industrial Engineer in this sector and was frustrated by a lack of collaboration and knowledge management solutions available. This went really well ( paid the bills and I enjoyed it immensely ) until 2008 when a combination of the recession and the dwindling Notes install base caused 50% of our work to disappear very quickly.

We took the enforced free time and took the best of our bespoke Notes solutions and offered them as XPage based SaaS solutions. Being web based they were much more expensive to develop and less reliable but it transformed the business into one where 85% + of our revenue is recurring and we have a much more natural relationship with our clients where there is a strong ongoing relationship that is entirely based upon the success of the products in everyday use.

In the previous bespoke world I had always struggled to let go of the projects knowing that just 10% more effort on application tuning during the first 6 months of use would make a 25% improvement in the usefulness of the application. In the SaaS world continuing to invest in the success of the application is now our decision and one that typically pays off.

We are still 100% on Domino / XPages and having settled on a good application architecture it works very well for us.

Infrastructure

Each customer has one or more Domino servers, typically licenced via XWorks. The servers are virtual servers running on VMWare or increasingly on MS Azure. The Operating system of choice was CentOS Linux but we are mid migration to Windows Server. I personally would not have predicted this migration and I will write about it another time. The crux was support for CentOS on Azure and who owns an issue when IBM ( as it was ) blames the OS for the problems and MS owns the ”hardware” and some MS utilities that are required to make it work.

Domino Version

Our applications all run on Domino 9.x and we have not yet had the confidence to move to 10.x. The issue is the perception of a loss of reliability and developer efficiency but to be fair to HCL we have not yet invested much ( @ 3 man days ) in trying 10.x and that was mostly trying DQL.

We were a fairly early adopter of XPages and although there was good IBM sponsored developer evangelism we still found it very tough. As a direct consequence we have a policy of staying away from the bleeding edge where there is an alternative. Having said that the architecture that we settled on for XPages is pretty advanced in some respects.

Our applications

Our applications are the good old fashioned bread and butter collaboration and workflow applications that naturally found their home on Lotus Notes. Things like Shift Logs or Management of Change workflow applications for the chemical and utility sectors. We have SaaS applications with over 1M records but most have less than 50 documents created per day and less than 50 concurrent users. We also have the luxury of very few applications per server compared to most Domino servers.

How are applications are architected

We are on our 5th iteration of an XPages architecture having started with SSJS and a drag & drop approach to XPages. The current architecture has the following attributes :

  • We use Java for 90% of the code, all edited through Domino Designer.
  • Objects on the screen are bound to Managed Java Bean based representations of “documents” rather than directly to documents.
  • “Views” or lists of records are constructed from Full Text Searches rather than views – we have very few notes views relating to what users see.
  • Almost all user “views” have faceted search capability to help users filter down to the records that they want to work with.
  • The “forms” in the application are almost all generated via a “form builder” process in the application admin area where new tabs and fields can be added. This configuration is typically carried out by ourselves as opposed to customers.
  • We use an auto save feature on documents
  • We use either DocX4J to export data to Word and PDF.
  • Other processes can access the data via configuration driven REST APIs or XML feeds.

Taking these one by one :

Why Java ?

In the early days we needed to have quite sophisticated word / PDF export capability and we brought in Ben Poole to help us deliver this using DocX4J. Ben used Java for this and from working with Ben we came to understand the power of Java over SSJS. The availability of 3rd party libraries like DocX4J was an interesting part of this.

Now almost all of our code is now Java and from a manager’s perspective it has been lovely to see how the developers have evolved from the typical “Line of Business” Notes developers into much richer developers. This evolution has given me confidence that we can adapt to whatever the next platform is.

Why represent documents with Java Beans ?

We found that our “documents” were increasingly composite documents from multiple data sources and that we needed more control over how documents were saved. Another important factor was that because forms were based on a configuration rather than being hard coded ( via the “form builder” ) it was important to have quick cached access to the data representing the form structure.

Having become comfortable with Java it was a natural move to use Managed Java Beans and we were initially inspired by Karsten Lehmann. This made it much easier for us to work with the data and control how users edited it. The representation of “view” data is explained below.

Why did we move to Full Text Search to display “view data”

The biggest reason that we did this was to allow users to have “Faceted Search” capabilities when displaying “views”. This is a search where the search terms are progressively reduced as the user add more search criteria.
A strong secondary reason was that because the forms are configured using a form builder a field which has data key to a business process could have any field name and does not appear in a view.

Displaying a list of users, as an example, would go like this :

  • Skinny Bean :
    The records that are going to be displayed are collected by two searches. The first is a full text search and the 2nd is a formula search of the un-indexed documents. The 2nd search has to be relatively small for this method to work and we force an index update @5 minutes.Having collected the documents the “skinny bean” is constructed to represent the documents in such a way that they can be sorted for a view – only the minimum “columns” are added to the bean. To do this we go to each document and extract items of data – this is the time consuming part.
  • View Bean :
    Having a representation of the ordered document list we can now create a “view bean” which represents a page of the view – so between 25 and 100 documents. To do this data about each view column is constructed but this is only done for a page at a time as part of the view rendering process. This will typically include icons and hyperlinks.
  • Page Bean :
    When a user clicks into a row and as the document is displayed a Page Bean is constructed. This represents the document and is only constructed upon demand. The data may be pulled from multiple back end documents in a process that is a bit like SQL joins. If the “document” is edited then the edits get pushed back into the back end documents at the end of the editing process. The source documents will typically get locked to other edits during this process.

While this approach may seem complicated the same approach is used in most of our SaaS products so it is one code stream that is maintained in one place across all of these applications. It allows us to be able to produce some very powerful applications very quickly – mostly through configuration rather than coding.

Why Faceted Searches ?

Customers expect modern web applications. As our customer base is mostly Engineering types they do tend to push the boundaries in terms of wanting to cut and dice data. An interface without faceted search would have been less successful.

The trade off is obviously speed and this is further exacerbated by the use of a form builder to configure the forms – there is very limited opportunity for tailored views to make the display of data more efficient. Again commonality across most of our applications is key here as the same code is used very widely but maintained centrally

Why use a form builder approach ?

Although we have multiple products they almost all share the same code base – as in they are all based on the same NSF. To be fair we have to work quite hard to make sure that it does not become bloated but the advantages are obvious.

A key enabler of this approach is to configure applications rather than hard code them. To do this we developed a form configuration module within the web facing application admin area where new forms can be built tab by tab and field by field. We did initially try to store copies of this configuration on the actual documents but we quickly exceeded the available data volumes and that started us down the Java Bean approach that we now use.

The customers ( those Engineer types again ) love this approach because we can quickly add or remove fields from forms on the fly. A configuration re-visioning process means that the historical documents will still use the old configuration while new documents will use the modified version.
This approach also has the advantage that we can construct simplified export version of the forms ( word or PDF ) based on their configuration rather than having to hard code the exports.

We do fairly often need to resort to custom elements within the configured forms. We reference these bespoke custom controls in the form builder and they get inserted into the form when rendered just as a configured field would. Increasingly we are looking to have a suite of configurable custom controls that we can use.

We also need custom workflow code in some deployments. To do this we have a series of defined points within our code where we will go off and check a coded routine. If a particular customer needs something special then the routine will have custom code otherwise it is empty. We use a value within the faces-config.xml to define which “plug-in” to use. I am not an expert in Java but I think of it as a crude equivalent to overloading. All of the different customer code is in the same GIT repository / NSF and we don’t use extension libraries but I do wonder if we should ?

Auto Save

The nature of many of our applications are that, despite training, documents are often left open in edit mode for long periods of time. We developed an auto save feature that saves the “documents” after a configured period, typically 3 minutes. This works well.

DocX4J

We use the opensource DocX4J package to produce high fidelity Word / PDF exports. We also use a HTML rendering process for simpler exports. DocX4J is very powerful but has a lot of moving parts. Rather than converting the Word documents to PDF, which can be problematical, we sometimes leave them in word format but with a random password.

Other stuff

We backup applications to S3 with a simple script that drops Domino briefly and copying the files. DAOS really helps with this.

Things we don’t use and maybe we should :

We do not use the OpenNTF Domino API ( alternative Java ). It is very appealing but has fallen on the wrong side of our wish to keep Domino Vanilla. This is probably the one area that I would be keenest to look at.

We do not use any extension libraries other than the IBM issued ones.

Since you got this far – why do we like XPages

I need to write a longer piece about this but the main reason why we like XPages is that the UI almost comes for free. Developers are almost automatically full stack developers whereas with many other platforms there are Front End and Back End developers. As a small development team ( 4 ) the idea of full stack developers is very appealing.

Other secondary reasons include the versatility and relative simplicity of the Domino server, the security and  the robustness.

But that is all for another day 🙂

Show the product !

I always used to criticise IBM for not “showing the product” so here is a quick screen shot of one of ours 🙂

The moral of the story – do some monitoring

We have been battling with one of 6 Domino XPage servers that is running a consistently higher load average than the other servers. The performance was OK from a customer point of view but differences in systems are always a worry.

2016-08-16_08-57-35

We tried everything I could think of – probably spending 2 – 3 days on this overall.  We checked the XPage application, we checked Domino configuration, we even moved it to a different host server and we even rebuilt it from a (new) standard image. Anyhow, the problem has been solved 🙂

We recently did a routine update to the linux OS and the issue went away. It seems that there was a bug in the linux release that we were using which –

“Due to prematurely decremented calc_load_task, the calculated load
average was off by up to the number of CPUs in the machine. As a
consequence, job scheduling worked improperly causing a drop in the system
performance – https://rhn.redhat.com/errata/RHSA-2016-0494.html

So the morale of this story

  1. Do some monitoring, that way you can see how updates to the OS and other software are affecting performance. If you roll out an update look for improved or decreased performance – always being on the latest release is not necessarily the solution.
  2. Never build a standard recovery image and assume that it will run correctly in operation unless you have run it in production. This server was first based on our latest build template ( including a new OS version – which was broken ) and when we rebuilt it we used that image again ( still with the broken OS )

We are using the Opsview Atom monitoring suite which is based on the Nagios open source platform. It is fairy good ( and well priced ). I will be post some articles on it soon.

loadaverage

Chuffed to bits – our first XPage product goes live

I started FoCul way back in 2000. At the time I was an experienced Industrial Engineer with a passion for process improvement  and who thought he knew about IT and Lotus Notes in particular ( Donald Rumsfeld has a saying about that )

For many years we went from exciting project to exciting project doing some really good stuff but not really thinking about how we could turn these great bespoke projects into repeatable solutions.

We were also in a double niche, customers from the manufacturing sector who had an established Notes Infrastructure.

Fast forward 10 years and today is the start of a new era. We have just launched our XPage based “Management of Change for Process Safety” product and our lead customer is nearing the end of their evaluation and are really pleased.

Try It Now

Not only that but they are a non notes shop ( actually an ex Notes shop ) running the application on a black box linux based virtual appliance on VMware. We have an “Application Specific Licence ( ASL ) ”  in place so for Non Notes shops the cost of the Domino server is bundled into the cost of the application or the SAAS service, the customer doesn’t even need to worry about IBM licensing ( although PVUs are still an issue ).

I remember the feeling of excitement back in 2002 the first time I designed a database that was dual browser / Notes client. It seemed like a whole new set of possibilities had opened up. In reality it was somewhat of a false start. The web functionality wasn’t RAD and the cost of dual browser / client development seldom made sense for my customers. For the last 8 years I have been eagerly anticipating RAD web and I really do think that it is here ( or almost here ) and improving day by day. I never ever anticipated that it would be truly a case of coding once for the web and for the client at the same time !

XPages has been a steep learning curve. I first started trying to develop for XPages in 8.5.0 and it was a pig. As a traditional Notes Developer ( i.e. not really that good at browser stuff ) I was never sure if it was me or bugs in XPages. The DDE client was unstable, it was slow and to be honest in hindsight I am so pleased that we didn’t win any commercial XPages work because it would have been torrid trying to develop a robust application in an affordable time. Myself and a colleague did eventually develop an in house help-desk system but it was tough going.

Roll on to 2010 and we had Domino 852, DDE 852, a really vibrant community offering loads of examples and some inspired sites such as IdeaJam, YouatNotes, Xtalk, Xpages Wiki, Discussion Forum etc.. There are also training resources like XPages101 and Notesin9. And, thank God, there is a book on the way ( still patchy IBM documentation but that’s a whole other discussion about how there can be so much IBM content and  yet there are still big gaps ). I know that my colleague Andrew Champion ( who has done most of the coding ) has found it challenging but it has been manageable this time around.

So to sum it all up I am really chuffed to be here. We can still get to do our niche work, we still get to work with Domino ( which I believe gives us a great set of advantages ) but we can do it with a wider set of customers through a range of delivery mechanisms.

A big Thank You to IBM and in particular to the XPages team and a big Thank You to the Lotus Community. We couldn’t have done it without you !