The moral of the story – do some monitoring

We have been battling with one of 6 Domino XPage servers that is running a consistently higher load average than the other servers. The performance was OK from a customer point of view but differences in systems are always a worry.

2016-08-16_08-57-35

We tried everything I could think of – probably spending 2 – 3 days on this overall. Ā We checked the XPage application, we checked Domino configuration, we even moved it to a different host server and we even rebuilt it from a (new) standard image. Anyhow, the problem has been solved šŸ™‚

We recently did a routine update to the linux OS and the issue went away. It seems that there was a bug in the linux release that we were using which –

“Due to prematurely decremented calc_load_task, the calculated load
average was off by up to the number of CPUs in the machine. As a
consequence, job scheduling worked improperly causing a drop in the system
performance –Ā https://rhn.redhat.com/errata/RHSA-2016-0494.html

So the morale of this story

  1. Do some monitoring, that way you can see how updates to the OS and other software are affecting performance.Ā If you roll out an update look for improved or decreased performance – always being on the latest release is not necessarily the solution.
  2. NeverĀ build a standard recovery image and assume that it will run correctly in operation unless you have run it in production. This server was firstĀ based on our latest build template ( including a new OS version – which was broken ) and when we rebuilt it we used that image again ( still with the broken OS )

We areĀ using the Opsview Atom monitoring suite which is based on the Nagios open source platform. It is fairy good ( and well pricedĀ ). I will be postĀ some articles on it soon.

loadaverage

Chuffed to bits – our first XPage product goes live

I started FoCul way back in 2000. At the time I was an experienced Industrial Engineer with a passion for process improvement  and who thought he knew about IT and Lotus Notes in particular ( Donald Rumsfeld has a saying about that )

For many years we went from exciting project to exciting project doing some really good stuff but not really thinking about how we could turn these great bespoke projects into repeatable solutions.

We were also in a double niche, customers from the manufacturing sector who had an established Notes Infrastructure.

Fast forward 10 years and today is the start of a new era. We have just launched our XPage based “Management of Change for Process Safety” product and our lead customer is nearing the end of their evaluation and are really pleased.

Try It Now

Not only that but they are a non notes shop ( actually an ex Notes shop ) running the application on a black box linux based virtual appliance on VMware. We have an “Application Specific Licence ( ASL ) ”  in place so for Non Notes shops the cost of the Domino server is bundled into the cost of the application or the SAAS service, the customer doesn’t even need to worry about IBM licensing ( although PVUs are still an issue ).

I remember the feeling of excitement back in 2002 the first time I designed a database that was dual browser / Notes client. It seemed like a whole new set of possibilities had opened up. In reality it was somewhat of a false start. The web functionality wasn’t RAD and the cost of dual browser / client development seldom made sense for my customers. For the last 8 years I have been eagerly anticipating RAD web and I really do think that it is here ( or almost here ) and improving day by day. I never ever anticipated that it would be truly a case of coding once for the web and for the client at the same time !

XPages has been a steep learning curve. I first started trying to develop for XPages in 8.5.0 and it was a pig. As a traditional Notes Developer ( i.e. not really that good at browser stuff ) I was never sure if it was me or bugs in XPages. The DDE client was unstable, it was slow and to be honest in hindsight I am so pleased that we didn’t win any commercial XPages work because it would have been torrid trying to develop a robust application in an affordable time. Myself and a colleague did eventually develop an in house help-desk system but it was tough going.

Roll on to 2010 and we had Domino 852, DDE 852, a really vibrant community offering loads of examples and some inspired sites such as IdeaJam, YouatNotes, Xtalk, Xpages Wiki, Discussion Forum etc.. There are also training resources like XPages101 and Notesin9. And, thank God, there is a book on the way ( still patchy IBM documentation but that’s a whole other discussion about how there can be so much IBM content and  yet there are still big gaps ). I know that my colleague Andrew Champion ( who has done most of the coding ) has found it challenging but it has been manageable this time around.

So to sum it all up I am really chuffed to be here. We can still get to do our niche work, we still get to work with Domino ( which I believe gives us a great set of advantages ) but we can do it with a wider set of customers through a range of delivery mechanisms.

A big Thank You to IBM and in particular to the XPages team and a big Thank You to the Lotus Community. We couldn’t have done it without you !