update : there is now also an official IBM Domino AMI and a wiki article although I think it will take some work to make it persistent.
After reading Michael Brown’s excellent article about setting up an “Ephemeral” Domino server on EC2 I decided to look at using EC2 a bit harder. This blog is now running on EC2 and I have been very very impressed with the functionality that this service provides. I would obviously like to test the system more but early indications are that it has good potential for VSMB and SAAS Domino offerings.
When you follow Michael’s guide you get a Domino server which uses the “Ephemeral” disk which is automatically deleted when the the instance is terminated. An alternative approach is to build the system using the Amazon Elastic Block Storage ( EBS ) system for storage. These EBS volumes have the potential ( note the word potential ) to remain intact after the instance is terminated. This means that should the instance fail your data will remain intact. It also means that you can take advantage of options to use “spot pricing” and to resize your server.
This article will take you through a method to set up such a server using Ubuntu 8.04. I am not an expert in Linux or EC2 so I would appreciate your comments on any improvements that I could make to the scheme. You should also note that as yet there is no licence for Domino on EC2 but the licences for DB2 usage suggest that a small EC2 instance is 50 PVU points. Ed Brill has posted of a forthcoming licence but this is intended for “Development and Proof of Concept” usage.
Michael covers setting up the Amazon account, it is really straight forward. The console is a great example what can be achieved with CSS.
The Scheme
Once you have an account you can set up an instance. These instances are usually set up from a images called AMIs. There are many images to choose from but I choose to use Ubuntu 8.04 because we already use that for other servers. This was probably a more complicated choice than Michael’s Fedora but I wanted to be consistent with what we already do here.
A recent feature in EC2 is the idea of instances based on bootable EBS volumes which can ( but not always ) retain data after the instance is terminated. Although this seemed like a very straight forward concept there are some points of detail that I originally missed and which are important.
With the introduction of bootable EBS instances Amazon also introduced the ability to stop and start an instance as well as terminating an instance. You are not charged when your instance is stopped. The important point to note here is that the default EBS will retain its data while stopped but will loose its data when terminated. There is a way to stop this using the EC2 command line options but in this article I want to present a scheme that is straight forward for any admin to try ( including myself ! ).
The scheme I settled on ( any improvements welcomed ) was to create an EBS boot instance and have a second EBS volume which contained the Notes data. This allows me to terminate the instance and then restart it ( or a clone of it ) with no data loss. Termination is required if you want to take advantage of spot pricing or change the specification of server you are using in terms of CPU or memory. The production instance is based on an AMI which was created AFTER Domino was installed and configured. You could actually use this approach with an ephermeral boot instance and an EBS data volume.
Public ami-7fc6ed0b >> Instance >> Domino Installed >> Domino Configured >> Private ami -xxxxxxxxx >> Instance(s)
An “elastic IP” address was used to provide a static address for the server.
I got to this scheme after
1) trying a single volume approach and then realising that the EBS will reset and loose its data after a termination.
2) realising that although the EC2 snapshot facility is really powerful there is no easy way to restore a snapshot as an EBS boot instance whereas it is very easy to do this for a dedicated data volume.
Getting the Domino Install files
I found the easiest way to get the Domino install files onto the new instance was to fire up an off the shelf instance of a Linux Desktop so that I could download the installation files from the IBM Passport Advantage or Business Partner web portals. Having downloaded the files I then created the data volume that I would later use on the Domino server, mounted it on this temporary instance and saved the files for later use. You could use FTP or winSCP to upload them to the server instance but my upstream bandwidth is pretty limited.
I created the desktop instance using this AMI. ami-94446fe0 Note that the AMI codes are different for different regions – this one is eu-west-1.
login using putty and the username root and your existing key pair. Note that you will have to use puttygen to convert the Amazon keypair first.
Update the instance with any updates.
>>apt-get update
now set up a temporary user as the NXClient does not have the keypairs installed and I just want to use password authentication
>>user-setup
Now start the NXClient and login using the new account
Created a new volume in EC2 using the EC2 management console. I sized this to be my data drive for the domino server.
Now attach the volume using the EC2 management console Once it is attached in /dev/sdl ( or wherever ) you need to format it in a terminal window
>> sudo mkfs -t ext2 /dev/sdl
You can then make a directory and mount it
>>mkdir temp
>>sudo mount /dev/sdl /temp
I found that at this point the volume is owned by root so I had to chown it to my temp user acoount. There is probably a more graceful way of doing this.
>>sudo chown tempuser:tempuser temp
Now you can launch firefox and download your installation files from Passport Advantage onto this volume for use on the new Domino server later.
When you are finished unmount the volume >> sudo umount /dev/sdl ( note the spelling of umount )
Then go to the ec2 management console and detach this volume.
Then terminate your instance.
Creating the master Domino instance
I created a new bootable EBS image using ami-7fc6ed0b, my firewall rules and key pairs ( the username is ubuntu ) . Once the image had booted I attached the data volume to the instance as /dev/sdl ( note that this is sdL not sd1 ).
Don’t forget to update
>> sudo apt-get update
I created a
directory called local and mounted the data drive into this directory using the command
>>sudo mount /dev/sdl /local/
Then I changed the ownership to notes
>> sudo chown -R notes:notes /local
I installed Domino using my installation files as described by Michael but note that as Ubuntu does not have a root user you need to take some extra steps during the install.
Create the notes user and group
>>sudo useradd -m notes ( should create group at the same time )
>>sudo passwd notes
Now as per Michael’s scheme you need to copy the keypairs from the existing user ( in this case ubuntu ) to /home/notes/.ssh and then assign ownership to Notes. Notes that in this image of Ubuntu you are copying from the /home/ubuntu/.ssh directory.
from the /home/ubuntu directory do a directory listing using
>>ls -a . This shows that there is a .ssh directory
now copy the files to /home/notes/.ssh
>>sudo cp /home/ubuntu/.ssh/authorized_keys /home/notes/.ssh
>> sudo chown -R notes:notes /home/notes
Now expand the Domino install files as shown below
>>cd /local
>>sudo tar -xvf XXXXXXX.tar
now go to the expanded folders until you find the install file
>>sudo su <- this is important on ubuntu
>>./install
I then used WinSCP to copy the server the server ID file to the local/notesdata directory while logged in as the user notes. This ensures that the id file belongs to the Notes account.
Now complete the Notes configuration using the remote setup console as described by Michael.
Next I installed Daniel Nasheds startup scripts. These make it much easier to start, stop and monitor a domino server on Linux. The readme has the latest instructuons.
copy rc_domino to /etc/init.d ( use sudo )
copy rc_domino_script to /opt/ibm/lotus
>>cd /opt/ibm/lotus
>>chmod 755 rc_domino_script
>> cd /etc/init.d
>> chmod 755 rc_domino
Then set run level
>> sudo update-rc.d rc_domino defaults
you can now start Domino using the command /etc/init.d/rc_domino start
/etc/init.d/rc_somino stop and /etc/init.d/rc_domino monitor are also very useful
You also need to tell the server to mount the data volume at startup. As Ubuntu doesn’t give the option to use root access with Winscp the easiest way to do this is to use the vi editor
>> sudo vi /etc/fstab adding the line /dev/sdl /local etx2 defaults 0 0
Once the Domino server is just how you would like it ( don’t forget to set the correct host name in the server document and on the ports ) you need to make a new private AMI of this ready to run server.
Shut down Domino and Linux and then stop the instance ( NOT Terminate ). Now create a snapshot of the 15GB EBS boot volume and convert that to a private AMI so that when you terminate the instance you can get back to this point easily.
Now terminate the instance and create a new one using your own private AMI. Because the instance will start before you get the chance to attach the data volume you will need to stop it and attach the volume via the EC2 control panel – make sure that you use the same designation as earlier e.g /dev/sdl. When you now restart the instance it will automatically mount the data drive and launch Domino successfully.
You will also have to re-assign the elastic IP address. Note that I also found that I had to unassign and then reassign the elastic IP address anytime I rebooted the server from the linux console – I am not sure if this is a feature or how it should be.
With the server running you can take snapshots of the data drive as often as you want to. These can be used to restore data as they can be mounted as an additional volume and the files accessed. You can stop the server at will ( and your charges will stop ) or you can terminate the server and then quickly create a new instance. You can also create other instances from the same image meaning that you could very quickly establish a cluster pair as an example.
Setting up inbound and outbound email
There are quite a few details to be understood here and I am not yet convinced that all of them are sorted satisfactorily.
1) You will need to open port 25 in your firewall rules. You can do this via the ec2 control panel and the change is instantaneous
2) Amazon add all of their IP addresses to the black list at Spamhaus – so any emails that you send out will be blacklisted. You need to ask them to remove it from the black list using the form on this page. They send you a copy as soon as you try sending mail
3) You will need to use the same form to set up a reverse DNS which also means that you need a valid A record in place for your elastic IP
4) You may need to add the physical IP address of the network connection to the internet site document for SMTP.
This is the one that I am not convinced about ( and any advise would be gratefully received ) . Initially I was getting “Authentication is not enabled in the SMTP Internet Site Document” even though I had one set up ( and a Global settings document ). The move to the EC2 server did also coincide with a move to using “Internet Site Documents” as we no longer run Sametime so it may be that I have configured something wrong. The work around seems to be to add the actual network address to the SMTP internet site document. You can get this by using >>ifconfig in the console. The behaviour does not seem terribly consistent so I need to explore further. This restriction will add another task to your short list of things to configure when you launch a new instance.
update
I recently switched servers and this was an issue again. I definitely had to insert the local IP address for the server.
This also seemed to require a full Server restart
Costs
The costs of running the server are made up of four components
1) you need an IBM licence for the server which I have assumed you already had
2) the EBS volumes have a cost
3) the CPU usage has a cost and
4) there are data transfer costs.
Although these are clearly set out by Amazon It was not clear how much a chatty Domino server would cost overall – this was part of my wanting to try out the system – time shall tell.
One interesting point is the spot market pricing for CPU usage. I must be missing a trick here because there seems to be a loop hole. Normally an hour of CPU usage in Europe would be 0.095 dollars on the flat rate ( not yet sure about the VAT component of this ).
If you pay a fixed up front sum for 1 or 3 years you can get a reduced tariff of 0.045 dollars per hour. On the spot market you bid a price and your server r
uns while the price is below your cap – it terminates ( not stops ) once the price exceeds the cap. Historically this price has been below 0.045 dollars and in fact the most you can bid is 0.045 dollars. So I am running this instance you are looking at for <= 0.045 dollars with no up front payment - how does that work ?
Performance
Firstly let me say that this is not an enterprise deployment – it runs www.focul,.net, www.deliverytoolkit.com, this blog and is a demonstration and development server. Having said that the performance so far is at least on a par with the physical server that it replaces which was a 4 year old single cpu 2.8hz 1.5gb machine on Windows server 2003 ( yes I know not exactly a speed demon either ).
I have not turned on Transaction Logging as I am not sure what impact it would have ( no real spindles ) so that is an experiment for another day. The Amazon cloudwatch system ( a paid for extra ) is actually very basic showing CPU utilisation and network traffic. CPU Utilisation is low and memory utilisation ( 1.7 GB ) is over 95% but that is how Linux works. I’ll post the trends next week.
If anyone would like any more specific metrics please ask, I am interested in knowing what your impressions of the performance are.
Well that’s it, a bit long and rambling but hopefully of use. Please let me have any ideas or corrections so that I can update the article.