A couple of weeks ago, I introduced the server environment we’re going to build for the project. Today I’m going to get into the details and share some experiences.
We purchased four IBM HS23E (Type 8038) blade servers and installed them into our existing IBM BladeCenter-E chassis. The plan was also to use an external IBM DS4500 disk array with a fibre channel network to provide the storage capacity, high availability and scalability. Each server has two optical ports which are both connected to a different controller in the disk array via a switch. All the servers have a small dedicated volume for the operating system and such, and a large shared volume for virtual machines, configurations and backups. While we knew the design was possible, we didn’t have a clear picture on how to set up Ubuntu Server with the said configuration.
Some of the Ubuntu Server releases and IBM hardware are certified to work together. Canonical has published a list of certified combinations:
As expected, there were several issues which we needed to solve during the installation. First up, we needed the latest firmware update for the servers. Of course the installation package wouldn’t fit a CD and the media bay didn’t have a DVD drive. Also the USB boot didn’t work for some reason still unknown to date. While waiting for an upgrade drive to arrive for the media bay, we continued with the old firmware. It could be one possible cause for future issues we experienced. Next we tried to install Ubuntu from a standard installation CD. But the installer didn’t recognize any of the attached logical disks. We triple checked storage configurations and the documentation, finally concluding the cause must be with the installer. The solution was to download QLogic drivers and provide them for the Ubuntu installer. The trick is, however, that the driver is not native to Ubuntu. It is originally provided with Red Hat Enterprise Linux (RHEL) and Suse Linux Enterprise Server (SLES) and must be converted to work with Ubuntu Server. We managed to get the installer to recognize the attached volumes but the partitioning software interpreted each physical port as a separate disk. So we had four volumes instead of the two (the operating system reserved and the shared). At least it was able to determine the correct LUNs for the mappings. Another interesting issue was that it took well over 30 minutes for the installer to scan the available drives. The outcome of the installation was not that glorious, even the drives were now identified, because the installation process halted when any I/O operation should have happened and an error was thrown. In the end, we were unable to install Ubuntu with this setup.
There could be a few causes for the said issues and we have a plan on how to continue to troubleshoot. First thing to test is removing the configuration for all other volume mappings but a single fibre channel port to a single controller on the array. No redundant zones or anything too complex. If the operating system install fine, we can install DM-Multipath tool, which should help Ubuntu to manage logical volumes as they’re mapped, not as separate devices. The problem is that we need to have Ubuntu up and running in order to do the configuration, and in order to have it running we need to have the configuration in place. We hope to break the loop by simplifying the environment for the installer. Another step is to upgrade the blade server firmware as soon as we get the replacement DVD drive for the chassis. And as the last option we can defer using physical hard disks for the operating system installation and use shared SAN for virtual machines only. I will keep you up to date as soon as the testing proceeds and we know better. There are user stories with successful results, so I’m confident that Ubuntu can boot from SAN and can be configured to work with a redundant fibre channel network.
It’s a strange discovery that even though a major part of the server installations are based on Linux, the hardware certifications are mostly limited to commercial Linux distributions. Of course, the operating system can work with most distributions but there is no official support. And some software required by enterprise class hardware, like in our case the SAN drivers, needs to be converted, reverse engineered or otherwise hacked from a closed source or otherwise limited release. At least Canonical has shown some progress but there is more to be done if Linux aims to take over more of the server world.
If you have any first hand experiences, tips or can recommend a solution that solves these issues, please comment or send us an email.