Stand Sure Systems Blog

Welcome to the Standsure Weblog

Greetings!

This blog is for engineers and other people who want to find and share methods to improve system reliability, geek out on cool code snippets, and generally proclaim their technology expertise and/or concerns.

Enjoy, contribute if you like, and play safe!

 - Ron Gould & Dave Pierce

Switch Trunking and "Fail-Over"

My July 19, 2009 entry, Linux Ethernet Bonding for Switch Fail-Over, talks about two switches connected via a valid Inter-Switch Link (ISL).  What does that mean?

In order to answer that seemingly simple question, we need to answer another question: "How do switches fail-over?"

Well, actually, they don't.  Or at least, not exactly.

Switches serve a more physical function than things like databases or firewalls that we often associate with failover.  This is because servers and other devices are physically plugged into switches, often as their only connection to the network.  Because of this, switches can't fail-over in the way that, for example, a Cisco ASA firewall can fail-over.  If a firewall fails, its backup unit can seamlessly take over traffic.  But if a switch fails, there is no seamless way to unplug connected devices and plug them into a different switch.  (Hmmm, perhaps a business opportunity for a robotics startup?  Nah...) 

But we can create redundancy at the switching layer of the network.  This redundancy is generally accomplished through two mechanisms: host distribution and trunking.

Host distribution means ensuring that hosts are connected to multiple switches.  In other words, if you have six web servers, all six of them shouldn't be connected to the same switch.  Trunking means allowing switches to pass ethernet traffic to each other, regardless of which VLAN that traffic belongs to.

The best way to achieve host distrubution varies.  The least expensive way is to simply divide your hosts in half, and connect one half to one switch, and one to another.  This is simple and cheap, but it has a couple of drawbacks.  First, if a switch does fail you lose half of your capacity.  Your site might still be up, but unless you are running at 50% capacity, you may be serving packets slower than you would like.  And second, if a switch fails you'll need to failover any monolithic services, like databases, which could have survived the switch failure if they had an alternate connection.

Another way is to simply use ethernet bonding to have every host connected to two different switches.  This provides the greatest flexibility, and prevents failover of monolithic services like databases.  But it comes at a significant cost, you have to purchase twice as many switchports for a given number of hosts.

Many sites us a hybrid.  Horizontally scalable servers like databases are connected to one switch each, and a carefully divided between switches so that you only lose a percentage of them in any given switch failure.  Monolithic hosts like databases or storage heads use ethernet bonding so that the host remains available in a switch failure scenario.

So that's host distrubution, what about trunking?  Why can't two switches just be connected with a cable?

Well, if you are not using VLANs on a switch, they can be.  But there are two problems with that scenario.  One is that the cable becomes a Single Point of Failure (SPOF) for your network.  And two is that the inter-switch traffic will be limited to the size of one port.

But even if you were willing to accept those problems, if you are using VLANs you have another problem.  A simple cable will only transmit packets for the VLAN it is connected to.  Traffic on other VLANs will be orphaned.  Trunking solves this problem via "tagging" of packets -- the switch tags every packet with a VLAN identifier, which is removed on the receiving switch and sent to the correct VLAN on that swtich.  It sounds like a lot of work for the switch, but modern switches handle it effortlessly.

So a trunk is usually two components.  First, multiple ports on a switch are joined together in an "etherchannel", which always many ports to act as a single port.  Thus, a gigabit switch can have a 5Gbps connection to another switch by creating a 5-port etherchannel.  Then, the etherchannel port is trunked, using IEEE 802.1q trunking.  Most major switch vendors support multiple trunking protocols; I like to use 802.1q because it works well across mutliple vendors.

Linux Ethernet Bonding for Switch Fail-Over

We recommend that all PROD sites deploy switches in redundant fail-over pairs.  However, even if the switches fail over, the servers connected to any single switch will not.

Unless we deploy Linux Ethernet bonding as summarized below.

We use "active-backup" mode.  This requires that two of the the server's Ethernet interfaces are connected to two separate switches on the same VLAN, and that those switches are connected via a valid Inter Switch Link (ISL). We typically use IEEE 802.1q trunking for the ISL because it is a standard which is portable among different switch vendors.

Once the bonding module is loaded and properly configured the host has a new logical interface, bond0.  bond0 has two slave interfaces, eth0 and eth1.  One of those two interfaces is active at any given point in time, and the other only takes over if the kernel detects a failure in the active interface.  When it does detect such a failure, it begins using the other interface, and it sends a gratuitous ARP out the other interface, so that upstream switching devices can update their MAC address tables.  This covers many -- but not all -- possible network failure scenarios.

The Linux kernel documentation has a fairly good technical document about bonding, which you can find here: http://www.mjmwired.net/...bonding.txt

In an upcoming blog post, we will describe how to configure the servers and switches to support Linux Ethernet Bonding.

 - Dave Pierce, VP Engineering

10 Steps to Reliability -- Part 2

To get the most out of a change management (CM) process, preparing good change and rollback plans are required.  Forunately this can be simplified through application of a good CM notice form.

Here is the form we use for all of our changes:

----
SUBJECT:  Change Notice for <title>
(Start SUBJECT with "EMERGENCY" if appropriate.)

TITLE or NAME OF CHANGE:

IS THIS AN EMERGENCY CHANGE:  (No/Yes)
IF YES, NAME OF APPROVING EXECUTIVE: 

DESCRIPTION OF CHANGE:

EXPECTED START TIME:
EXPECTED DURATION:

RISK LEVEL: High/Medium/Low

CONTACT INFO FOR PERSON EXECUTING THE CHANGE:
  Name:
 Email:
  Cell:

SYSTEMS IMPACTED BY CHANGE:

HOW HAS THIS CHANGE BEEN TESTED:   (NOT "why this change doesn't require testing")
IF UNTESTED, NAME OF EXECUTIVE OR ENGINEER APPROVING AN UNTESTED CHANGE: 

WHAT ARE THE REQUIRED CHANGES TO THE MONITORING SYSTEM:

IS A USER-FACING MAINTENANCE WINDOW (SITE OFFLINE) REQUIRED?  Yes/No

WILL INDIVIDUAL SYSTEMS BE OFFLINE OR REBOOTED/POWER-CYCLED (even if they are load-balanced)?  Yes/No

WHAT ARE THE STEPS TO IMPLEMENT THE CHANGE:

WHAT WILL BE THE TESTING/VALIDATION THAT THE CHANGE IS SUCCESSFUL:

WHAT ARE THE STEPS TO ROLLBACK THE CHANGE:

IF CHANGE IS ROLLED BACK, NAME OF ENGINEER WHO WILL REVIEW THE ROLLBACK:
----
In an upcoming article I will describe each element of the form above, but today I will focus on preparing the  CHANGE and ROLLBACK plans.

A.  PREPARING THE CHANGE PLAN:

This is often done in an iterative fashion.  That is, first the step-by-step plan is drafted, then as the change is tested (always a good plan itself!) the step-by-step is refined.  We like to do this on a wiki so we can easily make and undo changes.

The plan itself should cover each step that will be taken.  This is important for two reasons.  First, it forces the author to think through each step in advance, thus preventing many problems that might have happened if the chagen was done in an ad-hoc manner.  Second, and equally important, such a process allows other people review the change to see problem and alert the author in advance.  (A real-world example we had was a client doing an unplanned service restart on a NAS device that triggered an FSCK of a few TB of storage.  THAT resulted in a multi-hour downtime event.  Was the service restart or FSCK on the CM?  Nope!)

A snippet of an actual CM plan is below:

----
1. Copy 1.15.0 clone master, but do NOT overwrite LocalSettings.php or images/.
cp -rv --no-dereference
/var/www/html/mediawiki/mwiki-1.15.0-clone_master/* .
chmod a+w images/
chmod a+w config/

2. Create the maintenance directory and run the update script
cp -r ../mediawiki-1.15.0/maintenance .
cd maintenance/
php update.php --aconf ../AdminSettings.php

3. Wait for a long time
4. Validate the new wiki, especially the DPL scripts
5. Remove the maintenance directory
cd ..
rm -rf maintenance/

----
Any linux-aware reader can see what will be done during that cutover.  It is NOT necessary to include the shell commands; I just do that to prevent typos when I execute the change.

B.  PREPARING THE ROLLBACK PLAN
This step is routinely overlooked by engineers who believe that most of the time it won't be needed.  However, by thinking through how to recover from a failed change, you often discover the need to take snapshots and backups during the change itself. AND IT IS MUCH EASIER TO TAKE A RECOVERY SNAPSHOT *BEFORE* YOU CHANGE THE SYSTEM.

Here is an example:

----
1. Copy 1.11.0 clone master, but do NOT overwrite LocalSettings.php
or images/.
cp -rv --no-dereference
/var/www/html/mediawiki/mwiki-1.11.0-clone_master/* .
chmod a+w images/
chmod a+w config/

2. Modify the LocalSettings.php:
vi LocalSettings.php
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "mwiki_1_11_prod";

3. Log onto wiki to verify nothing has changed
4. Change a page normally, then verify change is visible in
mwiki_1_11_prod database
----

During testing, one of the change commands was discovered to be wrong, so the rollback plan had to be executed.  Because it was thought-through in advance, the rollback was a trivial event with no head-scratching or time wasted.

One of the side benefits we find when we have good change and rollback plans, is that if a change does not go quite right in PROD, we can recover without exceeding the planned downtime window.  Marketing and our customers ALWYS appreciate that.

Next up:  Pulling it all together in a CM notice.

10 Steps to Reliability through Change Management

After years of helping companies recover from avoidable system crashes, it has become very clear that more technology might not be the first step in improving reliability.  Utilizing a simple Change Management process can provide immediate reliability improvements at little or no cost.

724 Engineering has developed a 10-step Change Management (CM) process that we utilize when making any change to a production system.  This same process has been successfully applied for hardware, software, network and many other changes.

Here are the steps:

1) 48 hours in advance of change, send plain text mail to changelog mailing list using CM Template
NOTES:
 (a) If change is to be made on an EMERGENCY basis, executive approval is required.
 (b) If change has not been tested or cannot be tested, executive approval is required.

2) Objections must be filed within 24 hours

If no objections are filed:
3) If downtime is needed or possible, Marketing posts PM notification on www or support site
4) “Change Window Starting” mail sent to changelog
5) Change is made and tested

If successful:
6) “Change Made Successful” mail sent to changelog
7) Documentation updated
<end of process>

If unsuccessful:
8) Rollback per plan
9) “Change NOT MADE” mail sent to changelog
10) Post-mortem meeting scheduled with rollback engineer

Following this simple process ensures that everyone who might be impacted knows a change is coming and can be prepared.  In addition, preparing the CM notice and rollback plan ensures that the person making the change has thought it through fully.

We have utilized the 10-step process above for literally thousands of changes.  The result is invariably a measurable improvement in system reliability.

Next in this series:  Preparing a CM notice and roll-back plan.