Why You Should Use IPMP

IPMP provides increased reliability, availability, and network performance for systems with multiple physical interfaces. Occasionally, a physical interface or the networking hardware attached to that interface might fail or require maintenance. Traditionally, at that point, the system can no longer be contacted through any of the IP addresses that are associated with the failed interface. Additionally, any existing connections to the system using those IP addresses are disrupted.

By using IPMP, you can configure one or more physical interfaces into an IP multipathing group, or IPMP group. After configuring IPMP, the system automatically monitors the interfaces in the IPMP group for failure. If an interface in the group fails or is removed for maintenance, IPMP automatically migrates, or fails over, the failed interface's IP addresses. The recipient of these addresses is a functioning interface in the failed interface's IPMP group. The failover feature of IPMP preserves connectivity and prevents disruption of any existing connections. Additionally, IPMP improves overall network performance by automatically spreading out network traffic across the set of interfaces in the IPMP group. This process is called load spreading.

Solaris IPMP Components

Solaris IPMP involves the following software:

The in.mpathd daemon, which is explained fully in the in.mpathd(1M) man page.
The /etc/default/mpathd configuration file, which is also described in the in.mpathd(1M) man page.
ifconfig options for IPMP configuration, as described in the ifconfig(1M) man page.

Multipathing Daemon, `in.mpathd`

The in.mpathd daemon detects interface failures, and then implements various procedures for failover and failback. After in.mpathd detects a failure or a repair, the daemon sends an ioctl to perform the failover or failback. The ip kernel module, which implements the ioctl, does the network access failover transparently and automatically.

Note - Do not use Alternate Pathing while using IPMP on the same set of network interface cards. Likewise, you should not use IPMP while you are using Alternate Pathing. You can use Alternate Pathing and IPMP at the same time on different sets of interfaces. For more information about Alternate Pathing, refer to the Sun Enterprise Server Alternate Pathing 2.3.1 User Guide.

The in.mpathd daemon detects failures and repairs by sending out probes on all the interfaces that are part of an IPMP group. The in.mpathd daemon also detects failures and repairs by monitoring the RUNNING flag on each interface in the group. Refer to the in.mpathd(1M) man page for more information.

IPMP Terminology and Concepts

This section introduces terms and concepts that are used throughout Part V, IPMP.

IP Link

In IPMP terminology, an IP link is a communication facility or medium over which nodes can communicate at the data-link layer of the Internet protocol suite. Types of IP links might include simple Ethernets, bridged Ethernets, hubs, or Asynchronous Transfer Mode (ATM) networks. An IP link can have one or more IPv4 subnet numbers, and, if applicable, one or more IPv6 subnet prefixes. A subnet number or prefix cannot be assigned to more than one IP link. In ATM LANE, an IP link is a single emulated local area network (LAN). With the Address Resolution Protocol (ARP), the scope of the ARP protocol is a single IP link.

Note - Other IP-related documents, such as RFC 2460, Internet Protocol, Version 6 (IPv6) Specification, use the term link instead of IP link. Part VI uses the term IP link to avoid confusion with IEEE 802. In IEEE 802, link refers to a single wire from an Ethernet network interface card (NIC) to an Ethernet switch.

Physical Interface

The physical interface provides a system's attachment to an IP link. This attachment is often implemented as a device driver and a NIC. If a system has multiple interfaces attached to the same link, you can configure IPMP to perform failover if one of the interfaces fails. For more information on physical interfaces, refer to IPMP Interface Configurations.

Network Interface Card

A network interface card is a network adapter that can be built in to the system. Or, the NIC can be a separate card that serves as an interface from the system to an IP link. Some NICs can have multiple physical interfaces. For example, a qfe NIC can have four interfaces, qfe0 through qfe3, and so on.

IPMP Group

An IP multipathing group, or IPMP group, consists of one or more physical interfaces on the same system that are configured with the same IPMP group name. All interfaces in the IPMP group must be connected to the same IP link. The same (non-null) character string IPMP group name identifies all interfaces in the group. You can place interfaces from NICs of different speeds within the same IPMP group, as long as the NICs are of the same type. For example, you can configure the interfaces of 100-megabit Ethernet NICs and the interfaces of one gigabit Ethernet NICs in the same group. As another example, suppose you have two 100-megabit Ethernet NICs. You can configure one of the interfaces down to 10 megabits and still place the two interfaces into the same IPMP group.

You cannot place two interfaces of different media types into an IPMP group. For example, you cannot place an ATM interface in the same group as an Ethernet interface.

Failure Detection and Failover

Failure detection is the process of detecting when an interface or the path from an interface to an Internet layer device no longer works. IPMP provides systems with the ability to detect when an interface has failed. IPMP detects the following types of communication failures:

The transmit or receive path of the interface has failed.
The attachment of the interface to the IP link is down.
The port on the switch does not transmit or receive packets.
The physical interface in an IPMP group is not present at system boot.

After detecting a failure, IPMP begins failover. Failover is the automatic process of switching the network access from a failed interface to a functioning physical interface in the same group. Network access includes IPv4 unicast, multicast, and broadcast traffic, as well as IPv6 unicast and multicast traffic. Failover can only occur when you have configured more than one interface in the IPMP group. The failover process ensures uninterrupted access to the network.

Repair Detection and Failback

Repair detection is the process of detecting when a NIC or the path from a NIC to an Internet layer device starts operating correctly after a failure. After detecting that a NIC has been repaired, IPMP performs failback, the process of switching network access back to the repaired interface. Repair detection assumes that you have enabled failbacks. See Detecting Physical Interface Repairs for more information.

Target Systems

Probe-based failure detection uses target systems to determine the condition of an interface. Each target system must be attached to the same IP link as the members of the IPMP group. The in.mpathd daemon on the local system sends ICMP probe messages to each target system. The probe messages help to determine the health of each interface in the IPMP group.

For more information about target system use in probe-based failure detection, refer to Probe-Based Failure Detection.

Outbound Load Spreading

With IPMP configured, outbound network packets are spread across multiple NICs without affecting the ordering of packets. This process is known as load spreading. As a result of load spreading, higher throughput is achieved. Load spreading occurs only when the network traffic is flowing to multiple destinations that use multiple connections.

Dynamic Reconfiguration

Dynamic reconfiguration (DR) is the ability to reconfigure a system while the system is running, with little or no impact on existing operations. Not all Sun platforms support DR. Some Sun platforms might only support DR of certain types of hardware. On platforms that support DR of NIC's, IPMP can be used to transparently fail over network access, providing uninterrupted network access to the system.

For more information on how IPMP supports DR, refer to IPMP and Dynamic Reconfiguration.