By Brandon Patterson,2014-12-29 23:18
13 views 0


    Joseph Mack

    mack (at) wm7d (dot) net

Copyright ? 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006 Joseph Mack

v2006.07 Jul 2006, released under GPL.


Install, testing and running of a Linux Virtual Server with 2.2.x, 2.4.x, 2.6.x


search the LVS documentation

search the LVS documenation with htdig.

    search the two mailing list archives


Table of Contents

1. LVS: Introduction

    1.1. Thanks

    1.2. About the HOWTO

    1.3. Nomenclature/Abbreviations

    1.4. What is an LVS? Can I use an LVS?

    1.5. Minimal knowledge required

    1.6. Getting Technical Help

    1.7. After you've Got Technical Help

    1.8. Mailing list: subscribing, unsubscribing, searching 1.9. Mailing list: posting to

    1.10. Bug Fixes

    1.11. Other load balancing solutions, GPL, opensource and commercial

    1.12. Books on LVS

    1.13. LVS in the news

    1.14. Software/Information/HOWTOs useful/related to LVS 2. LVS: Install, Configure, Setup

    2.1. Installing from Source Code

    2.2. Ultra Monkey

    2.3. Keepalived

    2.4. Alternate hardware: Soekris (and embedded hardware) 2.5. LVS on a CD: Malcolm Turnbull's ISO files

3. LVS: Ipvsadm and Schedulers

    3.1. Using ipvsadm

    3.2. Compile a version of ipvsadm that matches your ipvs

    3.3. put realservers in /etc/hosts

    3.4. RR and LC schedulers

    3.5. Netmask for VIP

    3.6. LBLC, DH schedulers

    3.7. LVS with mark tracking: fwmark patches for multiple firewalls/gateways 3.8. Wensong's SH scheduler

    3.9. What is an ActiveConn/InActConn (Active/Inactive) connnection? 3.10. FAQ: ipvsadm shows entries in InActConn, but none in ActiveConn, connection hangs. What's wrong?

    3.11. FAQ: initial connection is delayed, but once connected everything is fine. What's wrong?

    3.12. unbalanced realservers: does rr and lc weighting equally distribute the load? - clients reusing ports

    3.13. Changing weights with ipvsadm

    3.14. Dynamically changing realserver weights

    3.15. connection threshold

    3.16. Flushing connection table

    3.17. Thundering herd problem, Slow start code for realserver(s) coming on line 3.18. Handling kernel version dependant files e.g. System.map and ipvsadm 3.19. Limiting number of clients connecting to LVS

    3.20. Who is connecting to my LVS?

    3.21. experimental scheduling code

    3.22. Ratz's primer on writing your own scheduler

    3.23. changing ip_vs behaviour with sysctl flags in /proc

    3.24. Counters in ipvsadm

    3.25. Exact Counters

    3.26. Scheduling TCP/UDP/SCTP/TCP splicing/

    3.27. patch: machine readable error codes from ipvsadm

    3.28. patch: stateless ipsvadm - add/edit patch

    3.29. patch: fwmark name-number translation table

    3.30. ip_vs_conn.pl

    3.31. Luca's php monitoring script

    3.32. ipvsadm set option

    4. LVS-NAT

    4.1. Introduction

    4.2. Example 1-NIC, 2 Network LVS-NAT (VIP and RIPs on different network) 4.3. All packets sent from the LVS-NAT realserver to the client must go through the LVS-NAT director

    4.4. Run the configure script

    4.5. Setting up demasquerading on the director; 2.4.x and 2.2.x 4.6. rewriting, re-mapping, translating ports with LVS-NAT

4.7. masquerade timeouts

    4.8. Julian's step-by-step check of a L4 LVS-NAT setup 4.9. How LVS-NAT works

    4.10. In LVS-NAT, how do packets get back to the client, or how does the director

    choose the VIP as the source_address for the outgoing packets? 4.11. One Network LVS-NAT

    4.12. re-mapping ports, rewriting is slow for 2.0, 2.2 kernels 4.13. Two instances of demon running on realserver 4.14. Performance of LVS-NAT

    4.15. Various debugging techniques for routes

    4.16. Connecting directly from the client to a service:port on an LVS-NAT realserver

    4.17. Masquerading clients on LVS-NAT realservers to the outside world 4.18. Realserver as client in LVS-NAT

    4.19. A NAT router has no connections

    4.20. Thoughts on extending NAT

    4.21. Postings from the mailing list

    4.22. Ken Brownfield's LVS-NAT routing patch

    4.23. LVS-NAT bug when running ftp helper

    4.24. LVS-NAT FTP Recipe

    4.25. LVS-NAT vhosts with apache

    5. LVS: The ARP Problem

    5.1. The problem

    5.2. Put the VIP on the realservers lo device

    5.3. The Cure(s)

    5.4. The Cure: 2.0 kernels

    5.5. The Cure: 2.2.x kernels

    5.6. The Cure: 2.4.x kernels

    5.7. The Cure: 2.6.x kernels

    5.8. arptables

    5.9. The arp problem is on the realserver's VIP not the RIP 5.10. Testing an interface for replies to arp requests 5.11. Normal machines, Solaris

    5.12. problems with switches

    5.13. The ARP problem, the first inklings

    5.14. A posting to the mailinglist by Peter Kese explaining the "arp problem"

    5.15. arp bouncing

    5.16. Lar's Method

    5.17. Static Routing to Director

    5.18. iproute2 arp on|off flag

    5.19. Is the arp behaviour of 2.2.x kernel a bug? 5.20. The device doesn't reply to arp requests, the kernel does. 5.21. Properties of devices for the VIP

    5.22. Topologies for LVS-DR and LVS-Tun LVS's

    5.23. Why do all devices broadcast the arp replies

5.24. A discussion about the arp problem

    5.25. ATM/ethernet and router problems

    5.26. Same IP on multiple NICs

    6. LVS-DR

    6.1. How LVS-DR works

    6.2. Handling the arp problem for LVS-DR

    6.3. LVS-DR scales well

    6.4. LVS-DR director as default gw for realservers, transparent proxy and Julian's

    martian and forward_shared patches

    6.5. Accepting packets on LVS-DR director by fwmarks 6.6. security concerns: default gw(s) and routing with LVS-DR/LVS-Tun 6.7. routing to realserver from director

    6.8. LVS-DR, LVS-Tun need rp_filter=0

    6.9. Director as client in LVS-DR

    6.10. Realserver as client in LVS-DR

    6.11. from the mailing list

    6.12. rewriting, re-mapping, translating ports with LVS-DR 7. LVS-Tun

    7.1. You need a tunl0 device

    7.2. How LVS-Tun works

    7.3. Configure LVS-Tun

    7.4. set rp_filter correctly

    7.5. FreeBSD realservers with LVS-Tun

    7.6. W2K realservers with LVS-Tun

    7.7. packets bigger than MTU

    7.8. tunl mtu solved

    7.9. rewriting, re-mapping, translating ports with LVS-Tun 8. LVS: LocalNode

    8.1. Two Box LVS

    8.2. Testing LocalNode

    8.3. Localnode on the backup director

    8.4. rewriting, re-mapping, translating ports with Localnode 9. LVS: You can't map (or rewrite) ports with LVS-DR, LVS-Tun or localnode (but you

    can with iptables)

    9.1. You can't rewrite ports with localnode (but you can with iptables) 9.2. rewriting, re-mapping, translating ports with iptables in LVS-DR 9.3. can't port map with LVS

    10. LVS-J: Ludo's reiJect Forwarder: using the director as a gateway to load balance

    connections to the internet

    10.1. Introduction

    10.2. reinJect setup with ipvsadm

    10.3. The target LVS: sending packets with dst_addr=0/0 to ip_vs 10.4. setting up LVS-J forwarding

    10.5. SNAT'ing the output

10.6. LVS-J discussion by Ludo

    11. LVS: Services: general, setup, debugging new services 11.1. Single port services are simple

    11.2. setting up a (new) service

    11.3. services must be setup for forwarding type

    11.4. Realservers present the same content: Synchronising (filesharing) content and

    config files, backing up realservers

    11.5. cfengine for synchronising files

    11.6. File Systems for (really big) Clusters: Lustre, Panasas 11.7. File Systems for Clusters: Samba waits for a commit and is slow, NFS fills

    buffers and is fast

    11.8. Discussion on distributed filesystems

    11.9. load balancing and scheduling based on the content of the packet: Cookies, URL,

    file requested, session headers

    11.10. Idle timeouts for TCP/UDP connections to services 11.11. name resolution on realservers: running name resolution friendly demons on


    11.12. Debugging new services

    11.13. "broken" services:servlets and j2ee

    11.14. http logs, error logs

    12. LVS: Services: single-port

    12.1. ftp, tcp 21

    12.2. ssh, sftp, scp, tcp 22

    12.3. telnet, tcp 23

    12.4. smtp, tcp 25; pop3, tcp 110; imap tcp/udp 143 (imap2), 220(imap3). Also sendmail,

    qmail, postfix, and mailfarms.

    12.5. Mail Farms

    12.6. dns, tcp/udp 53

    12.7. http name and IP-based (with LVS-DR or LVS-Tun), tcp 80 12.8. http with LVS-NAT

    12.9. httpd is stateless and normally closes connections 12.10. netscape/database/tcpip persistence (keepalives) 12.11. dynamically generated images on web pages

    12.12. http: sanity checks, shutting down, indexing programs, htpasswd, apache proxy

    and reverse proxy to look at URL, mod_backhand

    12.13. HTTP 1.0 and 1.1 requests

    12.14. Large HTTP /POST with LVS-Tun

    12.15. Microsoft http clients and servers violate the RFC for TCP/IP 12.16. http keepalive - effect on InActConn

    12.17. Apache setup for DoS

    12.18. squids, tcp 80, 3128

    12.19. authd/identd, tcp 113 and tcpwrappers (tcpd) 12.20. ntp, udp 123

    12.21. https, tcp 443

12.22. name based virtual hosts for https

    12.23. Obtaining certificates for https

    12.24. Self made certificates

    12.25. SSL Accelerators and Load Balancers

    12.26. r commands; rsh, rcpi (and their ssh replacements), tcp 514 12.27. lpd, tcp 515

    12.28. Databases

    12.29. Databases: mysql

    12.30. Using Zope with databases

    12.31. Databases: Microsoft SQL server, tcp 1433 12.32. Databases: Oracle

    12.33. nfs, udp 2049 (and possible replacements for nfs) 13. LVS: Services: multi-port

    13.1. Introduction

    13.2. ftp general, active tcp 20,21; passive 21,high_port 13.3. ftp (active) - the classic command line ftp 13.4. ftp helper modules: ip_vs_ftp/ip_masq_ftp 13.5. ftp (passive)

    13.6. ftp is difficult to secure

    13.7. ftps (ssl based ftp), tcp 21, 22?

    13.8. samba, udp 137, udp 138, tcp 139, tcp 445 13.9. xdmcp, X-window, udp 177 (xdmcp), tcp 6000 (and ssh X-forwarding)

    13.10. r commands; rsh, rcp, and their ssh replacements, tcp 513 (,514) and another


    13.11. Streaming Media: RealNetworks, Quicktime, Windows Media Server, tcp/udp 554

    (and other ports)

    13.12. Radius, udp 1645,1646

    14. LVS: Services that we haven't got to work with LVS yet 14.1. SIP (Session Initiation Protocol)

    14.2. Kerberos

    14.3. ldap

    14.4. RMI

    15. LVS: Routing and packet delivery to a director without a VIP (for fwmark and

    transparent proxy)

    15.1. Introduction

    15.2. Routing to and accepting packets by a VIP-less director 15.3. Routing to the MAC address of the director 15.4. Julian's iproute2 solutions

    15.5. Ludos LVS target in iptables

    15.6. Transparent proxy Q and A

    15.7. Other tricks

    16. LVS: Fwmarks (firewall marks)

    16.1. Introduction

    16.2. ipvsadm syntax for fwmark

    16.3. setting up routing and packet delivery to the director 16.4. single-port service: telnet with fwmarks

    16.5. Grouping services: single group, active ftp(20,21) 16.6. Grouping services: two groups, active ftp(20,21) and e-commerce(80,443) 16.7. passive ftp

    16.8. fwmark with LVS-NAT

    16.9. collisions between fwmark and VIP rules

    16.10. persistence granularity with fwmark

    16.11. fwmark allows LVS-DR director to be default gw for realservers 16.12. fwmark simplifies configuration for large numbers of addresses 16.13. Example: firewall farm

    16.14. Example: LVS'ing a CIDR block

    16.15. Example: forwarding based on client source IP

    16.16. Example: load balancing multiple class C networks 16.17. Example: proxy server

    16.18. Example: transparent web cache

    16.19. Example: Multiply-connected router

    16.20. httpd clients (browsers)

    16.21. Example: dynamically generated images in webpages 16.22. Example: Balancing many IPs/services as one block 16.23. Example: Source controlled LVS - services and realserver customised by Client


    16.24. Appendix 1: Specificiations for grouping of services with fwmarks 16.25. Appendix 2: Demonstration of grouping services with fwmarks 16.26. Appendix 3: Announcement of grouping services with fwmarks 16.27. fwmark examples from the mailing list

    17. LVS: Transparent proxy (TP or Horms' method)

    17.1. setting up routing and packet delivery to the director 17.2. General

    17.3. How you use TP

    17.4. The original 2.2 TP setup method

    17.5. Transparent proxy for 2.4.x (and presumably 2.6.x) 17.6. Experiments showing that 2.4TP is different to 2.2TP 17.7. What IP TP packets arriving on?

    17.8. Take home lesson for setting up TP on realservers 17.9. Handling identd requests from 2.4.x LVS-DR realservers using TP 17.10. Performance of Transparent Proxy

    17.11. The difference between REDIRECT and TPROXY

    18. LVS: Transparent Bridging

    19. LVS: Persistent Connection (Persistence, Affinity in cisco-speak) 19.1. LVS persistence

    19.2. Single Session

    19.3. Scheduling looks different under persistence

    19.4. Persistent and regular (non-persistent) services together on the same


    19.5. Tracing connections: where will the client connect next? 19.6. Bringing down persistent services.

    19.7. Forcing a break in a persistent connection: Horms code (Nov 2004) for quiescing persistent connections

    19.8. what if a realserver holding a persistent (sticky) connection crashes 19.9. Load Balancing time constant is longer with persistence 19.10. The tcp NONE flag

    19.11. Resetting the persistence timeout counter (persistence behaviour for short timeout values)

    19.12. Why you don't want persistence for your e-commerce site: why you should rewrite your application

    19.13. more about e-commerce sites: we used to think memory was the problem - it isn't 19.14. persistence with windows realservers

    19.15. IIS session management: how it works

    19.16. messing with the ipvsadm table while your LVS is running 19.17. Persistence for multiport services

    19.18. Proxy services, e.g. AOL

    19.19. key exchanges (SSL)

    19.20. About longer timeouts

    19.21. passive ftp and persistence

    19.22. The Persistence Template (about port 0)

    19.23. persistent clients behind a proxy or nat box

    19.24. Rogue clients hidden by persistence

    20. LVS: Running a firewall on the director: Interaction between LVS and netfilter (iptables).

    20.1. Introduction

    20.2. Path of an ip_vs controlled packet

    20.3. how to filter with netfilter

    20.4. ipvs_nfct, netfilter connection tracking for ipvs

    20.5. LVS-NAT netfilter conntrack example with ftp

    20.6. tcpdump is LVS compatible

    20.7. Writing Filter Rules

    20.8. The Antefacto Netfilter Connection Tracking patches

    20.9. The design of LVS as a netfilter module, pt1

    20.10. The design of LVS for Netfilter and Linux 2.4, pt2

    20.11. Example ip_tables filter scripts

    20.12. performance hit on director with iptables/netfilter 21. LVS: Cluster friendly versions of applications that need to maintain state 21.1. rewriting your application/service

    21.2. Session Data, maintaining state in a cluster, from Andreas Koening 21.3. Maintaining state with persistence

    21.4. How others maintain state

    22. LVS: Squid Realservers (poor man's L7 switch)

22.1. Terminology

    22.2. Preview

    22.3. Let's start assembling

    22.4. One squid

    22.5. Another squid

    22.6. Combining pieces with LVS

    22.7. Problems

    23. LVS: 3-Tier LVS

    23.1. Introduction

    23.2. Routes needed for 3-Tier LVS

    23.3. Setting up routes using iptables and iproute2 23.4. authd/identd and other 3-Tier clients 23.5. Masquerading clients on realservers to the outside world

    23.6. Masquerading clients on LVS-NAT realservers 23.7. Masquerading clients on LVS-DR realservers 23.8. Masquerading clients on LVS-Tun realservers 23.9. Masquerading clients through the VIP on the director 23.10. from the mailing list

    24. LVS: Authd/Identd

    24.1. What is authd/identd?

    24.2. symptoms of the identd problem

    24.3. comp.os.linux.security FAQ on identd 24.4. Russ Nelson on identd

    24.5. Why identd is a problem for LVS

    24.6. tcpdumps of connections delayed by identd 24.7. There are solutions to identd problem in some cases 24.8. Turn off tcpwrappers

    24.9. Identd and smtp/pop/qmail

    25. LVS: Performance and Kernel Tuning

    25.1. Performance Articles

    25.2. Estimating throughput: 100Mbps FE is really 8000packets/sec ethernet

    25.3. Jumbo frames

    25.4. Network Latency

    25.5. Mixture of 100Mbps and GigE ethernet 25.6. NICs and Switches, 100Mbps (FE) and 1Gbps (GigE) 25.7. NIC bonding

    25.8. NIC problems - eepro100

    25.9. NIC problems - tulip

    25.10. dual/quad ethernet cards, IRQ sharing problems 25.11. Flakey Switch

    25.12. performance testing tools

    25.13. Max number of realservers

    25.14. FAQ: What is the minimum hardware requirements for a director

    25.15. FAQ: How fast/big should my director be?

25.16. Does SMP help?

    25.17. Performance Hints from the Squid people 25.18. Conntrack, effect on throughput

    25.19. Don't use the Pre-emptible kernels

    25.20. 9.6Gbps served using LVS-DR with gridftp 26. LVS: Monitoring

    26.1. CPU usage/load level on the director?

    26.2. LVS throughput at the director with ipvsadm 26.3. Monitoring: LVS director throughput statistics from the /proc system

    (originally /proc/net/ip_vs_stats)

    26.4. MRTG family: Intro

    26.5. MRTG family: LVSGSP

    26.6. MRTG

    26.7. MRTG family: RRDtool

    26.8. MRTG family: cacti

    26.9. MRTG family: Ganglia (incl. INSTALL)

    26.10. MRTG family: rrd images

    26.11. Nagios

    26.12. MIB/SNMP

    26.13. home brew MIB/SNMP

    26.14. Disks

    26.15. Other output GUIs

    27. LVS: Details of LVS operation, Security, DoS 27.1. Top 20 security vunerabilities

    27.2. Top 75 security tools from the people at nmap 27.3. Do I need security, really?

    27.4. What to do after a break-in, prevention strategies 27.5. More about syncookies

    27.6. Can filter rules stop the intruder hopping to other machines? 27.7. Where filter rules act

    27.8. /proc filesystem flags for ipv4, e.g.rp_filter 27.9. tcp timeout values, don't change them

    27.10. /proc file system settings for LVS: security and private copies of tcp timeouts

    for LVS connections (you can change these)

    27.11. timeouts the same for all services

    27.12. Director Connection Hash Table

    27.13. Hash table connection timeouts

    27.14. Hash Table DoS

    27.15. Hash table size, director will crash when it runs out of memory.

    27.16. The LVS code does not swap

    27.17. Other factors determining the number of connections 27.18. Port range: limitations, expanding port range on directors 27.19. Director does not have any ports (connections) open for an LVS connection

    27.20. apps starved for ports

Report this document

For any questions or suggestions please email