A New Real-Time Control Architecture for Advanced Machines:
Distributed Automation and Digital Machine Vision using IEEE-1394
Edison Hudson, President
MetaControl Technologies Inc.
Industrial automation is commonplace in most competitive manufacturing activities today. Advanced automation control systems are central to increasing productivity not only in traditional manufacturing, but also in new fields such as drug discovery, clinical assay, and a diverse range of industries. The movement to higher throughput and extreme miniaturization, (micro and nano scale), makes human control of many processes beyond the dexterity, visual acuity, and attention bandwidth of even the most skilled technician. High performance machine systems executing multi-parameter control are commonplace in automotive, semiconductor, electronics, and increasingly in pharmaceutical production and laboratory environments.
Modern automatic machine systems must be able to control with very high precision a wide range of actuators and sensors simultaneously. Closed loop control of precision motion, force, temperature, flow rate, etc. in complex recipes is already a norm in high performance capital machinery. Increasing competitive pressure insists that these complex control regimes be implemented with cost effective electronics and robust software. Though high end machines for advanced industries are built in relatively small volumes compared to consumer goods, and yet they are under the same competitive pressures for each successive generation to perform at higher levels and lower costs. The industrial adoption of components and methods originally designed high volume consumer applications is a certainty for next generation industrial controls.
?IEEE-1394 or FireWire ( also “1394”), was developed as a consumer technology but shows tremendous potential to tackle both higher performance needs and lower systems costs demanded in advanced machinery. The combination of IEEE-1394 high speed serial bus with fast embedded processors enables a new cost effective, high performance architecture for advanced machine design and other demanding real-time automation tasks.
By enabling the physical distribution of computing power linked by synchronized, deterministic high bandwidth messaging, IEEE-1394 obviates the need for centralized backplane based machine controls. By utilizing the isochronous, peer-to-peer modes and serial data clock of IEEE-1394, the architecture described in this paper shows how distributed embedded processor nodes and intelligent digital cameras can become a preferred alternative to the centralized backplane approach in dominant use today.
Traditional Automation Control Architecture
Most current solutions in high performance machine automation and instrumentation can be characterized as centralized and backplane oriented. It has been a normal assumption in advanced motion, machine vision, and analog process automation that only backplane based controllers could provide the high communication speeds needed to synchronize motion, images, and data acquisition events. The standard implementation of most industrial and laboratory automation controllers is the rack mounted backplane, primarily using bus solutions such as VME, VXI, PXI, or proprietary buses. In recent years, PCI and Compact PCI systems have gained a significant stake in these markets, driven by the advancing capabilities of Windows based PC’s.
In the conventional architecture, all sensors, motors, digital inputs and outputs, and analog signals are cabled from the point of use to converge at the backplane resident cards Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 1 / 21
designed to handle each specialized function. All signals are brought to the physical location of the system controller typically using multi-wire cable bundles. In a typical machine used in semiconductor back-end processing, thousands of individual wires converge from many locations in the machine to the central rack mounted controller. Diagram 1 is a schematic of a typical automation machine with 6 axes of motion control, machine vision, and process control. Examples of systems that use this style of control include semiconductor manufacturing equipment, electronics manufacturing equipment, packaging machines, mechanical assembly, molecular genetic laboratory systems, among others.
The traditional architecture, as depicted by Diagram 1 often contains several specialized backplanes to implement different control functions. Typically, motion control is handled by a specialized controller or board, machine vision by another board or controller, while digital and analog I/O functions add additional subsystems. Bus-to-bus communication between these various subsystems is often through traditional RS-232/422/485 serial communication channels, or in some cases with bus converters. Cabling of these systems is complex and represents a major constraint on complexity. The centralized approach also limits reliability and configurability as hundreds of conductors, many often traversing moving axes, are required to route signals to the central control chassis.
Overall, the traditional approach is cumbersome, physically large, and invariably results in a controls solution per machine that today might typically range from $10,000 to $30,000 depending on performance specifics. The era of the low cost PC begs for solutions that are on of a factor of 10 lower in cost. The software development for traditional architectures such as depicted in Diagram 1 is often expensive and time consuming, due to the fact the subsystems often come from several vendors and are developed with different software approaches and lack standard interfaces..
Distributed Control Systems
As opposed to the centralized, backplane oriented approach illustrated above, a distributed control system ( “DCS”) architecture uses some form of serial or parallel cable to link already
digitized information from the point of use. In a DCS, analog signals are quickly digitized, and functions are localized that do not need to be centrally supervised. The conventional wisdom held by most automation designers has been that the distributed approach can only be used for low speed performance or when coordination of devices and events is not highly time critical. Whenever exact time synchronization or computational speed is critical, backplane bus based systems were assumed to always be superior in performance. Additionally in recent years, distributed systems have often been more expensive to implement than low end backplane systems.
However, many advantages of DCS are well accepted by industry including:
； Greater signal integrity, (S/N ), can be achieved by reducing the distance that
analog signals must travel before digitization occurs, important in applications
where signal to noise maximization is demanded;
； Cabling can be simplified and functional subsystems can be modularized
allowing configuration at a higher level of integration. System configuration is
greatly ease by subsystems that can plug into a network, particularly in complex
machines such as electronics assembly machines, semiconductor processing,
and multi-axes robots;
； Large physical scale systems can be controlled more easily with distributed
systems, especially large physical plants such as paper processing, chemical
refineries, textile processing, and commercial printing systems.
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 2 / 21
Remote monitoring of signals or control functions over a corporate or public network is generally more straightforward with DCS systems which are naturally data packet driven.
Diagram 1 – Traditional control architecture diagram
Shielded CablesVisionVisionmonitormonitor ( 8 to 24 wires )
Typical 6-axis system using conventional control architecture
There have been many distributed control schemes developed and implemented for industrial applications over the past three decades. The oldest of these distributed schemes are based on Field-Bus and its derivatives, Device Net, CanBus, ProfiBus and others. Most of these Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 3 / 21
schemes are limited in their ability to deal with data rates more than a few megabits / second, far below the capabilities of backplane buses like VME or PCI. In the motion control arena there is a fast serial bus called Sercos, that gained some market share, but in recent years has languished due to high cost and a data rate limitation of 4 megabits / second. For a more comprehensive review of existing distributed control architectures, see the NIST report found at http://www.isd.mel.nist.gov/projects/openarch/motstds.doc .
IEEE-1394 Distributed Control System
Unlike the aforementioned distributed control standards, IEEE-1394, (“1394”), has most of
the advantages and few of their disadvantages of earlier generation DCS buses. Particularly with regards speed of operation, functional signaling modes, and cost, 1394 has the ability to radically change the approach of automation control design. With 1394, the idea that all signals must converge on the bus backplane location is inverted, since 1394 can bring an adequately fast bus to the signals and the point of control.
Though 1394 was originally conceived as a consumer oriented bus, many of its technical features are particularly well suitable to advanced control systems. The following attributes of 1394 are specifically important to distributed control design:
； High speed – with 1394-A, 400 megabit/sec is already faster than nearly every
industrial DCS serial bus by 3 orders of magnitude. As compared to the widely used
industrial distributed buses based on Fieldbus derived technologies, 1394 is nearly
1000 times faster. 1394 B begins to compete in absolute speed with parallel bus
backplane solutions (see Chart 1. below). With a roadmap to 400 megabytes/ second
(3,200 mbs ), 1394-B on glass fiber offers speed that only a few high performance
back planes contemplate, ( future buses like PCI-X, VME-64, and Star Fabric are
faster, but multiple times as expensive). In many cases, backplanes systems like
VME are more than adequate in speed for most controls and instrument applications.
1394 provides a bandwidth option that meets the majority of machine control and
instrumentation needs where messages are typical short in data length, but
numerous and frequent
Chart 1 - Maximum Data Rate of Various Bus Types
including IEEE-1394 A/B
50megabytes / second0
PCMCIA1394AVME/ VXI1394BPCI / PXI
Machine vision is one control technology that demands high bus speed and as a
result has not until recently been amenable to distributed control architectures. The
high speed of 1394 is needed to allow the digital acquisition of video from multiple
cameras, and it is fast enough to replace backplane based framegrabbers.
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 4 / 21
； Isochronous mode– Most serial communication schemes do not have a mode of
operation that guarantees the time based delivery of data packets. This time
determinism of 1394 is very beneficial in application to closed loop servo control, data
acquisition from analog sources, and machine vision using digital video. In addition to
the need for guaranteed delivery time, guaranteed delivery order is important for
these applications, as each message represents the state of a machine or instrument
function at a given point in time and may be part of a closed loop in which the order
of the data must be sequential. Networks that do not support ordered and timely data
sequences, such as traditional Ethernet, pose impossible conundrums to control
； Asynchronous mode – A common requirement of most control systems is to have the
ability to respond to instantaneous events. Within the time windows permitted by the
clocking scheme of 1394, asynchronous event messages can be generated within
every bus clock cycle by any node, thereby allowing a high priority message to
propagate with a known latency. The 1394-A asynchronous window of 125
microseconds is adequate for most control applications, with the 1394-B interval
dropping to 62.5 microseconds, and ultimately to less than 16 microseconds,
accommodating most events on even the most demanding control applications in the
future. Another important use of asynchronous mode is in control systems with
intelligent nodes, in which asynchronous data packets provide a means to change
control parameters on the fly in parallel loop operation. This is very significant for
dynamic loop control devices where the initial parameters need to be modified with
changing system conditions.
； Peer-to-Peer mode - The facility to allow 1394 nodes to communicate directly without
sending each message through a host decreases the latency that is associated with
host centric networks, such as USB and Ethernet. In many cases, this can be a
significant advantage of 1394 when a change in state at one node needs to be
passed to another node as quickly as possible and with great determinism. Since
host centric networks may suffer from varying delays in message propagation
depending on the processing load at the host, the peer-to-peer mode enables control
designers to implement schemes in which a message can be sent just to the affected
nodes. In low-end applications, where a PC may not be present, peer-to-peer offers
the possibility of embedded solutions that do not require a PC.
； Broadcast mode – In distributed control, many nodes may need to start or stop a
process in synchronization with an event or start signal generated by a coordinating
processor. Also safety violation conditions that affect the whole system that all nodes
need to be aware of can be implemented using 1394’s broadcast capability that
sends a message to all nodes at once. This is a powerful feature for the
synchronization and control of multi-axis motion control over a distributed
network.With open host controller interface (OHCI) chip sets using direct memory
access transfers that OHCI supports, data transfer speed is not dependent on
operating system's interrupt latency.
Other distributed network schemes for industrial automation include some of the above
control features of 1394. 1394 is unique in combining all of these desirable modes in a single
architecture, with a promising roadmap of increasing performance that mimics that of
1394-1995 and 1394-A had some limitations and deficiencies that were of concern to control
design. In particular, the following issues present design problems for machine control
； Galvanic isolation – the earlier versions of 1394 were subject to influence of
system level ground fluctuations that could produce unintended “unplugs” and
potential data corruption due to the limited electrical isolation. Though these
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 5 / 21
grounding issues could be resolved with system engineering, these earlier
versions were less robust than some industrial norms.
； Distance between nodes – For most intra-machine applications the 4.3 meter
limitation of cabling between 1394 A nodes is not an issue, but limit the scope of
application to larger systems or factory level process control where nodes may
be tens of meters apart.
； RFI interference – In factory and machine environments, the level of radiated
electrical noise can be very high when compared to consumer environments
where such noise is strictly regulated. In high power applications, such as large
motors systems, radiated noise can be intense and potentially poses
susceptibility challenges to the earlier 1394 standards.
Each of the above concerns can be mitigated by careful electrical design, and strict attention to system grounding rules. Distributed control systems based on the earlier 1394 A standard have proven to be robust and reliable by adherence to good electrical design, even though usually at a cost of added system components. The distance issues has been overcome in these earlier systems by employing long haul adapters that convert from 1394 twisted pair to plastic or optical signals over greater distances, converting back to 1394 at each end.
Fortunately, the 1394-B standard has eliminated all of the above concerns in its improvements, making the new standard ever more compatible with control system design needs. The simplified electrical isolation scheme adopted by 1394B is also very low cost by comparison to the magnetics approach taken by other buses. By incorporating these enhancements into the consumer standard, 1394-B based systems will be both robust and highly competitive with any specialized industrial DCS network for advanced control.
Comparison to the Predominant Network- Ethernet
Many expect the pervasiveness of Ethernet to be a decisive factor in its selection as the media on which most future distributed control networks will be implemented. This belief is largely based on the wide availability of Ethernet ports on PC and the pre-wired infrastructure support at the corporate enterprise level down to the factory floor. Diagram 2 illustrates the current factory network status and the normal use of Ethernet in the factory.
While some factors favor the incumbency of Ethernet, the installed Ethernet base is less of a factor inside automation machines, or “intra-machine”. Because of the value placed on
performance in machine design, technical factors are decidedly against Ethernet as the logical choice for high performance real-time control.
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 6 / 21
Diagram 2 – Factory Communication Hierarchy
File oriented File oriented
Time critical Time critical
bus domainbus domain
The main control related problem with traditional Ethernet is the Collision Sense, Multiple Access, Collision Detection ( “CSMA/CD”) scheme that is used in the vast majority of installed Ethernets. This issue and the potential solutions are summarized by the following abstract from an industrial journal:
“Traditional Ethernet is not realtime friendly. The CSMA/CD scheme makes access inherently non-deterministic. An Ethernet controller connected to a thin Ethernet (coax - 10BASE-2) or a hub (10BASE or 100BASE) is not able to send a packet as long as the medium is busy sending another packet. The Ethernet controller is free to send its packet as soon as the Ethernet is idle.
The probability for a collision depends on the collision domain, i.e. the range of the Ethernet, and the network load. A traditional CSMA/CD Ethernet with 20% utilization has less than 0.1% collision, while as much as 5% of the packets will experience collisions if the network utilization is above 40%. A CSMA/CD network with 40% utilization is in trouble, and the net data rate will in fact decrease due to collisions if the load is further increased. However, bear in mind, those collisions are not errors. A collision is a normal part of Ethernet networks.
Ethernet switches today may have support for priority containing two or more output queues per port, where the high priority queue(s) are reserved for real time critical data offering Quality of Service (QoS). How the switch alternates between the priority queues varies from vendor to vendor. Relevant alternating schemes for a switch with two priority queues could be:
； Round-robin weighting, i.e. N packets are sent from the high priority queue before
one packet is sent from the low priority queue.
； Strict priority, i.e. all packets will be transmitted from the high priority queue. Packets
from the low priority queue will only be sent in case the high priority queue is empty.
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 7 / 21
Note that a high priority packet will be delayed due to a low priority packet if the transmission of this packet is started before the high priority packet enters the output port. The high priority packet will then be delayed by the time it takes to flush the rest of the packet. Worst case will be that the transmission of an Ethernet packet with maximum packet length (1518 bytes) is just started. The extra delay will then be 122ms in case of 10 Mbps, and 1.22 ms in case of 100 Mbps. Typical worst-case switch latency for a high priority packet in such a system will be a few hundred ms in case of 100 Mbps on each drop link. .”
The latency problem that is inherent in non-switch based Ethernet has drawn many solutions proposed by industrial organizations. Several proprietary designs have arisen to overcome these deficiencies. Most of these schemes are focused on reducing traffic and implementing known performance switch nodes. A recipe for deterministic Ethernet often cited is described in the same article, as follows:
100 Mbps with full duplex connectivity is used on all drop links.
； The switch is a store-and-forward switch, with a minimum switch latency of 10us.
； The switch uses strict priority scheduling.
； The real time packet has a length of 200 bytes including preamble, MAC, IP, UDP,
payload, Frame Check Sequence (FCS) and minimum Inter Packet Gap (IPG).
； The realtime packets are treated as high priority packets, all other packets have less
； Up to five other stations may generate similar real time packets of 200 bytes that may
be in the same priority queue before the packet enter queue, and cause extra switch
； All realtime packets are generated in a cyclic manner.
The worst case switch latency of a realtime packet will then be:
16 us, store-and-forwards.
10 us, minimum switch latency.
122 us, worst case latency due to flushing of a packet with maximum packet length.
80 us, five real time packets already in the same priority queue.
228 us, total.”
Industrial Ethernet, Issue 5, “VoIP Drives Real-Time EtherNet”, Øyvind Holmeide, Tom Skeie
( http://ethernet.industrial-networking.com/articles/i05voipdrives.asp )
Latency in control systems is inherently bad in that it reduces phase margin and the bandwidth of control. When latency is high or variable, many processes cannot not be controlled in a stable way, especially not when traditional PID, (Proportional Integral Derivative) type control loops are used. Non-PID type control is very rare, and control schemes that allow variable latency are complex to implement. Therefore, minimized latency is of high value in controls, particularly in high performance, closed loop machine control . As the above example shows, by paying close attention to configuration rules and to using Ethernet switches with the correct specifications, a decent level of determinism is achievable. Table 1 below shows comparable latency factors for Ethernet and 1394.
The interesting note is that even after significantly upgrading existing “office grade” to make it
Table 1 - Network Latencies
Closed Loop Fundamental -Domain Latency, 10Latency,
-66 sec sec10
1394A, 400 mbs125250
1394B, 800 mbs62.5125
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 8 / 21 Switched Ethernet228456
industrial Ethernet and investing significant care to configure it to provide a degree of determinism, its inherent performance in closed loop real time control is still inferior to the 1394 standard consumer technology. Most of the “deterministic” Ethernet schemes strongly recommend keeping network utilization below 30%, but this degrades 100 mhz “fast” Ethernet to a mere 30 mbs, significantly slower than traditional solutions for real-time control. At these speeds, the mixing data intensive activities like machine vision or closed loop servo synchronization on a single network is not feasible.
Furthermore, though the above example addresses issues of the physical transport layers in Ethernet, there are still other performance issues to consider. It is still not inherently a DMA type bus with independence of host load, as is 1394 under OHCI operation. Most of these “deterministic” schemes are also likely to exhibit a wide range of actual deterministic behavior in real world installations depending on the operating system configuration and loading. The efficiency of implementation of the TCP-IP stack is also a determinant of actual performance. It is unlikely that embedded web servers will all perform with the same latency characteristics.
Another driver behind Ethernet’s perception as the popular choice for control networks is
likely a miscalculation of, or out of date information about, the cost of Ethernet relative to 1394. With the broadening acceptance of 1394 for consumer appliances and PC’s, 1394 could likely have a numeric advantage in absolute volume, since Ethernet volume is primarily driven by corporate levels of demand. With the advent of Windows XP and Mac OS, nearly every home PC will have a 1394 port, ( required by all PC’s that are officially” Windows XP Ready” ).
The new 1394 B PHY devices with galvanic isolation are already expected to cost less than just the passive isolation magnetics required by Ethernet ports. The 1394 Trade Association estimates that 1394 B node hardware costs will drop to around $5, a cost that is substantially below that of most Ethernet node designs. Regardless of the actual volumes and designs, the relative difference in component cost to implement 1394 relative to Ethernet will be negligible in the near future. Since 1394-B also uses the same mediums, CAT-5 and fiber, total systems costs should be at parity.
Table 2 below compares other technical differences between 1394 and Ethernet, and clearly indicate that 1394 is superior in features and performance for intra-machine controls. Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 9 / 21
Table 2 – 1394 and Ethernet Properties Comparison
1394 A/B Ethernet
S200/400/800/ S10/100/ Data Rates 1600/3200 1000
-12 -9 Bit error rate (1/ errors) 1010
Tree, Daisy, Star/ Topologies supported Star / hub hub
Max length between hubs 100 m 75 m
Isochronous mode Yes No
Peer to Peer mode Yes No
Many, 6 std., TCPIP, UDP Protocols including TCPIP
TP, Plastic, Glass, CAT5 Media supported Glass, CAT5
Universal Plug-and-Play YES No
It should also be noted that many of the industrial Ethernet standards groups are proposing changes to standard Ethernet that are so gross that the industrial versions will have little compatibility with the incumbent infrastructure. For example:
； Several industrial proposals advocate abandoning TCP based protocols at the host
level as well in favor of a more controls oriented protocols such as UDP.
； The German based Profibus group recently advocated an implementation of
industrial Ethernet, which even abandons the standard’s synonymous RJ-45
connector in favor of a DIN type connector.
These significant deviations destroy the attraction and economics of Ethernet as a well defined communication commodity. Imposing non-standard protocols and hardware modifications to “band-aid” Ethernet into being a better vehicle for industrial controls, is likely to result in high cost, proprietary, or narrowly supported industrial niche products that are Ethernet in name only.
Clearly there are cases in which Ethernet as a control network is a logical choice. Specifically, the node cost and performance issues do not affect applications with the following characteristics:
； Traditional Ethernet applied to monitoring and supervision of sensor network for
informational and data logging uses, where update are likely to be sub-hertz collected
occasionally or on demand.
； Switched Ethernet configured for more deterministic behavior in control applications
with requirements less than 1 kilohertz update rate, and bandwidth less than 25
Distributed Machine Control with 1394 E. Hudson 1394 Devcon June 25, 2002 10 / 21