Scaleable Event Infrastructure for P2P Grids
A Scaleable Event Infrastructure for Peer to Peer Grids
Geoffrey Fox, Shrideep Pallickara, Xi Rao and Qinglin Pei
Community Grid Labs, Department of Computer Science
Indiana University
The peer-to-peer (abbreviated as P2P) style interaction [10] model facilitates sophisticated resource sharing environments between “consenting” peers over the “edges” of the internet; the “disruptive” [11] impact of which has resulted in a slew of powerful applications built around this model. Resources shared could be anything – from CPU cycles, exemplified by SETI@home (extraterrestrial life) [14] and Folding@home
(protein folding) [15], to files (Napster and Gnutella [17]). Resources in the form of direct human presence include collaborative systems (Groove [18]) and Instant Messengers (Jabber [16]). Peer “interactions” involves advertising resources, search and subsequent discovery of resources, request for access to these resources, responses to these requests and exchange of messages between peers. An overview of P2P systems and their deployments in distributed computing and collaboration can be found in [9]. Systems tuned towards large-scale P2P systems include Pastry [19] from Microsoft, which provides an efficient location and routing substrate for wide-area P2P applications. Pastry provides a self-stabilizing infrastructure that adapts to the arrival, departure and failure of nodes. The JXTA [12] (from juxtaposition) project at Sun Microsystems is another research effort
that seeks to provide such large-scale P2P infrastructures. Discussion pertaining to the adoption of event services as a key building block supporting P2P systems can be found in [8,9]. In this paper we propose a peer-to-peer (P2P) grid comprising resources such as relatively static clients, high-end resources and a dynamic collection of multiple P2P subsystems. We investigate the architecture, comprising a distributed brokering system that will support such a hybrid environment. Services can be hosted on such a P2P grid with peer groups managed locally and arranged into a global system supported by core servers. Access to services can then be meditated either by the “broker middleware” or alternatively by direct peer-to-peer (P2P) interactions between
machines “on the edge”. The relative performance of each approach (which could reflect computer/network cycles as well as the existence of firewalls) would be used in deciding on the implementation to use. P2P approaches best support local dynamic interactions; the distributed broker approach scales best globally but cannot easily manage the rich structure of transient services, which would characterize complex tasks. We use our research system Narada as the distributed brokering core to support such a hybrid environment.
There are several attractive features in the P2P model, which motivate the development of such hybrid systems. Deployment of P2P systems is entirely user driven obviating the need for any dedicated management of these systems. Peers expose the resources that they are willing to share and can also specify the security strategy to do so. Driven entirely on demand a resource may be replicated several times; a process that is decentralized and one over which the original peer that advertised the resource has sometimes little control over. Peers can form groups with the fluid group memberships. In addition P2P systems tend to be very dynamic with peers maintaining an intermittent digital presence. P2P systems incorporate schemes for searching and subsequent discovery of resources. In P2P systems, not every request (search) goes through, and even if it does, there could be zero or more valid responses (discovery). Peers anticipate neither the template that the responses conform to nor the order in which these responses would be received. Furthermore, responses are not identical with each responding peer processing any given request based on the resources at its disposal and it’s interpretation of the request. Communication between a requesting peer and responding peers is facilitated by peers en route to these destinations. These intermediate peers are thus made aware of capabilities that exist at other peers. This discovery of services offered by other peers constitutes dynamic real time knowledge propagation. Furthermore, since peer interactions, in most P2P systems, are XML based, peers could be written in any language and can be compiled for any platform.
There are also some issues that need to be addressed while incorporating support for P2P interactions. P2P interactions are self-attenuating with interactions dying out after a certain number of hops. These attenuations in tandem with traces of the peers, which the interactions have passed through, eliminate the continuous echoing problem that result from loops in peer connectivity. However, attenuation of interactions sometimes prevents clients from discovering certain services that are being offered. This results in P2P interactions being very localized. These attenuations thus mean that the P2P world is inevitably fragmented into many small subnets that are not connected. Peers in P2P systems interact directly with each other and sometimes use other peers as intermediaries in interactions. Specialized peers are sometimes deployed to enhance routing characteristics.
Page 1 of 10
Scaleable Event Infrastructure for P2P Grids
Nevertheless, sophisticated routing schemes are seldom in place and interactions are primarily through simple forwarding of requests with the propagation range being determined by the attenuation indicated in the message.
1.0 Narada
Narada is an event brokering system designed to run on a large network of cooperating broker nodes. Communication within Narada is asynchronous and the system can be used to support different interactions by encapsulating them in specialized events. Events are central in Narada and encapsulate information at various levels as depicted in the figure 1. Clients can create and publish events, specify interests in certain types of events and receive events that conform to specified templates. Client interests are managed and used by the system to compute destinations associated with published events. Clients, once they specify their interests, can disconnect and the system guarantees the delivery of matched events during subsequent reconnects. Clients reconnecting after prolonged disconnects, connect to the local broker instead of the remote broker that it was
last attached to. This eliminates bandwidth Event Originsdegradations caused by heavy concentration of clients
from disparate geographic locations accessing a certain SourceExplicitknown remote broker over and over again. The DestinationsDestinationsdelivery guarantees associated with individual events
and clients are met even in the presence of failures. Used to computeEvent DescriptorsDestinations
Content Descriptors1.1 Broker Organization & small world behavior
Uncontrolled broker and connection additions, result in Used to handleContent Payloadcontenta broker network susceptible to network-partitions and Event Distribution Traces /devoid of any logical structure making the creation of TimeTo Live (TTL)Used for eliminatingefficient broker network maps (BNM) an arduous if continuous echoing/not impossible task. The lack of this knowledge attenuation of event.hampers development of efficient routing strategies,
which exploit the broker topology. Such systems then Figure 1: Event in the Narada System resort to “flooding” the entire broker network, forcing
clients to discard events they are not interested in. To
circumvent this, Narada incorporates a broker organization protocol, which manages the addition of new brokers and also oversees the initiation of connections between these brokers. The node organization protocol incorporates IP discriminators, geographical location, cluster size and concurrent connection thresholds at individual brokers in its decision making process to prevent these situations.
In Narada we impose a hierarchical structure on the broker network, where a broker is part of a cluster that is part of a super-cluster, which in turn is part of a super-super-cluster and so on. Clusters comprise strongly connected brokers with multiple links to brokers in other clusters, ensuring alternate communication routes during failures. This organization scheme results in “small world networks” [1,2] where the average
communication “pathlengths” between brokers increase logarithmically with geometric increases in network size, as opposed to exponential increases in uncontrolled settings. This distributed cluster architecture allows Narada to support large heterogeneous client configurations that scale to arbitrary size. Creation of BNMs and the detection of network partitions are easily achieved in this topology. We augment the BNM hosted at individual brokers to reflect the cost associated with traversal over connections, for e.g. intra-cluster communications are faster than inter-cluster communications. The BNM can now be used not only to compute valid paths but also for computing shortest paths. Changes to the network fabric are propagated only to those brokers that have their broker network view altered. Not all changes alter the BNM at a broker and those that do result in updates to the routing caches, containing shortest paths, maintained at individual brokers.
1.2 Dissemination of events
Every event has an implicit or explicit destination list, comprising clients, associated with it. The brokering system as a whole is responsible for computing broker destinations (targets) and ensuring efficient delivery to these targeted brokers en route to the intended client(s). Events as they pass through the broker network are be updated to snapshot its dissemination within the network. The event dissemination traces eliminate continuous echoing and in tandem with the BNM – used for computing shortest paths – at each broker, is used to deploy a
near optimal routing solution. The routing is near optimal since for every event the associated targeted set of brokers are usually the only ones involved in disseminations. Furthermore, every broker, either targeted or en
Page 2 of 10
Scaleable Event Infrastructure for P2P Grids
route to one, computes the shortest path to reach target destinations while employing only those links and brokers that have not failed or been failure-suspected.
1.3 Failures and Recovery
In Narada, stable storages existing in parts of the system are responsible for introducing state into the events. The arrival of events at clients advances the state associated with the corresponding clients. Brokers do not keep track of this state and are responsible for ensuring the most efficient routing. Since the brokers are stateless, they can fail and remain failed forever. The guaranteed delivery scheme within Narada does not require every broker to have access to a stable store or DBMS. The replication scheme is flexible and easily extensible. Stable storages can be added/removed and the replication scheme can be updated. Stable stores can fail but they do need to recover within a finite amount of time. During these failures the clients that are affected are those that were being serviced by the failed storage.
1.4 Support for dynamic topologies
Support for local broker accesses, client roams and stateless brokers provide an environment extremely conducive to dynamic topologies. Brokers and connections could be instantiated dynamically to ensure the optimal bandwidth utilizations. These brokers and connections are added to the network fabric in accordance with rules that are dictated by the agents responsible for broker organization. Brokers and connections between brokers can be dynamically instantiated based on the concentration of clients at a geographic location and also based on the content that these clients are interested in. Similarly average pathlengths for communication could be reduced by instantiating connections to optimize clustering coefficients within the broker network. Brokers can be continuously added or fail and the broker network can undulate with these additions and failures of brokers. Clients could then be induced to roam to such dynamically created brokers for optimizing bandwidth utilization.
1.5 JMS Compliance
Narada is JMS [21] compliant and provides support not only for JMS clients, but also for replacing single/limited server JMS systems transparently [24] with a distributed Narada broker network. Since JMS clients are vendor [22,23] agnostic, this JMS integration has provided Narada with access to a plethora of applications built around JMS, while the integrated Narada-JMS solution provides these applications with scaling, availability and dynamic real time load balancing. Among the applications ported to this solution is the Anabas distance education conferencing system [25] and the Online Knowledge Center (OKC) portal [26] being developed at the IU Grid labs. Comprehensive results comparing Narada and SonicMQ can be found in [24].
1.6 Results from the prototype
Figure 3 illustrates some results [4,6] from our initial research where we studied the message delivery time as a function of load. The results are from a system comprising 22 broker processes and 102 clients in the topology outlined in figure 2. Each broker node process is hosted on 1 physical Sun SPARC Ultra-5 machine (128 MB RAM, 333 MHz), with no SPARC Ultra-5 machine hosting two or more broker node processes. The publisher and the measuring subscriber reside on the same SPARC Ultra-5 machine. In addition to this there are 100 subscribing client processes, with 5 client processes attached to every other broker node (broker nodes 22 and
10 do not have any other clients besides the publisher and measuring subscriber respectively) within the system. The 100 client node processes all reside on a SPARC Ultra-60 (512 MB RAM, 360 MHz) machine. The run-time environment for all the broker node and client processes is Solaris JVM (JDK 1.2.1, native threads, JIT). The three matching values correspond to the percentages of messages that are delivered to any given subscriber. The 100% case corresponds to systems that would flood the broker network. The system performance improves significantly with increasing selectivity from subscribers. We found that the distributed network scaled well with adequate latency (2 milliseconds per broker hop) unless the system became saturated at very high publish rates. We expect the latency to decrease by a factor of about three in an “optimized production system”.
Page 3 of 10
Scaleable Event Infrastructure for P2P Grids
PublisherTransit Delay under different matching rates:22 Brokers 102 Clients22Match Rate=100% Mean Match Rate=50% Transit Delay Match Rate=10% 11 (MilliSeconds)10k4501240035030054250i1314200216l150h10015350087j500450940035030017025016100200200Event Size m30015040018500100 (Bytes)60070050Publish Rate 80009001000 (Events/sec)1920nMeasuring21 Subscriber Figure 3: Transit delays for different matching rates Figure 2: Test Topology
2.0 Narada and P2P interactions
Issues in P2P systems pertaining to the discovery of services and intelligent routing can be addressed very well in the Narada brokering system. The broker network would be used primarily as a delivery engine, and a pretty efficient one at that, while locating peers and propagating interactions to relevant peers. The most important aspect in P2P systems is the satisfaction of peer requests and discovery of peers and associated resources that could handle these requests. The broker network forwards these requests only to those peers that it believes can handle the requests. Peer interactions in most P2P systems are achieved through XML-based data interchange. XML’s data description and encapsulation properties allow for ease of accessing specific elements of data. Individual brokers routing interactions could access relevant elements, cache this information and use it subsequently to achieve the best possible routing characteristics. The brokering system, since it is aware of advertisements, can also act as a hub for search and discovery operations. These advertisements when organized into “queryspaces” allow the integrated system to respond to search operations more efficiently.
Resources in Narada are generally within the purview of the broker network. P2P systems replicate resources in an ad hoc fashion, the availability of which is dependent on the peer’s active digital presence. Some resources, however, are best managed by the brokering system rather than being left to the discretion of peers who may or may not be present at any given time. An understanding of the network topology and an ability to pin point the existence of peers interested in that resource are paramount to efficient replications of a resource. The distributed broker network, possessing this knowledge, best handles this management of resources while ensuring that these replicated resources are “closer” and “available” at locations with a high interest in that resource. Furthermore, the broker network is also better suited, than a collection of peers, to eliminate race conditions and deadlocks that could exist due to a resource being accessed simultaneously by multiple peers. Broker networks can be responsive to changes in peer concentrations, volumes of peer requests, and resource availability. Brokers and associated interconnections can be dynamically instantiated or purged to compensate for affected routing characteristics due to changes in peer interactions.
As mentioned earlier, P2P systems fragment into multiple disconnected sub-systems. The brokering system could also be used to connect islands of peers together. Peers that are not directly connected through the peer network could be indirectly connected through the broker network. Peer interactions and resources in the P2P model are traditionally unreliable, with interactions being lost or discarded due to peer failures or absences, overloading of peers and queuing thresholds being reached. Guaranteed delivery properties existing in traditional brokering systems can augment peer behavior to provide a notion of reliable peers, interactions and resources.
Such an integrated brokering solution would also allow for hybrid interaction schemes to exist alongside each other. Applications could be built around hybrid-clients that would exhibit part peer behavior and part
Page 4 of 10
Scaleable Event Infrastructure for P2P Grids
traditional client behavior (e.g. JMS). P2P communications could be then used for traffic where loss of information can be sustained. Similarly, hybrid-clients needing to communicate with each other in a “reliable”
fashion could utilize the brokering system’s capabilities to achieve that. Sometimes, hybrid-clients can satisfy
each other’s requests, in which case they would, obviating need for funneling interactions through the broker network. The broker merely serves as an efficient conduit for supporting interaction between different applications (clients, peers or hybrid).
3.0 JXTA
JXTA is a set of open, generalized protocols to support peer-to-peer interactions and core P2P capabilities such as indexing, file sharing, searching, peer grouping and security. The JXTA peers, and rendezvous peers (specialized routers), rely on a simple forwarding of interactions for disseminations and rely on time-to-live (TTL) indicators and peer traces to attenuate interaction propagations. However JXTA interactions are unreliable, tend to be very localized and are based on simple forwarding. Figure 4 depicts the protocols that
comprise the XML encoded JXTA protocol suite. JXTA is independent of transport protocols and can be implemented on top of TCP/IP, HTTP, BEEP (Block Extensible Exchange Protocol IETF RFC 3080), TLS, Bluetooth, HomePNA, and many other protocols. JXTA provides features such as dynamic discovery and a rich search mechanism while allowing peers to communicate across NAT, DHCP, and firewall boundaries. In JXTA a peer is any node that supports JXTA protocols and could be any digital device. Peers that seek to collaborate could come together to form a peer group. Peers within a peer group can identify each other, agree on group memberships and exchange information with each other. Peers publish the existence of a resource through an advertisement, which is simply an XML document describing the resource. Peers locate other peers, peer groups and properties pertaining to them. Once a peer joins a JXTA group, JXTA’s discovery capabilities support
queries for services, resources and other peers. The queries could be centralized or a decentralized one involving the entire peer group.
Publish & receiveBind 2 endpointsadvertisementsand create pipe forfor a peer groupcommunication.
Check anotherpeer's statusPeer InfoPeer DiscoveryPipe BindingProtocolProtocolProtocolResponsible forintiating propagationsPeer Resolver Protocolto peersSend/Receivequeries to/fromRendezvous Protocolpeers
Peer Endpoint ProtocolUses networkprotocols tohandle routing
Figure 4: The JXTA protocol suite
JXTA is also programming language independent; the C language binding for JXTA core was released earlier this year and is presently being remodeled now for the new version of JXTA core. Implementation of the core JXTA protocols in Perl 5, Object C, Ruby and Python are currently underway. It is expected that existing P2P systems would either support JXTA or have bridges initiated to it from JXTA. Support for JXTA would thus enable us to leverage other existing P2P systems along with applications built around these systems. With regards to bridges to other existing P2P systems LimeWire Inc. ((http://www.limewire.com) is investigating
shared development between the Gnutella and Project JXTA developer communities. There also proposals pertaining to the management of interactions between Jabber and JXTA peers. To extend JXTA support to lightweight devices projects such as pocketjxta have been started to implement the JXTA platform and build
applications for Personal Digital Assistants (PDA). Narada’s support for JXTA in addition to the support for
JMS would result in interactions that are robust and dynamic while being supported by a scalable and highly available system.
4.0 NARADA JXTA Integration
In our strategy for providing support for P2P interactions within Narada, we need to ensure –
Page 5 of 10
Scaleable Event Infrastructure for P2P Grids
; Minimal or zero changes to the Narada system core and the associated protocol suites.
; We also make no changes to the JXTA core and the associated protocols. We make additions to the
rendezvous layer for integration purposes. Peers do not communicate directly with the Narada and
continue to interact with other peers and rendezvous peers just as they presently do.
; Furthermore, this integration should entail neither any changes to the peers nor a straitjacketing of the
interactions that these peers could have had prior to the integration.
The integration is based on the proxy model, which essentially acts as the go-between the Narada system and JXTA. The Narada-JXTA proxy, operating inside the JXTA rendezvous layer, serves in a dual role as both a rendezvous peer and as a Narada client providing a bridge between Narada and JXTA. Narada could be viewed as a service by JXTA. The discovery of this service is automatic and instantaneous due to the Narada-JXTA proxy’s integration inside the rendezvous layer. Any peer can utilize Narada as a service so long as it is connected to a Narada-JXTA proxy. Nevertheless, peers do not know that the Narada broker network is routing some of their interactions. . Furthermore, these Narada-JXTA proxies, since they are configured as clients within the Narada system, inherit all the guarantees that are provided to clients within the Narada system.
4.1 The interaction model
Different JXTA interactions are queued at the queues associated with the relevant layers comprising the JXTA
protocol suite [13]. Each layer performs some operations High end "long lived"/including the addition of additional information. The persistent resources
rendezvous layer processes information arriving at its
input queues from the peer-resolving layer and the pipe-
binding layer. Since the payload structure associated with NARADA-different interactions is different we can easily identify the JXTA proxyinteraction types associated with the payloads. Interactions NARADApertaining to discovery/search or communications within a broker cloudpeer group would be serviced both by JXTA rendezvous
peers and also by Narada-JXTA proxies. Interactions that
peers have with the Narada-JXTA proxies are what are
routed through the Narada system. JXTA peers can
continue to interact with each other and of course some of
these peers can be connected to pure JXTA rendezvous Peerspeers. Peers have multiple routes to reach each other and JXTADynamic/fluidRendezvoussome of these could include the Narada brokering system peer groupsPEERand some of them need not. Such peers can interact
directly with each other during the request/response
interactions. Figure 5 outlines the Narada JXTA Figure 5: The Narada-JXTA interaction model interaction model.
4.2 Interaction Disseminations
Peers can create a peer group; request to be part of a peer group; perform search/request/discovery all with respect to a specific targeted peer group. Peers always issue requests/responses to a specific peer group and sometimes to a specific peer. Peers and peer groups are identified by UUID [33] (IETF specification guarantees
uniqueness until 3040 A.D.) based identifiers. Every peer generates its own peer id while the peer that created the peer group generates the associated peer group id. Each rendezvous peer keeps track of multiple peer groups through peer group advertisements that it receives, any given peer group advertisement could of course be received at multiple rendezvous peers. These rendezvous peers are then responsible for forwarding interactions; if it had received an advertisement for the peer group contained in these interactions.
Narada-JXTA proxies are initialized both as rendezvous peers and also as Narada clients. During its initialization as a Narada client every proxy is assigned a unique connection ID by the Narada system, after which the proxy subscribes to a topic identifying itself as a Narada-JXTA proxy. This enables Narada to be aware of all the Narada-JXTA proxies that are present in the system. The Narada-JXTA proxy in its role as a rendezvous peer to peers receives –
; Peer group advertisements
; Requests from peers to be part of a certain peer group and responses to these requests
Page 6 of 10
Scaleable Event Infrastructure for P2P Grids
; Messages sent to a certain peer group or a targeted peer
To ensure the efficient dissemination of interactions, it is important to ensure that JXTA interactions that are routed by Narada are delivered only to those Narada-JXTA proxies that should receive them. This entails that the Narada-JXTA proxy perform a sequence of operations, based on the interactions that it receives, to ensure selective delivery. The set of operations that the Narada-JXTA proxy performs comprise gleaning relevant information from JXTA’s XML encapsulated interactions, constructing an event based on the information gleaned and finally in its role as a Narada client subscribing (if it chooses to do so) to a topic to facilitate selective delivery. By subscribing to relevant topics, and creating events targeted to specific topics each proxy ensures that the broker network is not flooded with interactions routed by them. The events constructed by the Narada-JXTA proxies include the entire interaction as the event’s payload. Upon receipt at a proxy, this payload
is de-serialized and the interaction is propagated as outlined in the proxy’s dual role as a rendezvous peer. Events constructed from interactions need to have a unique identifier associated with them. Advertisements, since, they encapsulate information pertaining to uniquely identifiable resource can use the UUID associated with the advertised resource as the interaction identifier of the constructed event. The interaction type along with the interaction identifier allow us to uniquely identify each event. In the case of JXTA messages the unique interaction identifier is constructed based on the peer Id issuing the message and the timestamp in milliseconds (based on the system-clock at the peer node) associated with the message. We now proceed to outline the sequence of operations associated with different JXTA interactions.
4.2.1 Peer Group Advertisements Narada HeadersWhen peer group advertisements propagated by a peer are received at a
Narada-JXTA proxy, it creates the event depicted in figure 6. The peer Interaction IDgroup advertisement is the payload contained in this event. The proxy JXTA Interaction Typeproceeds to initiate a subscription to the peer group with the subscription (Subscription)being registered to the connection that the proxy has into the Narada system. Peer group idThis enables Narada to identify this Narada-JXTA proxy as a destination
when certain interactions are targeted to that specific peer group. Narada Peer Group Advertisement
delivers this peer group advertisement to all the Narada-JXTA proxies Narada Event Distributionwithin the integrated Narada-JXTA system. Proxies that receive this Tracesadvertisement do not initiate any actions and the proxy deals with this
advertisement just as a JXTA rendezvous peer would. Figure 6: Event for Peer Group Advertisements 4.2.2 Requests to be part of a peer group, and responses to these
requests
When a peer issues a request to be part of a certain peer group, the event constructed by the proxy is depicted in figure 7.(a). The advertisement is contained in the payload, and the targeted topic contained in this event is the peer group id. Narada thus propagates the event to the proxies that had subscribed to this topic. To ensure that responses to this advertisement are targeted to the proxy forwarding this request, the event also encapsulates its Narada connection information within this event. Narada-JXTA proxies receiving this event maintain the JXTA request and the connection associated with the request; to be used during propagation of responses. These Narada-JXTA proxies then behave as a normal rendezvous peer, processing the request as it normally would. This entails forwarding/routing of events en route to peers that are part of the peer group.
When responses, initiated by authorized peers, are received at the Narada-JXTA proxy, the proxy checks to see if there was a request associated with it and proceeds to forward the response encapsulated in a Narada event depicted in figure 7.(b). Responses include the peer-id that is assigned to the requesting peer, and this information is included in the event. The proxy also retrieves the target connection that this response should be sent to and includes it in the event. Upon receipt of this response at the initiating Narada-JXTA proxy, the proxy initiates operations in its role as a rendezvous to ensure propagation of the response to the requesting peer. The proxy then proceeds to subscribe to both the peergroup-id and the peer-id. This event is depicted in figure 7.(c),
and as can be seen there is no payload contained in this event. The peergroup-id subscription ensures that interactions for the peer group that the serviced peer will be a part of are always received at the proxy; this includes advertisements to be part of that peer group. Furthermore, a lot of JXTA interactions are sometimes targeted to specific peers so we also subscribe to the peer-id contained in the received response. When events are sent to a specific peer in a peer group, the Narada system routes the event to the proxy (or proxies since a
Page 7 of 10
Scaleable Event Infrastructure for P2P Grids
given peer can be attached to multiple Narada-JXTA Proxies/Rendezvous peers) that can route the event to the targeted peer.
Narada Headers
Narada HeadersInteraction ID
Narada HeadersJXTA Interaction TypeInteraction ID
Interaction IDNarada Connection InfoJXTA Interaction TypeJXTA Interaction TypeNarada Connection InfoPeer group id(Subscription)
Peer group idPeer group idPeer id
Peer AdvertisementPeer idPeer Advertisement
Narada Event DistributionNarada Event DistributionNarada Event DistributionTracesTracesTraces
Peer AdvertisementPeer AdvertisementAfter Peer AdvertisementRequestResponseResponse(a)(b)(c) Figure 7: Dealing with requests and responses, to peers being part of a peer group
4.2.3 JXTA Messages
When JXTA messages arrive at a Narada-JXTA proxy, the event constructed for routing within the Narada system is depicted in figure in figure 8.
These are the elements that the Narada For duplicateNarada Headersdetectionsystem will operate on. The rest of the Request/withinJXTA message needs to be serialized and Interaction IDResponseNARADAwould be the payload contained within the JXTA Interaction Typeconstructed event. Narada then routes this Allows theevent to only those proxies that had Peer group idNARADAPresent ifsubscribed to either the peergroup-id or the system tointeraction isPeer idpeer-id contained in this event. Upon receipt route event totrageted to aat the Narada-JXTA proxies the payload specific JXTAJXTA Event Payloadspecificproxiescontained in the event is unmarshalled and PeerNarada Eventthe JXTA message is recreated. With the Distribution TracesNarada-JXTA proxy now behaving as any
other rendezvous peer, propagation from this
point on proceeds as dictated by the JXTA Figure 8: The Event for JXTA messages reference implementation. Narada for its
part deals with the efficient routing of the
Narada events based on the topic and subscriptions (propagated during receipt of peer advertisements at the proxies).
4.3 The Duplicate detection problem:
Unlike Narada clients that interact with only one broker at any given time and never with any other client directly; peers can be connected to multiple peers and rendezvous peers. This situation leads to having the same interaction entering the Narada system through multiple points and also due to loops in peer connectivity. Under conditions of increasing loads and high peer concentrations the cumulative effects of these JXTA interactions entering the distributed broker network could entail prohibitive CPU/bandwidth costs. The most crucial thing is to ensure that the Narada broker network is not inundated with such duplicate interactions.
JXTA interactions, when they are propagated into the Narada system, are encapsulated inside events outlined in earlier sections. Destinations computed for duplicate events are identical; when these events are being routed within the Narada network the route they take is more or less the same, i.e. they propagate through the same
Page 8 of 10
Scaleable Event Infrastructure for P2P Grids
cluster controllers. Paths traversed by later events (duplicates) en route to final destinations are more or less the same as the event that traversed prior to it. These paths only vary during sudden network changing conditions such as failures of brokers/links and spikes in network usage over sustained durations in which case the paths computed based on network costs would vary. Even in such cases subsequent duplicates continue to traverse along identical computed paths. Having brokers keep track of the events (event id’s specifically) they received and discarding duplicates allows us to solve the duplicate detection problem and prevent the network from expending network cycles for routing duplicates. This scheme allows for faster disseminations, with a survival of the fittest scheme, where duplicate events are discarded if they were not the first to arrive at a broker.
5.0 Narada JXTA Systems and Applications
The Narada JXTA integration service scales naturally since Narada provides dynamic “long distance” support while JXTA provides for dynamic localized supports. In the combined global scale infrastructure Narada works best for “long lived” and persistent resources that would be efficiently replicated within Narada. This integrated
model provides efficient search/discovery not only for static activity but also for dynamic activity while allowing JXTA interactions at the edges. The resultant system also scales with multiple JXTA Peer Groups linked by Narada, which can dynamically instantiate new brokers to balance the load. As opposed to the simple forwarding of interactions, the intelligent routing in Narada in tandem with the duplicate detection scheme in our solution ensures faster disseminations and improved communication latencies for peers. Furthermore, targeted peer interactions traversing along shortest paths within the broker network obviates the need for a peer to maintain dedicated connections to a lot of peers. Proxies, due to their initialization as clients within Narada, inherit all the guarantees accorded to clients within the Narada such as guaranteed delivery in the presence of failures and fast disseminations. Discovery of rendezvous peers in JXTA is a slow process. A rendezvous peer generally downloads a list of other rendezvous peers from a server, not all of which would be up and running at that time. We allow for dynamic discovery of Narada-JXTA proxies, which need not be directly aware of each other, but do end up receiving interactions sent to a peer group if they had both received peer group advertisements earlier. The scheme also allows us to connect islands of peers and rendezvous peers together, allowing for a greater and richer set of interactions between these peers.
A typical application of the hybrid Narada/JXTA technology is distance education. This often consists of multiple linked classrooms where the participants in each classroom are individually linked to collaborative environment. Here a peer-to-peer model (such as JXTA) can be used in a classroom to give fast dynamic response to shared input control while the JMS style global Narada capability is used to multicast between classrooms. More generally this combination of global structured and local dynamic messaging scales to support general applications. We can package the JXTA/JMS/Narada hybrid as an event or messaging web service whose different modes could correspond to different ports in WSDL [27]. These ports trade off scaling, performance and reliability in a fashion that can be chosen by the user. Alternatively web services communicate via channels and we could use the technology of this paper to flexibly link different services together with dynamic choice of service provider. Other applications we are pursuing include workflow where administrative agents control the traffic between different web services. We have explained in more detail how one can base a P2P Grid architecture on a generalized publish/subscribe mechanism in [8].
7.0 Conclusion
In this paper, we presented our strategy for providing support for JXTA interactions. We also described the benefits that can be accrued by clients from these systems. It is our belief that this integration, to go along with the JMS integration, has added considerable value to Narada and that we are well positioned to Web Service “enable” Narada. We are currently gathering performance numbers for the JXTA based integration, which would contrast the latencies in pure JXTA environments with those in Narada’s integrated environment. The
final version of this paper would incorporate these comprehensive results.
References
1. D.J. Watts and S.H. Strogatz. Collective Dynamics of Small-World Networks. Nature. 393:440. 1998.
2. R. Albert, H. Jeong and A. Barabasi. Diameter of the World Wide Web. Nature 401:130. 1999.
3. The Narada Event Brokering System http://grids.ucs.indiana.edu/ptliupages/projects/narada/
Page 9 of 10
Scaleable Event Infrastructure for P2P Grids
4. Geoffrey Fox and Shrideep Pallickara, An Event Service to Support Grid Computational Environments, to be
published in Concurrency and Computation: Practice and Experience, Special Issue on Grid Computing
Environments.
5. Geoffrey C. Fox and Shrideep Pallickara, An Approach to High Performance Distributed Web Brokering. ACM
Ubiquity Volume2 Issue 38. November 2001.
6. Pallickara, S., "A Grid Event Service." PhD Syracuse University, 2001.
7. Geoffrey C. Fox and Shrideep Pallickara .The Narada Event Brokering System: Overview and Extensions To
appear in the Procedings of the 2002 International Conference on Parallel and Distributed Processing Techniques
and Applications (PDPTA'02).
8. Geoffrey Fox, Ozgur Balsoy, Shrideep Pallickara, Ahmet Uyar, Dennis Gannon, Aleksander Slominski.
Community Grids. Proceedings of the International Conference on Computational Science (ICCS 2002).
Amsterdam, Netherlands April 2002.
9. Geoffrey Fox, "Peer-to-Peer Networks," Computing in Science & Engineering, vol. 3, no. 3, May2001.
10. openp2p P2P Web Site from O’Reilly http://www.openp2p.com.
11. “Peer-To-Peer: Harnessing the Benefits of a Disruptive Technology”, edited by Andy Oram, O’Reilly Press March
2001.
12. Sun Microsystems. The JXTA Project and Peer-to-Peer Technology http://www.jxta.org
13. The JXTA Protocol Specifications. http://spec.jxta.org/v1.0/docbook/JXTAProtocols.html
14. SETI@home Project http://setiathome.ssl.berkeley.edu.
15. Folding@home Project http://www.stanford.edu/group/pandegroup/Cosm
16. Jabber http://www.jabber.org
17. Gnutella. http://gnutella.wego.com
18. Groove Network http://www.groove.net
19. Antony Rowstron and Peter Druschel. Pastry: Scalable, decentralized object location and routing for large-scale
peer-to-peer systems. Proceedings of Middleware 2001.
20. Ken Arnold, Bryan O'Sullivan, Robert Scheifler, Jim Waldo and Ann Wollrath. The Jini Specification. Addison-
Wesley. June 1999.
21. Mark Happner, Rich Burridge and Rahul Sharma. Sun Microsystems. Java Message Service Specification. 2000.
http://java.sun.com/products/jms
22. SonicMQ JMS Server http://www.sonicsoftware.com/
23. The OpenJMS Project http://openjms.exolab.org/
24. Geoffrey C. Fox and Shrideep Pallickara JMS Compliance in the Narada Event Brokering System. To appear in
the proceedings of the 2002 International Conference on Internet Computing (IC-02).
25. The Anabas Conferencing System. http://www.anabas.com
26. The Online Knowledge Center (OKC) Web Portal http://judi.ucs.indiana.edu/okcportal/index.jsp
http://www.w3.org/TR/wsdl. 27. Web Services Description Language (WSDL) 1.1
28. Presentation on Web Services by Francesco Curbera of IBM at DoE Components Workshop July 23-25, 2001.
Livermore, California. http://www.llnl.gov/CASC/workshops/components2001/viewgraphs/ FranciscoCurbera.ppt
29. Frank Laymann, Web Services Flow Language WSFL, http://www.ibm.com/software/solutions/webservices/pdf/
WSFL.pdf
30. Harumi Kuno, Mike Lemon, Alan Karp and Dorothea Beringer, (WSCL – Web Services Conversational
Language), “Conversations + Interfaces = Business logic”, http://www.hpl.hp.com/techreports/2001/HPL-20 01-
127.html
31. Semantic Web from W3C to describe self organizing Intelligence from enhanced web resources. http://www.
w3c.org/2001/sw/
32. Berners-Lee, T., Hendler, J., and Lassila, O., "The Semantic Web," Scientific American, May2001.
33. Paul J. Leach and Rich Salz. Network Working Group. UUIDs and GUIDs. February, 1998.
Page 10 of 10