Transport types

Replies to This Discussion

Permalink Reply by christian neess on June 19, 2014 at 5:21am

I found this link with a good description

http://www.neudesic.com/neuron/Help3/Neuron.htm#Development/Develop...

• ▶ Reply

Permalink Reply by Martin Wasznicky on June 19, 2014 at 7:01am

Hello,

Each transport carries their own levels of quality of service.

TCP and named pipes are great when you need in memory routing, fast and low latency. These tend to be appropriate for request/response type of messaging (though they can be used for multicast/one way messaging), the type that occurs when using our service broker to mediate service calls (i.e. Client and Service Connectors are a good example). For these types of calls, durability and transactions don't necessarily make sense since if the call fails...times out, etc, the typical use pattern for a user is that they make the request again. Plus you can always capture the message with Neuron Auditing.  TCP and Named Pipes are a "Push" subscription model. That means the only way a subscriber will receive the message is if it is "alive" at the time publication and delivery. The internal TCP/Named Pipes Neuron Publishing service determines all the eligible "alive" i.e. connected recipients and "pushes" a copy of the message out to all "instances" of the recipients. In this case you may be using the same Neuron Party (i.e.recipient, subscriber, etc) in more than one place.  This means, as long as they are connected....each instance will receive a copy of the message. These transports can also handle large messages (i.e >10MB). They both are large message optimization knobs you can turn on in the transport properties. Named Pipes can sometimes be more convenient than TCP in so far as that you don't have to deal with TCP port configuration and ensuring that these Ports are not being used on the Neuron server. With both transports, since they are a push model, you could easily publish more messages than the subscribers can process. In this case, messages are stored in memory. In this situation you can apply topic level publisher throttling. In short:

TCP/IP

 - Allows for Server and Client Side Port configuration

 - based off WCF NetTcp Transport

 - Supports Transport level Ordered, Reliable Sessions, Encryption

 - Non Durable/In memory transport

 - Supports multiple “Instances” of a Party

 - Push Subscription Model

 - Can be used by remote client API hosted in .NET applications

Named Pipes

 - NO need for TCP Port configuration

 - based off WCF NetNamedPipe Transport

 - Supports Transport level Encryption

 - Non Durable/In memory transport

 - Supports multiple “Instances” of a Party

 - Push Subscription Model

 - Can NOT be used by remote client API hosted in .NET applications

PEER is an interesting protocol in so far that it is essentially in memory routing and can be used for one way as well as request/response traffic and can be used on remote machines by the client API.  However, its different in so far that there is no Neuron publishing service. There is a Neuron "Peer" resolver that every recipient (i.e. subscriber) registers with when they connect.  In fact, if you use the Neuron Test Client, you'll see that even when you connect a Party to the bus, it will not go Online until it detects that at least one other recipient is also connected.  This is the nature of the protocol in so far all connected clients communicate over the network and form a "mesh". They ping their nearest neighbor and internally store a list of closest neighbors in the mesh.  The mesh is constantly reorganizing itself. This protocol tends to scale very well for Many, many subscribers, in fact to hundreds, where Named Pipes and TCP would not. Also, once connected, if the Neuron runtime went down, the client API would still be able to send and receive messages. Peer can also handle relatively large messages (i.e. >10MB) in short:

Peer

 - WCF Peer Network Protocol (not UDP)

 - Neuron provides in Memory and Persisted Peer Resolver

 - Non Durable Transport

 - Scales out to hundreds/thousands of subscribers

 - Supports Pub/Sub even if Neuron ESB Service is offline

 - Supports multiple “Instances” of a Party

 - Broadcast Subscription Model

 - Topic = Mesh

MSMQ and RabbitMQ tend to be more appropriate for "fire and forget" traffic...or where you need to be able to "queue" up messages for subscribers that may not necessarily be available at the time.  Great for when you don't want to lose messages...since these can be configured for both durability and transactions. Imagine you publish messages, but the service or adapter endpoints are offline or disabled.  After the message has been published, you could bring these on line...and they would receive the published messages since they were all queued.  There are some essential differences though.

MSMQ - This ships with Windows and can be installed as a windows component.  It's message size limit is 4MB. It tends to be pretty fast and can be configured so that Durability and Transactions are optional.  When both are set to false, this transport is usually faster than TCP, Named Pipes and Peer. When using Durable/Transactions, its usually faster than RabbitMQ.  Both Rabbit MQ and MSMQ rely on disk I/O...so optimization the disk is important for performance. The Neuron Explorer allows users to manage the underlying queues and manage the messages in underlying queues. Users can even view/edit the messages in the underlying queues. The MSMQ Transaction property, when set, uses the MSDTC (the Microsoft Distributed Transaction Coordinator). This means that when used by remote Client API, the MSDTC must be installed (its a Microsoft windows component) on all machines involved and configured for remote network access and transactions. These are true atomic transactions. MSMQ as a protocol guarantees reliable, once only, delivery of the message. The transport can also be configured for Ordered message delivery. The upside of this "Pull" subscription model is that subscribers only take what they can consume at the time. Hence you can't overwhelm a subscriber. Pending messages just remain in the underlying queue that maps to the subscriber. Lastly, for both RabbitMQ and MSMQ, there is only one physical queue mapped to a subscriber. If multiple "instances" of the subscriber are used, only one instance will receive the published messages. A round robin load balance style of delivery will occur between the multiple instances. The underlying MSMQ queues that map to Neuron ESB Topics and Parties have to manually created though through the Neuron Explorer UI or through the scripts generated from the UI.

Rabbit MQ - Rabbit MQ offers both durability and transaction options. However, Rabbit MQ does not offer "Atomic" transactions. Their transactions are really based on acks/nacks which Neuron manages.  in some cases, Rabbit MQ says that their transaction acks/nacks can get lost on the network, so we've built in automatic resubmits within the transport. Also, Rabbit MQ does not (out of the box) provide guaranteed, reliable, once only delivery of messages, Neuron adds those capabilities to the transport though. Rabbit MQ by default is ordered delivery and does not support Request/Response traffic. Within the Neuron Explorer, you can view the number of messages that are pending delivery for the underlying rabbit mq queues mapped to Neuron parties, but Rabbit MQ doesn't expose a way to view or edit pending messages. Rabbit MQ does have a pretty cool clustering option that MSMQ doesn't have.  For MSMQ High Availablity, one has to use windows clustering service and cluster the disk resources. Essentially creating active/passive clusters. When one fails..the node fails over and becomes active on the other machine. With Rabbit MQ, you can use rabbit mq clustering to "link" two or more rabbit MQ nodes together (no need for Windows Clustering) and essentially "mirror" all queues between them. The idea is that when a message is written to one node..its automatically mirrored on the other.  Within neuron we've built in some incredible support for Rabbit MQ clustering.  If a node goes down and Neuron Parties are in the middle of publishing or receiving messages, there will barely be a hiccup noticed. Under the hood we'll automatically roll over to the other node if we detect any failure and resubmit any messages that were un acknowledged.  we've built in all the duplicate message detection. If Msmq fails in a clustered scenario, it could take up to several minutes for the MSMQ instance to roll over and become active. In those cases publication will fail and those messages would need to be manually resubmitted. The underlying Rabbit MQ queues that map to Neuron Topics and Parties are always automatically created. There is never a need to manually create these.

In short, Rabbit MQ and MSMQ are great for store and forward type of messaging. We've also built in automatic dead letter and poison queue handling..these messages are automatically detected and placed in our failed history table.  Although MSMQ has a 4MB limit on message size, Rabbit MQ does not.  Both MSMQ and Rabbit MQ can be used effectively with the Remote Client API. Though there is probably more configuration involved using MSMQ, MSMQ is probably preferred over using TCP for the remote Client API usage since you get automatic load balancing.

does this help?

• ▶ Reply