I want to implement following scenario using Neuron ESB:
- get big list of working items from somewhere (let's say some database)
- split this list to small "batches" of working items
- send these batches each to particular instance of specific subscriber (Worker), so each that instance will process assigned batch in parallel with other instances
- collect feedback (is processing complete/failed) + implement some fail-over mechanism (resend failed batches, monitor if batches weren't processed during some time interval and etc.)
I started to implement special adapter for Neuron, which I called Dispatcher.
The Dispatcher acts as a subscriber for one Topic (A) and can publish to another Topic (B).
The ides was to get from Topic A:
- list of working items
- requests from Worker to add/remove them into/from worker list
Then, when Dispatcher gets a list of working items, then split that list into batches, and then publish each batch to Workers using some load-balancing algorithm (let's say round-robin).
Unfortunately I've got following problems:
- I did not find how to identify an instance of Worker subscriber. The DestId property of ESBMessage works only when I put Party identifier in it. So, as in my case, when each Worker is actually an instance of the same Party, all Workers will get the message (I used TCP Topic);
- Modification of esbService.exe.config is required if you want to send messages under specific part from Neuron Adapter ( have to be added, otherwise Party.Connect() method will fail).
Does anybody tried to implement load-balancing in Neuron ESB (not using MSMQ)?
Or, do you know how to by-pass problems I have?
Tags: balance, dispatching, distribution, load
-
▶ Reply to This