Hi,
we ha a setup where a client subscribes to multiple topics. Sometime, due to wrong setup, the client cannot be reached due to firewall issue.
If a message cannot be delivered Neuron retries immediately over and over again without any timeout or TTL on the messages. This causes the CPU to spike and remain constantly above 100%.
The following error is logged:
2017-08-31 14:47:38.250+02:00 [71] ERROR - Unable to connect the client with the topic. Party Id MultipleSubscriber, session Id 4224f99e-676b-4cae-97e8-353cc2294f28, topic Location, callback net.tcp://XXX.XXX.XX:61009/ESBTcpPubSubClient/4224f99e-676b-4cae-97e8-353cc2294f28.
System.ServiceModel.CommunicationException: Unable to communicate with client. Ensure that the client's firewall is not blocking Neuron traffic. Party Id MultipleSubscriber, session Id 4224f99e-676b-4cae-97e8-353cc2294f28, topic Location, url net.tcp://DK-W-K1KEP.kamstrup.dk:61009/ESBTcpPubSubClient/4224f99e-676b-4cae-97e8-353cc2294f28, Exception: Could not connect to net.tcp://XXXXXXXXXX:61009/ESBTcpPubSubClient/4224f99e-676b-4cae-97e8-353cc2294f28. The connection attempt lasted for a time span of 00:00:21.0026596. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.20.203.10:61009. . ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://XXXXXXX:61009/ESBTcpPubSubClient/4224f99e-676b-4cae-97e8-353cc2294f28. The connection attempt lasted for a time span of 00:00:21.0026596. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.20.203.10:61009. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.20.203.10:61009
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open()
at Neuron.Esb.Channels.WcfPublishingService`1.CreateClientProxy(TProperties properties, String clientEndpointAddress, String clientIdentity)
at Neuron.Esb.Channels.WcfPublishingService`1.ClientProxyPool.CreateProxy()
at Neuron.Esb.Internal.Pool`1.Take()
at Neuron.Esb.Internal.Pool`1.Use(Action`1 action)
at Neuron.Esb.Internal.FixedSizePool`1.Use(Action`1 action)
at Neuron.Esb.Channels.WcfPublishingService`1.UseProxyConnect(TcpClientInfo client, Action`1 action)
at Neuron.Esb.Channels.WcfPublishingService`1.Connect(TcpClientInfo clientInfo)
--- End of inner exception stack trace ---
at Neuron.Esb.Channels.WcfPublishingService`1.Connect(TcpClientInfo clientInfo)
The obvious solution would be of course to solve the firewall issue, but is there any setting that allow us to discard the message if the client cannot be reached within x number of retries or seconds?
Thank you very much
Fabrizio
Tags:
unfortunately, not now. however, if this is coming from the TCP publishing log file (i.e. Topic), if the publishing service fails to reestablish communication with the client within a minute or so (I can't remember what the exact period is) the client will be removed from teh delivery pool and the topic will not attempt to talk to it again.....so this shouldn't report errors for longer than a minute or so
Hi Marty,
the log is taken from the TCP Publishing service but unfortunately the service retried for several hours, until we noticed the CPU spike and restarted the whole instance.
I'm looking at the log and it goes on from 14:47 to 23:59, and probably the day after as well, so maybe the client is not removed properly?
How would you recover once the client is removed, restarting the service?
Fabrizio
actually...I just looked. this is at the Connect stage...so it will simply try to connect forever. its only after connect....if we get blocked delivering a message is where we remove the client..
in the Neuron ESB endpoint health monitor, you can just restart the Topic, rather than restart the entire Neuron ESB runtime
Hi Marty,
we are still experiencing this issue.
Looking at the logs it seems like the client is not removed properly, there's a WARNING saying that the client will be removed and after 15sec it logs an error on the same client.
2017-10-17 00:46:46.418+02:00 [84] WARN - The client could not be contacted after 120 seconds. The client is now expired and will be removed from the connected pool list. Party MultipleSubscriber, Session Id 402ddc9f-918f-4e8a-a393-8ff5292808f6, Topic Container, URL net.tcp://DK-W-T1KEP.kamstrup.dk:61009/ESBTcpPubSubClient/402ddc9f-918f-4e8a-a393-8ff5292808f6
2017-10-17 00:47:07.457+02:00 [112] ERROR - Unable to connect the client with the topic. Party Id MultipleSubscriber, session Id 402ddc9f-918f-4e8a-a393-8ff5292808f6, topic Container, callback net.tcp://DK-W-T1KEP.kamstrup.dk:61009/ESBTcpPubSubClient/402ddc9f-918f-4e8a-a393-8ff5292808f6.
System.ServiceModel.CommunicationException: Unable to communicate with client. Ensure that the client's firewall is not blocking Neuron traffic. Party Id MultipleSubscriber, session Id 402ddc9f-918f-4e8a-a393-8ff5292808f6, topic Container, url net.tcp://DK-W-T1KEP.kamstrup.dk:61009/ESBTcpPubSubClient/402ddc9f-918f-4e8a-a393-8ff5292808f6, Exception: Could not connect to net.tcp://dk-w-t1kep.kamstrup.dk:61009/ESBTcpPubSubClient/402ddc9f-918f-4e8a-a393-8ff5292808f6.
Neuron ESB Product Support Forums and Communities
© 2024 Created by Neuron Admin. Powered by