Issue 12732
Message send on multiple threads appears to compete and increase per-thread send time
12732
Reporter: omeyn
Assignee: lfrancke
Type: Bug
Summary: Message send on multiple threads appears to compete and increase per-thread send time
Priority: Critical
Resolution: WontFix
Status: Closed
Created: 2013-02-18 14:26:33.047
Updated: 2013-12-06 12:06:32.51
Resolved: 2013-03-08 10:42:55.174
Description: When calling MessagingService.send(Message) I'm seeing the following increases in time spent in the send call (per thread).
threads time spent (ms per thread)
10 10
50 60
100 110
Suggests there's an underlying resource for which they're competing.]]>
Author: omeyn@gbif.org
Comment: Further testing/digging reveals all sending happens on one thread, so that's the contention. Adding a threadpool to do the sending (in my case 100 threads) fixes the problem. Suggest some kind of configurable threadpool for that.
Created: 2013-02-22 16:14:07.787
Updated: 2013-02-22 16:14:07.787
Author: lfrancke@gbif.org
Created: 2013-03-08 10:42:45.855
Updated: 2013-03-08 10:42:45.855
It is true that performance stalls somewhere between 10 and 100 concurrent threads. In naive tests with a local message broker I got a peak of about 2500 msg/s. Further debugging using YouTrack confirmed that the contention is down at the Socket layer.
To improve on this we could use connection pooling in the postal-service or the client itself could open multiple connections and use multiple MessagingServices.
Closing as WONTFIX for the moment but will create a feature request to implement pooling.