Issue 12732

Message send on multiple threads appears to compete and increase per-thread send time

Reporter: omeyn
Assignee: lfrancke
Type: Bug
Summary: Message send on multiple threads appears to compete and increase per-thread send time
Priority: Critical
Resolution: WontFix
Status: Closed
Created: 2013-02-18 14:26:33.047
Updated: 2013-12-06 12:06:32.51
Resolved: 2013-03-08 10:42:55.174
Description: When calling MessagingService.send(Message) I'm seeing the following increases in time spent in the send call (per thread).

threads time spent (ms per thread)
10      10
50      60
100     110

Suggests there's an underlying resource for which they're competing.]]>

Comment: Further testing/digging reveals all sending happens on one thread, so that's the contention.  Adding a threadpool to do the sending (in my case 100 threads) fixes the problem. Suggest some kind of configurable threadpool for that.
Created: 2013-02-22 16:14:07.787
Updated: 2013-02-22 16:14:07.787

Created: 2013-03-08 10:42:45.855
Updated: 2013-03-08 10:42:45.855
It is true that performance stalls somewhere between 10 and 100 concurrent threads. In naive tests with a local message broker I got a peak of about 2500 msg/s. Further debugging using YouTrack confirmed that the contention is down at the Socket layer.

To improve on this we could use connection pooling in the postal-service or the client itself could open multiple connections and use multiple MessagingServices.

Closing as WONTFIX for the moment but will create a feature request to implement pooling.

Created: 2013-03-08 17:19:59.693
Updated: 2013-03-08 17:19:59.693
Just to put my observations in this thread, to keep them:
  Single thread operation from laptop in office on the gave 4000 sends per second (scanning MySQL on a forward only cursor)