Issue 10748

Design and integrate the event broadcasting mechanism for registry operations

Reporter: trobertson
Assignee: kbraak
Type: Epic
Summary: Design and integrate the event broadcasting mechanism for registry operations
Priority: Major
Resolution: WontFix
Status: Closed
Created: 2012-02-02 16:06:21.04
Updated: 2017-10-06 15:25:13.756
Resolved: 2017-10-06 15:25:13.74
Description: Many people, modules and components wish to monitor the registry and be notified of changes.  This task involves thinking through this process and designing, and then implementing an event publishing mechanism.
Events might include:
- organization created
- data provider endorsed
- new dataset discovered
- dataset renamed

The events should be well documented, so others (e.g. Tim Hirsch) might read it and understand what events they could potentially subscribe to (with subsequent implementation).

It is possible in thinking through the endorsement procedure that there needs to be some subscription activities as well.  ]]>

Comment: Only applies in part, as far as notification subscription option is requested. Mostly, this will rather be handled in another work line concerning statistics/tracking (activity, validation, dynamics over time etc)
Created: 2012-02-02 16:44:09.15
Updated: 2012-02-02 16:44:09.15

Comment: I have added a new wiki page detailing the various events here: 
Created: 2012-02-09 20:02:07.103
Updated: 2012-02-09 20:02:07.103

Comment: I have updated the wiki page with a table for each set of events by Node, Organization, and Dataset. The table should be an easy reference for the list of events expected, and to understand what they mean and what their message payload will be.
Created: 2012-02-10 17:05:08.619
Updated: 2012-02-10 17:05:08.619

Comment: A 2nd draft of the document has been saved. Changes were made following discussion with Andrea this afternoon.
Created: 2012-02-14 19:41:58.95
Updated: 2012-02-14 19:41:58.95

Comment: Do we need to add Network and TechnicalInstallation events to that list? Or should some events apple to several classes?
Created: 2012-02-14 21:40:06.644
Updated: 2012-02-14 21:40:06.644

Created: 2012-02-15 10:24:29.581
Updated: 2012-02-15 10:26:35.44
(This is a bit technical, but don't know other place to discuss about it)

Yesterday, one concern surfaced about where are messages going to be sent from. As of now, messages are going to be triggered from these areas:

- the registry-synchronizer project (e.g. new DS discovered and added to the Registry DB)
- IPT (adding new DS)
- RegisterMe
- Admin console
- others...?

Yesterday having a quick chat with Fede, we talked about who should be the one sending this kind of messages. We kind of agreed that the registry-persistence project should be the one broadcasting these messages. the registry-service project can't be used as the reg-synchronizer will use the persistence layer directly. Would you agree that broadcasting these messages from reg-persistence is the best idea. Im eager to read any possible problems with this appraoch. With this approach, any creation/update/deletion will always emit a message.


Created: 2012-02-15 11:38:13.024
Updated: 2012-02-15 11:38:13.024
One problem we could face is that the operations are too granular. For example when merging 2 organizations we probably call a delete and an update method in the persistence layer. But planned right now is to emit a merged message. There might be similar cases. Don't know if this really is an issue - maybe it doesn't hurt to issue both the atomic operation events and any overarching, bigger events?

CLB Indexing might also be triggering new events btw. When the catalogue of life in indexed clb might find new subdatasets (GSDs) which need to be registered as datasets. But maybe this should be done by the dwca synchronizer instead???

Created: 2012-02-27 18:31:29.047
Updated: 2012-02-27 18:31:29.047
A 3rd draft of the document has been prepared.

A review of the events against the Registry API has been completed, and an additional column added to the event overview table detailing the service method that handles the event. Two new methods were added to the API in revision

A 1st set of interfaces covering all events in the overview table (that need to be implemented in the Registry's message broadcast notifications) have also been added to the API in revision