Change locking to allow multiple connections per URL and variable locking

12547
Reporter: lfrancke
Assignee: lfrancke
Type: Improvement
Summary: Change locking to allow multiple connections per URL and variable locking
Priority: Major
Resolution: Fixed
Status: Closed
Created: 2012-12-17 11:35:41.38
Updated: 2013-12-17 15:46:38.902
Resolved: 2012-12-20 11:20:58.44
        
Description: We want to be able to configure, on a per dataset basis, how many simultaneous connections we allow and to change the locking time.

* Decide if Locking is needed at all?
* If locking is needed how many connections do we allow?
* If locking is needed how long do we lock?
** Static: x seconds
** Dynamic: Lock as long as the lock was previously held (with min and max times)

This should be relatively easy to do by introducing a few more understood properties in the crawler coordinator and depending on those change locking strategies. We'd also need to change the current ZooKeeper lock to use a Semaphore instead of a simple lock.

And the dynamic behavior can either be added to the existing class or introduced as a new class.]]>