Author: jcuadra@gbif.org
Created: 2012-03-22 11:58:06.096
Updated: 2012-03-22 11:59:39.428
To recap my experience with this. The problem was present in the registry-persistence as well, because for each test method, a guice injector was created and this triggered connections being created in the pool. For some strange reason, after the method finished, those connections were not closed properly (apparently the default is 6 connections). After each test class was finished with its methods, the pool was cleaned. The persistence project never failed because the amount of tests per class never exceeded a certain number that will make the pool go over the limit. The "fix" in the registry-persistence was to create the injector one time per class (@BeforeClass) and not per method (@Before). This only left 6 (default) idle connections per class, but were cleaned after the class was over.
The problem also happens in the registry-ws-client, specifically the DatasetWsClientIT class, which has around 27 tests. Each test method was creating 6 connections and leaving them idle for all the duration of the test class...BUT before the last test method finished, the pool exploded (aprox 27x6 connections) so mysql sent a "Connection exceeded" exception , causing the test to fail.
I tried also to create the injector per class, but this didn't solve the problem. So now I hit this brickwall, and while debugging the application I noticed a Grizzly instance is being launched each time a test method is executed (This also might explain why the tests take soooo long). Grizzly logs that it starts a 20 item-pool, but also logs that it frees them up after Grizzly shuts down, BUT here is my main suspicion because if I debug the application I see that those connections are left in the DB.
This happens with Bone, C3p0. This does not happens with DBCP.
A thing I haven't tried yet is to launch one Grizzly instance per test class...
Author: lfrancke@gbif.org
Created: 2012-03-22 12:12:36.529
Updated: 2012-03-22 12:12:36.529
Obviously the connections are not closed properly. This is almost certainly our fault and not a bug. Additionally this sounds like a memory leak somewhere. Your fix is easy to break by just changing surefire to not run all classes in different JVMs hence the need to find the root cause.
Your change from {{@Before}} to {{@BeforeClass}} also changed the meaning and behavior of the tests and I'd consider it a problem at least without a careful look at the tests. Because all tests now share one Injector and DB instance etc. they might screw each other up depending on the order they run in (and this order may very realistically change).
Have you ever just attached a debugger and profiler and stepped through the code to see what happens after a test? Also yeah the tests take long but that's no big deal that's what we have Jenkins for. Faster tests are obviously better though. So if one shared Grizzly is fine for everything then even better but to share a DB you actually need to plan the tests to work that way.
Author: mdoering@gbif.org
Created: 2012-03-22 13:04:55.49
Updated: 2012-03-22 13:04:55.49
BoneCP requires that the pool is explicitly closed. Integration tests in the CLB project kept piling up idle connections too because the pool was never closed and the base minimum connections remained forever. We do close the datasource properly for the dbunit in DatabaseDrivenTestRule.after(), but I suspect that the pool remains in every grizzly instance and needs to be closed explicitly somehow for the real integration tests.
Author: trobertson@gbif.org
Created: 2012-03-22 16:02:27.97
Updated: 2012-03-22 16:02:27.97
This is fixed for the client tests, but potentially issues still exist with connection pools. Grizzly was being started up for each method, and on stop appeared to be doing a graceful shutdown. By making a reusable Grizzly container, we have 1 grizzly per test class, and thus less connection.
If there are leaky connections in the pool remains to be determined, but this specific issue is closed as all registry-ws-client tests are now uncommented and pass.