Skip to main content

Better tracing of Npgsql connection pool usage in the works



Sometimes, Npgsql users receive errors when working with connection pooling. The problem appears when they try to open a new connection and receive the following message: 

"Timeout while getting a connection from pool."

This is caused when an attempt is made to get a connection from a pool which  has all its connections used. Most of the time this problem is a difficult one to track because it generally happens when the system is in production and not in the development phase. Of course. :)

Some time ago, I received a report from Miłosz Kubański about such a problem. I told him I would work in a way to get more information so we could check what was happening.

In order to help us find the problem, I added a little "hack" to Npgsql: Whenever an error while getting a connection from the pool occurs, Npgsql would log a stacktrace of the allocation of all the connections which were in the pool. Theoretically, those connections which were allocated should not be retained and by taking a "snapshot" of those allocations, we could get some tips about which code allocated the connection and check for potential missing releases.

Last week I sent Milosz this modified version of Npgsql so he could give it a try and send me back the log. Yesterday he sent me the log. Inside it, I chose two stacktraces and asked him if would be possible there was any missing close.

Milosz replied saying there was indeed a missing close call. It was an exception inside a very complex system. 

Based on this feedback, I think this "hack" has shown to be very useful in future cases and I'll work to get this change in a "non-hack" status and add it to Npgsql code. I hope this helps Npgsql users to find possible causes for connection pooling problems.

When this feature is available I'll let you know.



Comments

Unknown said…
This would be very useful -- any timeframe on when this feature will be available?
Rui Guimarães said…
This seams really useful. Do you have any update on this?
Unknown said…
Francisco, did the "hack" ever made it to Npgsql 2.1.0 or 2.1.1?
Hi all!

Sorry, but this feature isn't present in current Npgsql release. I'm working to add it to master branch and I hope it will be present in the 2.2 release.
Unknown said…
Hi, This 'Hack' would be useful for an issue we are investigating right now. Is this logging available in the 2.2 beta 1 release?

Thanks.
Hi, all!

We have an issue open on github about that:
https://github.com/npgsql/Npgsql/issues/237

I already have a patch to include this feature. Unfortunately, I think it will be present in the 2.3 release as the 2.2 is already in the RC phase.

Please, check the 237 issue above for news about this feature. I'll update the patch so it can be cleanly applied and you all can test it. This would be very helpful.

Thanks in advance and sorry for not having this feature released yet.
I just added more information to the issue tracker. I also updated the patch and created binaries with this patch applied which you can give it a try.
Check out https://github.com/npgsql/Npgsql/pull/311 for more info.

I hope it helps.

Popular posts from this blog

Npgsql Tips: Using " in (...)" queries with parameters list and "any" operator

Hi, all! We have received some users questions about how to send a list of values to be used in queries using the "in" operator. Something like: select foo, bar from table where foo in (blah1, blah2, blah3); Npgsql supports array-like parameter values and the first idea to have this working would try to use it directly: NpgsqlCommand command = new NpgsqlCommand("select * from tablee where field_serial in (:parameterlist)", conn); ArrayList l = new ArrayList(); l.Add(5); l.Add(6); command.Parameters.Add(new NpgsqlParameter("parameterlist", NpgsqlDbType.Array | NpgsqlDbType.Integer)); command.Parameters[0].Value = l.ToArray(); NpgsqlDataReader dr = command.ExecuteReader(); but unfortunately this won't work as expected. Npgsql will send a query like this: select * from tablee where field_serial in ((array[5,6])::int4[]) And Postgresql will complain with the followin...

Fixed! LOG: unexpected EOF on client connection

Hi all! Since we implemented connection pool in Npgsql, we received some complaints about EOF log messages being generated on Postgresql logs when using Npgsql. This was caused by Npgsql not sending the proper terminate message to Postgresql on pooled connections when the application terminated or more specifically when the assembly was unloaded. This is a long time problem with Npgsql connection pool. I even talked about it in the past . Up to now, I had no idea about how to fix that as I wasn't able to close the connections in the pool. When I tried to put a finalizer in NpgsqlConnectorPool, which would be triggered when the assembly was unloaded, I received object already disposed exceptions when trying to send something to the stream. That's when I came up with the "excellent" idea of subclassing the networkstream class and override its Dispose method so that I could send the postgresql terminate message before it was disposed! :) It worked like a charm! ...

Npgsql 2.2.0 final release is out!

This is Npgsql 2.2.0 Final Release This release contains 249 commits since the last stable release. Includes bug fixes, improvements and new features. Update notice: If you have been using Npgsql 2.2.0-rc2, you don't need to update to this version. They are the same except for the Assembly version information. Major highlights Visual Studio DDEX support   Kenji Uno added support for DDEX. Now you can use Npgsql with Visual Studio data designer. This is a missing feature a lot of our users requested in the past. Kenji added a tutorial about how to use Npgsql with DDEX. You can find it here: https://github.com/npgsql/Npgsql/wiki/Visual-Studio-Design-Time-Support---DDEX-Provider#install-npgsqlddexprovidervsix   Entity Framework   David Karlaš added support for EFMigration and Database creation in EF6+. Now it is possible to start Code First projects without needing to create a database upfront. EntityFramework and Npgsql will take care of it. E...