Skip to main content

ConnectionPool performance improvements



Hi, all!

Today I committed a change to Npgsql which will improve connection pool performance. This change was motivated by Andrew's bug report where he noticed that a lot of threads were waiting to get a new connection from pool.

In order to keep consistence of the pool, Npgsql has to lock access to it. Andrew's problem appeared in a busy server where a lot of threads were trying to get a new connection from the pool. They had to wait in line. And obviously this isn't good.

The current implementation of Npgsql creates a big lock surrounding all the code needed to work with the pool and more! As Andrew noticed in his bug report, I/O operations were being done inside this lock which was contributing to more delays to get a connection from the pool.

So, to fix that, I rewrote connection pool logic to remove this big lock and break it down to smaller ones only when really needed. All the I/O operations were left out of the locks, this way, other threads waiting to get a new connection from the pool don't need to wait for those expensive operations to finish.

I made a lot of tests and could confirm that when I break the code inside the debugger, threads are spread throughout connection pool code as expected instead of waiting in line on the big lock.

As this change is somewhat critical to Npgsql usage, I'd like to ask you to download the code, compile it and give it a try and see if everything is working ok or even better than before. I expect busy servers to be able to increase their raw throughput because it will have to wait less to get connections from the pool.

As always, please, let me know if you have any problems and all feedback is very welcome!

Comments

Bryan Murphy said…
Wow! Can't wait to try the new versions. The behavior has been giving us grief for months. We ended up solving it by simply making the connection pool much bigger than necessary.
09 January, 2012 16:20
Slava said…
this is really cool. thanks for working on it!
10 January, 2012 21:52
Bruno Lessa said…
Bom dia Francisco !

Me diz uma coisa essa modificacao ja está na 11.93 ?

Estou teno mtos problemas com o pool na minha aplicação. Ultimamente tenho que restartar o IIS para que para de dar too many clients already.

Será que aumentando o maxpoolsize , melhora ?

abraco !
17 May, 2012 11:49
Bruno Lessa said…
Bom dia Francisco.

Esta aleração já está na 11.93 ? É pq comecei a ter problemas com as conexões na minha app. Tenho as vezes que restartar o IIS para que volte a funcionar. Fica dando too many clients already direto.

Será que esta alteraç~çao resolve ? Ou entao aumentar o maxpoolsize ?

Abs !!!
17 May, 2012 11:51
Post a Comment

Popular posts from this blog

Npgsql Tips: Using " in (...)" queries with parameters list and "any" operator

Hi, all! We have received some users questions about how to send a list of values to be used in queries using the "in" operator. Something like: select foo, bar from table where foo in (blah1, blah2, blah3); Npgsql supports array-like parameter values and the first idea to have this working would try to use it directly: NpgsqlCommand command = new NpgsqlCommand("select * from tablee where field_serial in (:parameterlist)", conn); ArrayList l = new ArrayList(); l.Add(5); l.Add(6); command.Parameters.Add(new NpgsqlParameter("parameterlist", NpgsqlDbType.Array | NpgsqlDbType.Integer)); command.Parameters[0].Value = l.ToArray(); NpgsqlDataReader dr = command.ExecuteReader(); but unfortunately this won't work as expected. Npgsql will send a query like this: select * from tablee where field_serial in ((array[5,6])::int4[]) And Postgresql will complain with the followin...
45 comments
Read more

Using Entity Framework 6 with Npgsql 2.1.0

UPDATE (2014-05-19): Marek Beneš noticed a problem in the default connection factory config. It is fixed now. Thanks, Marek! UPDATE (2014-02-20): I created a new post explaining how to get Npgsql 2.1.0. Although this post is about EF 6, I'd like to talk about our current situation to support both EF 6 and EF4.x which explain why there are some subtle changes between EF 4.x and EF 6.x App.config settings.  Support for EF versions 4.x and 6.x Sometime after we started to work on Npgsql 2.1.0, we started to add code to support EF6 and decided to reorganize our Entity Framework support code. Shay created a pull request to organize this change and isolate the EF code out of core Npgsql code. The result was the creation of two separated assemblies: Npgsql.EntityFramework.dll for EF6 and above; Npgsql.EntityFrameworkLegacy.dll for EF4.x. Only when using Npgsql with EF6 you will need to reference Npgsql.EntityFramework.dll assembly. This is needed because the EF ...
26 comments
Read more

Npgsql Design time support preview available for download!

UPDATE2 : For some reason, Blogger lost the original post. I had to republish it. UPDATE : Jerónimo told me that you  must  use the Npgsql version which is inside the zip file in order to make it work. After so much time (more than I wanted it to take) Npgsql finally has a initial design time support for Visual Studio! This work was done by Jerónimo Milea. Jerónimo let me know he was working on DDEX support on this thread  after I said I was working on DDEX support for Npgsql. He sent me his working copy and I started to play with it. Note that as a preview version many things may not work ok and we wanted you to provide us feedback so we can fix any bugs. You can download the project file from our downloads page . Design time support is provided as a zip file containing support code as well as a copy of Npgsql project file. So, everything you need to start working with design time support is already packaged for you. When you unzip the file, you will have ...
42 comments
Read more