From jan.murre at catalyz.nl Thu Nov 9 21:02:53 2017 From: jan.murre at catalyz.nl (Jan Murre) Date: Thu Nov 9 21:02:59 2017 Subject: [egenix-users] Pooling connections Message-ID: Hi, We are using mxODBC with the native Microsoft MS SQL Linux driver (version 13.0) and we are opening and closing a connection for every database call. This is considered to be not very efficient, although I am not sure if mxODBC does some clever pooling of connections or not. We tried to add pooling at the unixODBC level, according to this: http://www.unixodbc.org/doc/conn_pool.html [ODBC] Pooling=Yes And in the drivers section: CPTimeout=120 After that, our connections fail to work with this error: [unixODBC][Driver Manager]Driver does not support this function My question is, what is the best way to do connection pooling with mxODBC and the native Microsoft MS SQL linux driver? Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: /mailman-archives/egenix-users/attachments/20171109/32770b20/attachment.htm From mal at egenix.com Sun Nov 12 16:47:33 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Sun Nov 12 16:47:50 2017 Subject: [egenix-users] Pooling connections In-Reply-To: References: Message-ID: On 09.11.2017 21:02, Jan Murre wrote: > Hi, > > We are using mxODBC with the native Microsoft MS SQL Linux driver (version > 13.0) and we are opening and closing a connection for every database call. > > This is considered to be not very efficient, although I am not sure if > mxODBC does some clever pooling of connections or not. > > We tried to add pooling at the unixODBC level, according to this: > http://www.unixodbc.org/doc/conn_pool.html > > [ODBC] > Pooling=Yes > > And in the drivers section: > CPTimeout=120 > > > After that, our connections fail to work with this error: > [unixODBC][Driver Manager]Driver does not support this function unixODBC connection pooling often creates issues with drivers. We recommend against using this. > My question is, what is the best way to do connection pooling with mxODBC > and the native Microsoft MS SQL linux driver? The best way is to do connection pooling at the application level. You typically only need to open connections once per application (and reopen them in case they fail for some reason). In the application itself, you then just run conn.commit() and conn.rollback() to delimit the transactions and then open cursors for running queries. The simplest way is to open a new cursor for every query, but it's, of course, also possible to use cursors for multiple queries. You can also go one step further and cache cursors, i.e. always using the same cursor for the same common query, without closing them. You only have to make sure that you close the result sets when passing them back to the pool (using cursor.flush()). mxODBC it self does not provide connection pooling, since getting this right is difficult if you don't know what the application is doing (e.g. adjusting connection settings or keeping cursors alive). We do have connection pooling in mxODBC Zope DA and it's working well for Zope. Best Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 12 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From jan.murre at catalyz.nl Tue Nov 14 21:24:34 2017 From: jan.murre at catalyz.nl (Jan Murre) Date: Tue Nov 14 21:24:42 2017 Subject: [egenix-users] Pooling connections In-Reply-To: References: Message-ID: On Sun, Nov 12, 2017 at 4:47 PM, M.-A. Lemburg wrote: Thanks for your thourough answer! On 09.11.2017 21:02, Jan Murre wrote: > > Hi, > > > > We are using mxODBC with the native Microsoft MS SQL Linux driver > (version > > 13.0) and we are opening and closing a connection for every database > call. > > > > This is considered to be not very efficient, although I am not sure if > > mxODBC does some clever pooling of connections or not. > > > > We tried to add pooling at the unixODBC level, according to this: > > http://www.unixodbc.org/doc/conn_pool.html > > > > [ODBC] > > Pooling=Yes > > > > And in the drivers section: > > CPTimeout=120 > > > > > > After that, our connections fail to work with this error: > > [unixODBC][Driver Manager]Driver does not support this function > > unixODBC connection pooling often creates issues with drivers. > We recommend against using this. > > Yeah, we tried using the pooling feature, but it does not seem to work at all. Just strange errors! > > My question is, what is the best way to do connection pooling with mxODBC > > and the native Microsoft MS SQL linux driver? > > The best way is to do connection pooling at the application > level. You typically only need to open connections once > per application (and reopen them in case they fail for some > reason). > > In the application itself, you then just run conn.commit() > and conn.rollback() to delimit the transactions and then open > cursors for running queries. > > The simplest way is to open a new cursor for every query, > but it's, of course, also possible to use cursors for multiple > queries. > > You can also go one step further and cache cursors, i.e. always > using the same cursor for the same common query, without closing > them. You only have to make sure that you close the result sets > when passing them back to the pool (using cursor.flush()). > > mxODBC it self does not provide connection pooling, since getting > this right is difficult if you don't know what the application is > doing (e.g. adjusting connection settings or keeping cursors alive). > > We do have connection pooling in mxODBC Zope DA and it's working well > for Zope. > > I tried this approach, it works OK. The only thing is, that the overhead of opening and closing the connections each time is very small. I tried calling a particular stored proc 50 times in a row, with and without opening/closing the connection in between. Timings are: Re-using the connection: real 0m21.221s user 0m15.108s sys 0m1.352s Opening and closing each time. real 0m21.461s user 0m15.252s sys 0m1.384s So, that is only ~ 0.4 ms overhead per call. Maybe the native Microsoft Linux driver is doing some clever re-use of connections here? > Best Regards, > -- > Marc-Andre Lemburg > eGenix.com > > Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: /mailman-archives/egenix-users/attachments/20171114/566544fa/attachment.htm From info at egenix.com Thu Nov 23 11:31:47 2017 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Thu Nov 23 11:32:08 2017 Subject: [egenix-users] ANN: PyDDF Python Herbst Sprint 2017 Message-ID: [This announcement is in German since it targets a Python sprint in D?sseldorf, Germany] ________________________________________________________________________ ANK?NDIGUNG PyDDF Python Herbst Sprint 2017 in D?sseldorf Samstag, 25.11.2017, 10:00-18:00 Uhr Sonntag, 26.11.2017, 10:00-18:00 Uhr trivago GmbH, Karl-Arnold-Platz 1A, 40474 D?sseldorf Python Meeting D?sseldorf https://www.pyddf.de/sprint2017-11/ ________________________________________________________________________ INFORMATION Das Python Meeting D?sseldorf (PyDDF) veranstaltet mit freundlicher Unterst?tzung der *trivago GmbH* ein Python Sprint Wochenende im September. Der Sprint findet am Wochenende 25./26.11.2017 in der trivago Niederlassung am Karl-Arnold-Platz 1A statt (nicht am Bennigsen-Platz 1). Folgende Themengebiete haben wir als Anregung angedacht: * Willkommens-Chatbot f?r D?sseldorf mir RasaHQ * PyEditor oder django-reversion-compare * Django Autotask oder FirtzConnection * YouTube Video Manager CLI * Kivy * Formula Pi (Roboter Rennen) * Jython Nat?rlich kann jeder Teilnehmer weitere Themen vorschlagen. Alles weitere und die Anmeldung findet Ihr auf der Sprint Seite: https://www.pyddf.de/sprint2017-11/ Teilnehmer sollten sich zudem auf der PyDDF Liste anmelden, da wir uns dort koordinieren: https://www.egenix.com/mailman/listinfo/pyddf ________________________________________________________________________ ?BER UNS Das Python Meeting D?sseldorf (PyDDF) ist eine regelm??ige Veranstaltung in D?sseldorf, die sich an Python Begeisterte aus der Region wendet: * https://pyddf.de/ Einen guten ?berblick ?ber die Vortr?ge bietet unser YouTube-Kanal, auf dem wir die Vortr?ge nach den Meetings ver?ffentlichen: * http://www.youtube.com/pyddf/ Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, D?sseldorf: * http://www.egenix.com/ * http://www.clark-consulting.eu/ Mit freundlichen Gr??en, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 23 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/