From vladdrac at gmail.com Thu Sep 14 17:14:11 2006 From: vladdrac at gmail.com (Ivo van der Wijk) Date: Thu Sep 14 16:14:14 2006 Subject: [egenix-users] mxODBC issues with MS SQLServer, blocking question and ZODB bloat issue Message-ID: Hi All, I've been looking into an issue the last few days at a customer, where a Zope/Plone instance is suffering from occasional 'hanging'. The system uses mxODBC / mxODBCDA / ZSQL extensively, and the problem 'feels' alot asif all of the threads end up blocking on a database connection. When the system hangs, there is hardly any CPU usage, and no threads are available to handle new requests (i.e. ZMI management access). The queries that are used are trivial. We've already tried the threading fix as described at https://www.egenix.com/mailman-archives/egenix-users/2005-October/113974.html, but that didn't seem to fix the problem. Currently, IODBC in stead of unixODBC is being tested. The current configuration is as follows: mxODBC 1.0.9 (commercial license) Zope 2.8.5 Plone 2.1.2 unixODBC freesoft OOB if anyone has similar experiences (I saw a post recently by robert rotterman to the list describing a similar issue) or any ideas on how to test / trace / fix this issue, please let me know :) Additionally, I would like to know if there is any way to have a timeout set on the max. duration of a query through mxODBC. Lastly, while debugging issues and the software in general, we noticed a small, mysterious ZODB bloat. It appears that when the mxODBCDA has 'connect on demand' enabled, it will occassionally create new transactions, possibly because of some internal, persisted, bookkeeping of the current connection state. You might want to look into this, having read only queries / pageviews result in ZODB writes is usually a bad thing IMHO. Regards, Ivo van der Wijk -- Drs. I.R. van der Wijk / m3r Consultancy B.V. Linux/Python/Zope/Plone and Open Source solutions PO-box 51091, 1007 EB Amsterdam, The Netherlands Email: ivo m3r.nl From mal at egenix.com Tue Sep 19 13:45:14 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Sep 19 12:45:21 2006 Subject: [egenix-users] mxODBC issues with MS SQLServer, blocking question and ZODB bloat issue In-Reply-To: References: Message-ID: <450FCA3A.6060203@egenix.com> Ivo van der Wijk wrote: > Hi All, > > I've been looking into an issue the last few days at a customer, where > a Zope/Plone instance is suffering from occasional 'hanging'. The > system uses mxODBC / mxODBCDA / ZSQL extensively, and the problem > 'feels' alot asif all of the threads end up blocking on a database > connection. When the system hangs, there is hardly any CPU usage, and > no threads are available to handle new requests (i.e. ZMI management > access). The queries that are used are trivial. It is possible that the ODBC driver you are using locks up the thread. Could you enable ODBC tracing in the ODBC manager to see where the lockups occur ? mxODBC releases the Python thread lock for most ODBC calls. However it is possible that the ODBC driver implementation does some I/O in places where you'd normally not expect this to happen. If the driver locks up in one of the calls which don't release the thread lock, then you end up with a blocked Python interpreter. Which ODBC driver are you using to access SQL Server ? > We've already tried the threading fix as described at > https://www.egenix.com/mailman-archives/egenix-users/2005-October/113974.html, > > but that didn't seem to fix the problem. Currently, IODBC in stead of > unixODBC is being tested. > > The current configuration is as follows: > > mxODBC 1.0.9 (commercial license) > Zope 2.8.5 > Plone 2.1.2 > unixODBC > freesoft OOB > > if anyone has similar experiences (I saw a post recently by robert > rotterman to the list describing a similar issue) or any ideas on how > to test / trace / fix this issue, please let me know :) > > Additionally, I would like to know if there is any way to have a > timeout set on the max. duration of a query through mxODBC. If the driver supports such timeouts, you can provide such a timeout in the connection string. mxODBC itself doesn't have a way to enforce a timeout because the ODBC driver is in charge of the networking between the client and server. > Lastly, while debugging issues and the software in general, we noticed > a small, mysterious ZODB bloat. It appears that when the mxODBCDA has > 'connect on demand' enabled, it will occassionally create new > transactions, possibly because of some internal, persisted, > bookkeeping of the current connection state. You might want to look > into this, having read only queries / pageviews result in ZODB writes > is usually a bad thing IMHO. In lazy connect mode, the connection object keeps track of whether there is a true connection to the database or not. I guess we could change the code to not store this state in a persisted attribute in a future release of the Zope DA. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 19 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From vladdrac at gmail.com Tue Sep 19 14:38:23 2006 From: vladdrac at gmail.com (Ivo van der Wijk) Date: Tue Sep 19 13:38:30 2006 Subject: [egenix-users] mxODBC issues with MS SQLServer, blocking question and ZODB bloat issue In-Reply-To: <450FCA3A.6060203@egenix.com> References: <450FCA3A.6060203@egenix.com> Message-ID: On 9/19/06, M.-A. Lemburg wrote: > > It is possible that the ODBC driver you are using locks > up the thread. Could you enable ODBC tracing in the ODBC manager > to see where the lockups occur ? > I will try this once I'm onsite at the customer again, I can't access the systems externally. > mxODBC releases the Python thread lock for most ODBC calls. > However it is possible that the ODBC driver implementation > does some I/O in places where you'd normally not expect this > to happen. > > If the driver locks up in one of the calls which > don't release the thread lock, then you end up with a > blocked Python interpreter. > > Which ODBC driver are you using to access SQL Server ? > unixODBC currently, and we're experimenting with iODBC (or do you mean something else? I'm a bit confused about the entire chain of software involved :) > > > > Additionally, I would like to know if there is any way to have a > > timeout set on the max. duration of a query through mxODBC. > > If the driver supports such timeouts, you can provide such a > timeout in the connection string. mxODBC itself doesn't have > a way to enforce a timeout because the ODBC driver is in > charge of the networking between the client and server. > Ok, will try that. Thanks for your reply, regards, Ivo van der Wijk -- Drs. I.R. van der Wijk / m3r Consultancy B.V. Linux/Python/Zope/Plone and Open Source solutions PO-box 51091, 1007 EB Amsterdam, The Netherlands Email: ivo m3r.nl From mal at egenix.com Tue Sep 19 15:00:48 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Sep 19 14:01:46 2006 Subject: [egenix-users] mxODBC issues with MS SQLServer, blocking question and ZODB bloat issue In-Reply-To: References: <450FCA3A.6060203@egenix.com> Message-ID: <450FDBF0.7060604@egenix.com> Ivo van der Wijk wrote: > On 9/19/06, M.-A. Lemburg wrote: >> >> It is possible that the ODBC driver you are using locks >> up the thread. Could you enable ODBC tracing in the ODBC manager >> to see where the lockups occur ? >> > > I will try this once I'm onsite at the customer again, I can't access > the systems externally. > >> mxODBC releases the Python thread lock for most ODBC calls. >> However it is possible that the ODBC driver implementation >> does some I/O in places where you'd normally not expect this >> to happen. >> >> If the driver locks up in one of the calls which >> don't release the thread lock, then you end up with a >> blocked Python interpreter. >> >> Which ODBC driver are you using to access SQL Server ? >> > > unixODBC currently, and we're experimenting with iODBC (or do you mean > something else? I'm a bit confused about the entire chain of software > involved :) I meant the ODBC driver which is registered with the ODBC manager. Choosing the right ODBC driver usually makes a big difference in terms of stability, performance and robustness. The typical ODBC stack looks like this: Python Application | mxODBC | ODBC Manager (Windows/Mac OS X ODBC Manager, unixODBC, iODBC) | ODBC Driver (for SQL Server, Oracle, DB2, MySQL, etc.) | Network | Database Server (SQL Server, Oracle, DB2, MySQL, etc.) mxODBC can also be linked directly against the ODBC driver, but we usually recommend to use an ODBC manager for setting up the link to the database as these are easier to setup and configure. >> > Additionally, I would like to know if there is any way to have a >> > timeout set on the max. duration of a query through mxODBC. >> >> If the driver supports such timeouts, you can provide such a >> timeout in the connection string. mxODBC itself doesn't have >> a way to enforce a timeout because the ODBC driver is in >> charge of the networking between the client and server. >> > > Ok, will try that. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 19 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From info at egenix.com Wed Sep 20 13:23:56 2006 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Wed Sep 20 12:26:28 2006 Subject: [egenix-users] ANN: eGenix mxODBC Zope Database Adapter 1.0.10 for Intel Mac OS X Message-ID: <451116BC.8060700@egenix.com> ________________________________________________________________________ ANNOUNCEMENT EGENIX.COM mxODBC Zope Database Adapter Version 1.0.10 Usable with Zope and the Plone CMS. Available for Zope 2.3 through 2.10 on Windows, Linux, Mac OS X, Solaris and FreeBSD ________________________________________________________________________ INTRODUCTION The eGenix mxODBC Zope Database Adapter allows you to easily connect your Zope or Plone installation to just about any database backend on the market today, giving you the reliability of the commercially supported eGenix product mxODBC and the flexibility of the ODBC standard as middle-tier architecture. The mxODBC Zope Database Adapter is highly portable, just like Zope itself and provides a high performance interface to all your ODBC data sources, using a single well-supported interface on Windows, Linux, Mac OS X, Solaris and FreeBSD. This makes it ideal for deployment in ZEO Clusters and Zope hosting environments where stability and high performance are a top priority, establishing an excellent basis and scalable solution for your CMS. ________________________________________________________________________ NEWS Our mxODBC Zope DA product is now available and supported on both Intel and PPC Mac OS X versions. ________________________________________________________________________ UPGRADING If you have already bought mxODBC Zope DA 1.0.x licenses, you can use these license for the 1.0.10 version as well. There is no need to buy new licenses. The same is true for evaluation license users. ________________________________________________________________________ MORE INFORMATION For more information on the mxODBC Zope Database Adapter, licensing and download instructions, please visit our web-site: http://zope.egenix.com/ You can buy mxODBC Zope DA licenses online from the eGenix.com shop at: http://shop.egenix.com/ Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 20 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From moof at metamoof.net Thu Sep 21 11:24:31 2006 From: moof at metamoof.net (Moof) Date: Thu Sep 21 16:17:03 2006 Subject: [egenix-users] Python 2.5 binaries for widnows of the varios MX packages Message-ID: Hi there, Any ideas when the 2.5 binaries for windows for the various mx pyton extensions will come out? Thanks, Moof - currently planning 2.5 upgrade path From mal at egenix.com Fri Sep 22 12:01:55 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Sep 22 11:08:13 2006 Subject: [egenix-users] Python 2.5 binaries for windows of the various MX packages In-Reply-To: References: Message-ID: <4513A683.2070208@egenix.com> Moof wrote: > Hi there, > > Any ideas when the 2.5 binaries for windows for the various mx pyton > extensions will come out? We're currently planing to release a new version of both egenix-mx-base and egenix-mx-commercial in the next few weeks which will also support the new Python C API after PEP 353 was applied in Python 2.5: http://docs.python.org/dev/whatsnew/pep-353.html We've put up a Windows installer for Python 2.5 up on the web-site: http://www.egenix.com/files/python/eGenix-mx-Extensions.html#Download-mxBASE However, I suggest you use the current snapshot of egenix-mx-base, since this is already close to the new release which will use a new version number: http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py1.5.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py2.1.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py2.2.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py2.3.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py2.4.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.win32-py2.5.exe http://www.egenix.com/files/python/egenix-mx-base-2.1.0_20060922.zip Note that the eGenix mx extensions 2.0.x are *not* compatible with Python 2.5 on 64-bit platforms. This is due to the PEP 353 changes. You'll run into similar problems with lots of other C extensions for Python. Most of them will need to be updated before the compile with Python 2.5. > Thanks, > > Moof - currently planning 2.5 upgrade path -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 22 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::