From dan.fairs at gmail.com Mon Jul 13 16:20:59 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Mon Jul 13 16:21:04 2009 Subject: [egenix-users] Invalid cursor state Message-ID: Hi, We're experiencing intermittent problems using the mxODBCZopeDA connecting to a SQL Server database. Every so often (in our development environments), we'll see tracebacks that look like this: Site Error An error was encountered while publishing this resource. mx.ODBC.Error.InternalError Sorry, a site error occurred. Traceback (innermost last): ? Module ZPublisher.Publish, line 202, in publish_module_standard ? Module ZPublisher.Publish, line 150, in publish ? Module ZPublisher.Publish, line 119, in publish ? Module ZPublisher.mapply, line 88, in mapply ? Module ZPublisher.Publish, line 42, in call_object ? Module webui.browser.repositoriesui.base, line 28, in __call__ ? Module webui.browser.repositoriesui.repositories, line 43, in update ? Module webui.browser.repositoriesui.base, line 251, in getAssessments ? Module app.directdb.loaders, line 124, in repositories ? Module app.db._db, line 86, in execute ? Module app.db._db, line 119, in _execute ? Module Shared.DC.ZRDB.DA, line 500, in __call__ ? Module Products.mxODBCZopeDA.ZopeDA, line 1503, in query ? Module Products.mxODBCZopeDA.ZopeDA, line 1400, in run_cursor_callback ? Module Products.mxODBCZopeDA.ZopeDA, line 999, in errorhandler InternalError: ('24000', 0, '[unixODBC][FreeTDS][SQL Server]Invalid cursor state', 4543) Once this has happened, refreshing the page consistently returns the same error. Other pages in the site (which use other FSZSQLMethods using the same ZopeDA instance in the ZODB) continue to work fine - so the problem appears to be bound to a particular FSZSQLMethod. Restarting Zope makes the problem go away temporarily. The problem may recur quickly, or it may take days to appear again. When we load a page that provokes the error, while running SQL Profiler on the database server, we can see the SQL generated by the ZSQL method arrive and appear to be served correctly. I've turned on tracing in /etc/odbcinst.ini (though of course, since I've done that I can't provoke the error! I'll attach a trace log when I get one.) We know that the SQL that is being run is valid - we can copy it out of Profiler and run it in Management Studio; and besides, the page usually works flawlessly. The ZSQL method itself has no caching specified. All FSZSQLMethods live in FileSystemDirectoryViews, and are invoked directly from trusted Python code (ie. not a skin script). We're using the following software versions (from Ubuntu's package manager): unixODBC 2.2.11-16build2 tdsodbc 0.82-3ubuntu1 Zope 2.10.4 (built using buildout and sources, not Ubuntu packages) mxODBCZopeDA-1.0.10 SQL Server 2005 Developer (also happens on SQL Server 2008 Express) Zope server is Ubuntu 8.10, though this also happens with 9.04. Python 2.4.5 (built from source) The DA is configured with the following parameters: ODBC Driver Manager: unixODBC Connection pool size: 25 All connection options set to 'off' Connection state - 'Open Connection' is checked. We don't see anything in the logs on the SQL Server side. I'm not sure where to look next, or what further tests to carry out in order to debug this further. The query does appear to be running correctly (with both BatchStarting and BatchCompleted entries in SQL Profiler). Has anyone seen something like this before, or have any idea what to look at when it starts happening again? Many thanks, Dan -- Dan Fairs | http://www.fezconsulting.com/ From mal at egenix.com Mon Jul 13 17:52:55 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Jul 13 16:52:58 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: References: Message-ID: <4A5B4A47.4080608@egenix.com> Dan Fairs wrote: > Hi, > > We're experiencing intermittent problems using the mxODBCZopeDA > connecting to a SQL Server database. Every so often (in our development > environments), we'll see tracebacks that look like this: > > Site Error > An error was encountered while publishing this resource. > > mx.ODBC.Error.InternalError > > Sorry, a site error occurred. > > Traceback (innermost last): > > ? Module ZPublisher.Publish, line 202, in publish_module_standard > ? Module ZPublisher.Publish, line 150, in publish > ? Module ZPublisher.Publish, line 119, in publish > ? Module ZPublisher.mapply, line 88, in mapply > ? Module ZPublisher.Publish, line 42, in call_object > ? Module webui.browser.repositoriesui.base, line 28, in __call__ > ? Module webui.browser.repositoriesui.repositories, line 43, in update > ? Module webui.browser.repositoriesui.base, line 251, in getAssessments > ? Module app.directdb.loaders, line 124, in repositories > ? Module app.db._db, line 86, in execute > ? Module app.db._db, line 119, in _execute > ? Module Shared.DC.ZRDB.DA, line 500, in __call__ > > ? Module Products.mxODBCZopeDA.ZopeDA, line 1503, in query > ? Module Products.mxODBCZopeDA.ZopeDA, line 1400, in > run_cursor_callback > ? Module Products.mxODBCZopeDA.ZopeDA, line 999, in errorhandler > InternalError: ('24000', 0, '[unixODBC][FreeTDS][SQL Server]Invalid > cursor state', 4543) > > Once this has happened, refreshing the page consistently returns the > same error. Other pages in the site (which use other FSZSQLMethods using > the same ZopeDA instance in the ZODB) continue to work fine - so the > problem appears to be bound to a particular FSZSQLMethod. Restarting > Zope makes the problem go away temporarily. The problem may recur > quickly, or it may take days to appear again. > > When we load a page that provokes the error, while running SQL Profiler > on the database server, we can see the SQL generated by the ZSQL method > arrive and appear to be served correctly. I've turned on tracing in > /etc/odbcinst.ini (though of course, since I've done that I can't > provoke the error! I'll attach a trace log when I get one.) Since you are using FreeTDS, you can also try enabling TDS tracing which will have very low-level details about the error... Edit ~/.freetds.conf and enable these settings in the [global] section: [global] ... # Whether to write a TDSDUMP file for diagnostic purposes # (setting this to /tmp is insecure on a multi-user system) dump file = /tmp/freetds.log debug level = 10 > We know that the SQL that is being run is valid - we can copy it out of > Profiler and run it in Management Studio; and besides, the page usually > works flawlessly. The ZSQL method itself has no caching specified. All > FSZSQLMethods live in FileSystemDirectoryViews, and are invoked directly > from trusted Python code (ie. not a skin script). This sounds a lot like a bug somewhere in FreeTDS or unixODBC or perhaps a connection problem that causes the cursor to get invalidated. Have you tried using the latest unixODBC version ? > We're using the following software versions (from Ubuntu's package > manager): > > unixODBC 2.2.11-16build2 > tdsodbc 0.82-3ubuntu1 > Zope 2.10.4 (built using buildout and sources, not Ubuntu packages) > mxODBCZopeDA-1.0.10 > SQL Server 2005 Developer (also happens on SQL Server 2008 Express) > Zope server is Ubuntu 8.10, though this also happens with 9.04. > Python 2.4.5 (built from source) > > The DA is configured with the following parameters: > > ODBC Driver Manager: unixODBC > Connection pool size: 25 > All connection options set to 'off' > Connection state - 'Open Connection' is checked. > > We don't see anything in the logs on the SQL Server side. > > I'm not sure where to look next, or what further tests to carry out in > order to debug this further. The query does appear to be running > correctly (with both BatchStarting and BatchCompleted entries in SQL > Profiler). > > Has anyone seen something like this before, or have any idea what to > look at when it starts happening again? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Mon Jul 13 17:15:35 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Mon Jul 13 17:15:39 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: <4A5B4A47.4080608@egenix.com> References: <4A5B4A47.4080608@egenix.com> Message-ID: <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Hi, Many thanks for your reply. > Since you are using FreeTDS, you can also try enabling TDS tracing > which will have very low-level details about the error... > > Edit ~/.freetds.conf and enable these settings in the [global] > section: > > [global] > ... > # Whether to write a TDSDUMP file for diagnostic purposes > # (setting this to /tmp is insecure on a multi-user system) > dump file = /tmp/freetds.log > debug level = 10 > Great - I didn't know about that, thanks. > This sounds a lot like a bug somewhere in FreeTDS or unixODBC > or perhaps a connection problem that causes the cursor to > get invalidated. > > Have you tried using the latest unixODBC version ? > Not yet - we've just stuck to what's in the distribution's repositories, since that's the baseline already installed on the production servers. I'll experiment with a newer version, though. Cheers, Dan -- Dan Fairs | http://www.fezconsulting.com/ From charlie at egenix.com Tue Jul 14 13:22:50 2009 From: charlie at egenix.com (Charlie Clark) Date: Tue Jul 14 12:22:55 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: Am 13.07.2009, 17:15 Uhr, schrieb Dan Fairs : > Not yet - we've just stuck to what's in the distribution's repositories, > since that's the baseline already installed on the production servers. > I'll experiment with a newer version, though. Hiya Dan, I would just add that we never recommend FreeTDS for production environments. Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Tue Jul 14 17:48:06 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Tue Jul 14 17:48:14 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: Hi Charlie, Good to meet you at EuroPython! > I would just add that we never recommend FreeTDS for production > environments. > What would you normally recommend? Thanks Dan -- Dan Fairs | http://www.fezconsulting.com/ From charlie at egenix.com Tue Jul 14 19:49:32 2009 From: charlie at egenix.com (Charlie Clark) Date: Tue Jul 14 18:49:36 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: Am 14.07.2009, 17:48 Uhr, schrieb Dan Fairs : > Hi Charlie, > Good to meet you at EuroPython! Indeed. Still planning to add another answer to your report about the distinct lack of Zope/Plone talks at EP. > I would just add that we never recommend FreeTDS for production > environments. > What would you normally recommend? In a pure Python environment then mxODBC Connect - avoid all that messy configuration of managers, drivers and firewalls. We will hopefully soon be releasing a Zope version of this. What Zope version are you on as it looks as if we're going to have to choose between Zope >= 2.12 and earlier because of the move to eggs. Other than that then "proper" drivers from Openlink, et al. By proper I mean avoid the "instant" or "lite" versions as these tend to be wrappers around JDBC drivers and have their own problems. Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Tue Jul 14 19:06:01 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Tue Jul 14 19:06:08 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: > In a pure Python environment then mxODBC Connect - avoid all that > messy configuration of managers, drivers and firewalls. We will > hopefully soon be releasing a Zope version of this. What Zope > version are you on as it looks as if we're going to have to choose > between Zope >= 2.12 and earlier because of the move to eggs. > Ah, now that *would* be interesting. The client's currently on Zope 2.10.4, though I'd like to move to the 2.12. We did have problems with a 2.11 ZEO client talking to a 2.10 ZEO server, though; upgrading the ZEO server, is more major since so many things here use it. A release that works with 2.10 would certainly be more palatable. If we're in wish-list mode, a Zope 3 implementation would be handy as well. All the new code we're developing is Zope 3-style (using browser views and the component architecture), and digging around in the ZODB to pull out a ZSQL method hooked up to an mxODBCZopeDA instance living in content space simply isn't very nice. I'd like to be able to register a database connection as a named utility (configured through ZCML or optionally as a persistent item in the ZODB), and just pass it SQL to execute. > Other than that then "proper" drivers from Openlink, et al. By > proper I mean avoid the "instant" or "lite" versions as these tend > to be wrappers around JDBC drivers and have their own problems. OK - I didn't know about those, thanks; and thanks for the warning, too. Thanks, Dan -- Dan Fairs | http://www.fezconsulting.com/ From charlie at egenix.com Tue Jul 14 20:24:02 2009 From: charlie at egenix.com (Charlie Clark) Date: Tue Jul 14 19:24:07 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: Am 14.07.2009, 19:06 Uhr, schrieb Dan Fairs : > Ah, now that *would* be interesting. The client's currently on Zope > 2.10.4, though I'd like to move to the 2.12. We did have problems with a > 2.11 ZEO client talking to a 2.10 ZEO server, though; upgrading the ZEO > server, is more major since so many things here use it. A release that > works with 2.10 would certainly be more palatable. I think we're likely to release for >= 2.12 (at least as far as the packaging goes) Unfortunately there are other problems related to some of the global variables. > If we're in wish-list mode, a Zope 3 implementation would be handy as > well. All the new code we're developing is Zope 3-style (using browser > views and the component architecture), and digging around in the ZODB to > pull out a ZSQL method hooked up to an mxODBCZopeDA instance living in > content space simply isn't very nice. I'd like to be able to register a > database connection as a named utility (configured through ZCML or > optionally as a persistent item in the ZODB), and just pass it SQL to > execute. You have my sympathies although after the initial shock I found the migration not to be that difficuilt! Moving to file-system based ZSQLs should make life a bit easier (if only Mr. Withers had stuck with the same handling of parameters that the other file system products have...). Of course, the mxODBCZopeDA has the advantage of allowing you to use execute on connection instance objects. A Zope 3 release itself isn't on the cards as Zope 3 itself no longer exists. There are recipes for using mxODBC and mxODBC Connect with Zope 3 on non-windows systems. Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From info at egenix.com Thu Jul 16 11:28:19 2009 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Thu Jul 16 10:28:54 2009 Subject: [egenix-users] EuroPython 2009: Making 50 Mio. EUR per year using Python Message-ID: <4A5EE4A3.1070805@egenix.com> Now available as video... http://www.egenix.com/company/news/EuroPython-2009-Lightning-Talk.html Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 16 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From charlie at egenix.com Thu Jul 16 23:32:00 2009 From: charlie at egenix.com (Charlie Clark) Date: Thu Jul 16 22:32:07 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: Am 14.07.2009, 19:24 Uhr, schrieb Charlie Clark : > If we're in wish-list mode, a Zope 3 implementation would be handy as > well. All the new code we're developing is Zope 3-style (using browser > views and the component architecture), and digging around in the ZODB to > pull out a ZSQL method hooked up to an mxODBCZopeDA instance living in > content space simply isn't very nice. I'd like to be able to register a > database connection as a named utility (configured through ZCML or > optionally as a persistent item in the ZODB), and just pass it SQL to > execute. Dan, to come back to this. You want something a bit like this: class DatabaseConnection(Persistent): implements(IDatabaseConnection) registered locally and called as a named utility db = getUtility(IDatabaseConnection, u"the specific connection")() db.execute("INSERT (gromit, doodah, wotsit) INTO thingy (?, ?, ?)", (*args)) Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Fri Jul 17 10:34:42 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Fri Jul 17 10:34:49 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> Message-ID: <15117BBE-AB39-4C43-9813-8F96412F42E7@gmail.com> Hi Charlie, > to come back to this. You want something a bit like this: > > class DatabaseConnection(Persistent): > implements(IDatabaseConnection) > > registered locally and called as a named utility > > db = getUtility(IDatabaseConnection, u"the specific connection")() > db.execute("INSERT (gromit, doodah, wotsit) INTO thingy (?, ?, ?)", > (*args)) > Ah - that's great. I was wondering whether something like that would work. Presumably the DatabaseConnection would actually be the mxODBCZopeDA object itself? Thanks Dan -- Dan Fairs | http://www.fezconsulting.com/ From charlie at egenix.com Sat Jul 18 01:02:48 2009 From: charlie at egenix.com (Charlie Clark) Date: Sat Jul 18 00:02:56 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: <15117BBE-AB39-4C43-9813-8F96412F42E7@gmail.com> References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> <15117BBE-AB39-4C43-9813-8F96412F42E7@gmail.com> Message-ID: Am 17.07.2009, 10:34 Uhr, schrieb Dan Fairs : > Ah - that's great. I was wondering whether something like that would > work. Presumably the DatabaseConnection would actually be the > mxODBCZopeDA object itself? Yes, but the code is sketched. There is a reference implementation of a similar utility in SQLAlchemy for Plone that we might be able to use but the good news is that you can use the .execute() on the connection whenever you want. Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Tue Jul 21 18:21:04 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Tue Jul 21 18:21:10 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: <4A5B4A47.4080608@egenix.com> References: <4A5B4A47.4080608@egenix.com> Message-ID: > > Have you tried using the latest unixODBC version ? > >> We're using the following software versions (from Ubuntu's package >> manager): >> >> unixODBC 2.2.11-16build2 >> tdsodbc 0.82-3ubuntu1 >> Zope 2.10.4 (built using buildout and sources, not Ubuntu packages) >> mxODBCZopeDA-1.0.10 >> SQL Server 2005 Developer (also happens on SQL Server 2008 Express) >> Zope server is Ubuntu 8.10, though this also happens with 9.04. >> Python 2.4.5 (built from source) Just to follow up, we've tried the latest version of unixODBC (2.2.14) to no avail. We seem to be able to provoke the problem most reliably with Python code that does the following - using ZSQL methods, the same ZSQL method being used for each of the SELECTs: - Cause a row to be SELECTed from a table - UPDATE the same row - SELECT that same row again We'll then get the 'Invalid cursor state' message on the second select. Putting a time.sleep(1) before the update and the second select seems to prevent the error occurring. Googling the error does seem to indicate that this happens (at the C level) when two results handles are open simultaneously; the sleep seems to allow some opportunity for cleanup. Obivously not an ideal workaround, but it will at least keep us moving until we find a real solution (or re-jig the code flow so that select/ update/select pattern doesn't happen). Thanks Dan -- Dan Fairs | http://www.fezconsulting.com/ From mal at egenix.com Tue Jul 21 23:34:39 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Jul 21 22:34:41 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: References: <4A5B4A47.4080608@egenix.com> Message-ID: <4A66265F.8000409@egenix.com> Dan Fairs wrote: >> >> Have you tried using the latest unixODBC version ? >> >>> We're using the following software versions (from Ubuntu's package >>> manager): >>> >>> unixODBC 2.2.11-16build2 >>> tdsodbc 0.82-3ubuntu1 >>> Zope 2.10.4 (built using buildout and sources, not Ubuntu packages) >>> mxODBCZopeDA-1.0.10 >>> SQL Server 2005 Developer (also happens on SQL Server 2008 Express) >>> Zope server is Ubuntu 8.10, though this also happens with 9.04. >>> Python 2.4.5 (built from source) > > > Just to follow up, we've tried the latest version of unixODBC (2.2.14) > to no avail. > > We seem to be able to provoke the problem most reliably with Python code > that does the following - using ZSQL methods, the same ZSQL method being > used for each of the SELECTs: > > - Cause a row to be SELECTed from a table > - UPDATE the same row > - SELECT that same row again > > We'll then get the 'Invalid cursor state' message on the second select. > > Putting a time.sleep(1) before the update and the second select seems to > prevent the error occurring. Googling the error does seem to indicate > that this happens (at the C level) when two results handles are open > simultaneously; the sleep seems to allow some opportunity for cleanup. Are you using the ZSQL method from different threads ? I assume you are running the UPDATE using a different ZSQL method, right ? It is possible that the ODBC driver tries to do some optimization by keeping the result set and the cursor from the first SELECT alive even after the Zope DA has closed it. This would explain the invalid cursor state, since SQL Server does not support multiple active statements on the same connection unless you enable the MARS feature available in SQL Server 2005 and later: http://msdn.microsoft.com/en-us/library/ms131686.aspx (you can enable this via a connection parameter "Mars_Connection=on") > Obivously not an ideal workaround, but it will at least keep us moving > until we find a real solution (or re-jig the code flow so that > select/update/select pattern doesn't happen). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 21 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Jul 21 23:45:28 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Jul 21 22:45:32 2009 Subject: [egenix-users] HTML documentation for our products Message-ID: <4A6628E8.1050201@egenix.com> For a long while (ever since we dropped support for the 2.0 series of the eGenix mx products) we have only had PDF documentation for our products. This was mainly due to trying to streamline our documentation development and moving away from hand-coding HTML like we did in the days of mx Base 1.x and 2.x and using MS Word or OpenOffice to write the docs. Some users have complained about this and we've listened. Using mxTidy, elementtree and a few scripts, we've now managed to transform the office application HTML exports into more or less sane XHTML. We're currently converting the docs one by one to HTML and putting them online. These are the first few attempts: mxODBC: http://www.egenix.com/products/python/mxODBC/doc/ mxODBC Connect: http://www.egenix.com/products/python/mxODBCConnect/doc/ http://www.egenix.com/products/python/mxODBCConnect/doc/mxodbc/ mxDateTime: http://www.egenix.com/products/python/mxBase/mxDateTime/doc/ mxTextTools: http://www.egenix.com/products/python/mxBase/mxTextTools/doc/ mxODBC Zope DA: http://www.egenix.com/products/zope/mxODBCZopeDA/doc/ http://www.egenix.com/products/zope/mxODBCZopeDA/doc/mxodbc/ Over time, we'll add more converted HTML files and also improve the quality of the conversion some more (provided we find some clever AI library for Python that understand the infinite wisdom of the MS Word Export function). Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 21 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dan.fairs at gmail.com Wed Jul 22 10:24:25 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Wed Jul 22 10:24:31 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: <4A66265F.8000409@egenix.com> References: <4A5B4A47.4080608@egenix.com> <4A66265F.8000409@egenix.com> Message-ID: >> >> - Cause a row to be SELECTed from a table >> - UPDATE the same row >> - SELECT that same row again > > Are you using the ZSQL method from different threads ? > No, these are in the same request; therefore the same thread. > I assume you are running the UPDATE using a different ZSQL method, > right ? That's correct. (We do have ZSQL methods that do an UPDATE and then immediately SELECT the newly-updated data out, incidentally; however, the case I'm talking about here doesn't do that. Splitting those up was one of the stages in our investigation.) > > It is possible that the ODBC driver tries to do some optimization > by keeping the result set and the cursor from the first SELECT > alive even after the Zope DA has closed it. > > This would explain the invalid cursor state, since SQL Server > does not support multiple active statements on the same connection > unless you enable the MARS feature available in SQL Server 2005 > and later: > > http://msdn.microsoft.com/en-us/library/ms131686.aspx > > (you can enable this via a connection parameter "Mars_Connection=on") > Ah - now that's interesting. Now you mention it, I do remember MARS from my .NET days. Unfortunately the client's servers are only running SQL Server 2000; however, we've got 2005 and 2008 dev instances here, so we'll try it on those and see if does resolve the issue. Thanks Dan -- Dan Fairs | http://www.fezconsulting.com/ From dan.fairs at gmail.com Thu Jul 23 13:35:29 2009 From: dan.fairs at gmail.com (Dan Fairs) Date: Thu Jul 23 13:35:35 2009 Subject: [egenix-users] Invalid cursor state In-Reply-To: <4A66265F.8000409@egenix.com> References: <4A5B4A47.4080608@egenix.com> <4A66265F.8000409@egenix.com> Message-ID: <3B13B0B2-9028-4BCB-BCAE-C232BDBD3253@gmail.com> Another update, for the benefit of the archives. >> >> Putting a time.sleep(1) before the update and the second select >> seems to >> prevent the error occurring. Googling the error does seem to indicate >> that this happens (at the C level) when two results handles are open >> simultaneously; the sleep seems to allow some opportunity for >> cleanup. > > Unfortunately, the sleep just seemed to reduce the frequency of the error occurring. A colleague (working on Mac OS X with iODBC on FreeTDS rather than Linux, unixODBC and FreeTDS) has also just experienced the problem, leading to the conclusion that the problem is likely with FreeTDS. > It is possible that the ODBC driver tries to do some optimization > by keeping the result set and the cursor from the first SELECT > alive even after the Zope DA has closed it. > > This would explain the invalid cursor state, since SQL Server > does not support multiple active statements on the same connection > unless you enable the MARS feature available in SQL Server 2005 > and later: > > http://msdn.microsoft.com/en-us/library/ms131686.aspx > > (you can enable this via a connection parameter "Mars_Connection=on") We tried this, but we continued to get the problem. The avenue we're going down now is to move all the 'problematic' SQL into stored procedures on a case-by-case basis. So far, this has stopped the problem recurring in each place where the change is made. Thanks Dan -- Dan Fairs | http://www.fezconsulting.com/ From mal at egenix.com Fri Jul 24 20:50:38 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Jul 24 19:50:42 2009 Subject: [egenix-users] HTML documentation for our products In-Reply-To: <4A6628E8.1050201@egenix.com> References: <4A6628E8.1050201@egenix.com> Message-ID: <4A69F46E.8080906@egenix.com> M.-A. Lemburg wrote: > For a long while (ever since we dropped support for the 2.0 series > of the eGenix mx products) we have only had PDF documentation for > our products. > > This was mainly due to trying to streamline our documentation > development and moving away from hand-coding HTML like we did > in the days of mx Base 1.x and 2.x and using MS Word or OpenOffice > to write the docs. > > Some users have complained about this and we've listened. > > Using mxTidy, elementtree and a few scripts, we've now managed to > transform the office application HTML exports into more or less > sane XHTML. > > We're currently converting the docs one by one to HTML and putting > them online. These are the first few attempts: > > mxODBC: > http://www.egenix.com/products/python/mxODBC/doc/ > > mxODBC Connect: > http://www.egenix.com/products/python/mxODBCConnect/doc/ > http://www.egenix.com/products/python/mxODBCConnect/doc/mxodbc/ > > mxDateTime: > http://www.egenix.com/products/python/mxBase/mxDateTime/doc/ > > mxTextTools: > http://www.egenix.com/products/python/mxBase/mxTextTools/doc/ > > mxODBC Zope DA: > http://www.egenix.com/products/zope/mxODBCZopeDA/doc/ > http://www.egenix.com/products/zope/mxODBCZopeDA/doc/mxodbc/ > > Over time, we'll add more converted HTML files and also improve > the quality of the conversion some more (provided we find some > clever AI library for Python that understand the infinite > wisdom of the MS Word Export function). Here's the next batch: mxBeeBase: http://www.egenix.com/products/python/mxBase/mxBeeBase/doc/ mxTools: http://www.egenix.com/products/python/mxBase/mxTools/doc/ mxProxy: http://www.egenix.com/products/python/mxBase/mxProxy/doc/ mxURL: http://www.egenix.com/products/python/mxBase/mxURL/doc/ mxUID: http://www.egenix.com/products/python/mxBase/mxUID/doc/ mxStack: http://www.egenix.com/products/python/mxBase/mxStack/doc/ mxQueue: http://www.egenix.com/products/python/mxBase/mxQueue/doc/ mxNumber: http://www.egenix.com/products/python/mxBase/mxNumber/doc/ mxTidy: http://www.egenix.com/products/python/mxBase/mxTidy/doc/ Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 24 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From charlie at egenix.com Wed Jul 29 12:59:12 2009 From: charlie at egenix.com (Charlie Clark) Date: Wed Jul 29 11:59:16 2009 Subject: Wish list (was Re: [egenix-users] Invalid cursor state) In-Reply-To: References: <4A5B4A47.4080608@egenix.com> <3CF2A949-4EEE-459C-97CE-320450C43053@gmail.com> <15117BBE-AB39-4C43-9813-8F96412F42E7@gmail.com> Message-ID: Am 18.07.2009, 00:02 Uhr, schrieb Charlie Clark : > Yes, but the code is sketched. There is a reference implementation of a > similar utility in SQLAlchemy for Plone that we might be able to use but > the good news is that you can use the .execute() on the connection > whenever you want. As a matter of interest: is zope.rdb.interfaces sufficient for a utilities based approach? Charlie -- Charlie Clark eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/